Command input for remote supervision of multiple autonomous aircraft

Abstract

Embodiments provide an animated graphical user interface for simultaneously monitoring multiple autonomous aircraft. More specifically, the techniques disclosed herein provide a GUI that allows one-to-many supervision in which a remote supervisor (e.g., human or artificial intelligence) can continuously monitor and / or interact with multiple autonomous aircraft. For example, the autonomous aircraft may include one or more autonomous electric vertical take-off and landing (eVTOL) aircraft. Embodiments provide command input systems and techniques that reduce human errors by assigning input mechanisms and command input menus to unique flight parameters.

Description

[0001] Cross - reference to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 422,832, filed on November 4, 2022, and entitled "Command Input For Remote Supervision Of Multiple Autonomous Aircraft Command", the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0003] This application is related to U.S. Patent Application No. 18 / 469,965, filed on September 19, 2023, and entitled "Remote Supervision of Multiple Autonomous Aircraft", which claims the priority of U.S. Provisional Patent Application No. 63 / 408,088, filed on September 19, 2022, and entitled "Remote Supervision of Multiple Autonomous Aircraft", and U.S. Design Patent Application No. 29 / 853,825, filed on September 19, 2022, and entitled "Multi - Vehicle Supervisor Interface". Summary of the Invention

[0004] Embodiments provide techniques for communicating with a computer - implemented remote supervisor that simultaneously supervises the functionality of multiple autonomous aircraft. According to various embodiments, a graphical user interface (GUI) is provided for providing control input to a computer - implemented remote supervisor to simultaneously monitor, supervise, and / or communicate (e.g., interact) with multiple autonomous aircraft. The GUI can be an animated GUI that includes graphical representations of aircraft and associated information displayed on (e.g., overlaid on) a map that represents the area over which the autonomous aircraft are flying. The systems, devices, and techniques described herein provide a command input system and method that allows for the rapid manipulation of autonomous aircraft under supervision and manages clutter with respect to the display scale as the number of autonomous aircraft supervised by the same supervisor increases.

[0005] An embodiment provides a method for monitoring and interacting with a plurality of autonomous aircraft. The method includes displaying, by a server computer, a graphical user interface (GUI) on a display device. The method further includes representing, by the server computer, the plurality of autonomous aircraft using a plurality of icons displayed on the GUI. The server computer receives an input that selects a first icon among the plurality of icons displayed on the GUI. The server computer receives a first signal from an input device that includes a plurality of input mechanisms, where each input mechanism is assigned to a unique flight parameter. In response to receiving the first signal, the server computer displays a first menu among a plurality of menus on the GUI. The first menu is associated with a first flight parameter of the autonomous aircraft represented by the first icon. The first menu includes a plurality of first preset commands. The method further includes receiving, by the server computer, a selection of a first command among the plurality of first preset commands. The method includes transmitting, by the server computer, the first command to a first autonomous aircraft represented by the first icon on the GUI.

[0006] Some embodiments provide a system that includes a display device, one or more processors, an input device that includes a plurality of input mechanisms, and a memory that stores instructions, where the input device is communicatively coupled to the one or more processors, and the instructions, when executed by the one or more processors, cause the one or more processors to implement the above method.

[0007] Embodiments further provide one or more non-transitory computer-readable storage media that store instructions, which, when executed on a server computer for remotely monitoring and interacting with the plurality of autonomous aircraft, cause the server computer to implement the above method.

[0008] An embodiment provides an input device that includes a plurality of input mechanisms, each input mechanism being configured to provide a command to change a unique parameter, the unique parameter including a flight parameter associated with an autonomous aircraft. The input device is communicatively coupled to a server computer that transmits the command to the autonomous aircraft.

[0009] To better understand the nature and advantages of the present disclosure, reference should be made to the following description and drawings. However, it should be understood that each of the drawings is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure. Further, as a general rule, and unless it is apparent from the description to the contrary, when elements in different drawings are denoted by the same reference numerals, these elements are generally the same or at least similar in function or purpose. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1Illustrates a block diagram of an exemplary flight supervision platform for monitoring and interacting with a plurality of autonomous aircraft according to various embodiments;

[0011] Figure 2A Illustrates an exemplary animated graphical user interface (GUI) of a flight supervision platform for supervising and interacting with a plurality of autonomous aircraft according to various embodiments;

[0012] Figure 2B Illustrates another view of an exemplary animated GUI of a flight supervision platform according to various embodiments;

[0013] Figure 3 Illustrates an exemplary technique for allocating routes to autonomous aircraft using an exemplary animated GUI of a flight supervision platform according to various embodiments;

[0014] Figure 4A Illustrates adding a layer to the displayed exemplary animated GUI of a flight supervision platform according to various embodiments;

[0015] Figure 4B Illustrates removing a layer from the displayed animated GUI of a flight supervision platform according to various embodiments;

[0016] Figure 5 Illustrates tracking an autonomous aircraft during flight using an exemplary animated GUI of a flight supervision platform according to various embodiments;

[0017] Figure 6 Illustrates a magnified view of an exemplary animated GUI of a flight supervision platform according to various embodiments;

[0018] Figure 7 Illustrates another view of an exemplary animated GUI including a flight guidance pane according to various embodiments;

[0019] Figure 8 Illustrates another view of an exemplary animated GUI after receiving a task assignment according to various embodiments;

[0020] Figure 9 Illustrates an exemplary warning displayed using an exemplary animated GUI of a flight supervision platform according to various embodiments;

[0021] Figure 10 Illustrates an exemplary command input device according to various embodiments;

[0022] Figure 11 Illustrates an exemplary command input UI according to various embodiments;

[0023] Figure 12Illustrates an exemplary representation of multiple aircraft and associated data on a GUI according to various embodiments; and

[0024] Figure 13 Is a flowchart of an example process performed by a flight supervision platform according to various embodiments. Detailed Description

[0025] The techniques disclosed herein generally relate to an animated graphical user interface for simultaneously monitoring multiple autonomous aircraft. More specifically, the techniques disclosed herein provide a platform (e.g., a flight supervision platform) that allows one-to-many supervision, where a remote supervisor (e.g., a human or an artificial intelligence) can continuously monitor and / or interact with multiple autonomous aircraft. The interaction can include sending commands to the autonomous aircraft using GUI components of the platform. For example, the autonomous aircraft can include one or more autonomous electric vertical takeoff and landing (eVTOL) aircraft. Various inventive embodiments are described herein, including methods, processes, systems, devices, etc.

[0026] The technology also provides communication with a computer-implemented remote supervisor that simultaneously supervises the functionality of multiple autonomous aircraft. The platform is also configured to provide control inputs to the computer-implemented remote supervisor for simultaneously monitoring, supervising, and / or communicating (e.g., interacting) with multiple autonomous aircraft. Embodiments allow for providing quick, intentional inputs. Embodiments provide pop-up menus that optimize the use of screen real estate. Embodiments isolate commands into one category at a time and allow the remote supervisor to keep their eyes on a single point (e.g., without having to navigate to different parts of the platform or display).

[0027] Embodiments provide an input device (e.g., a three-button mouse) for providing commands to an aircraft. The input device may include dedicated mechanisms (e.g., buttons, wheels, touchpads) associated with one or more flight parameters (e.g., heading, true airspeed, altitude). In some embodiments, sensory cues (e.g., visual cues such as color or tactile cues such as a hash pattern that can be felt by touch and seen on a screen) may be used to encode the mechanisms. For example, the same color may be provided on the input mechanism and used to display a menu associated with the flight parameter assigned to that mechanism. For example, a button for providing a heading input may be colored orange on the input device, and a menu for providing a heading command may be displayed in orange on the GUI. Similarly, the heading information associated with the autonomous aircraft may also be displayed in orange on the GUI. Activating a mechanism on the input device (e.g., clicking a button) may cause a control menu to be displayed on the GUI. The user can then use the control menu to provide the desired commands. In some embodiments, the user may not be required to provide any alphanumeric input to provide flight commands to the autonomous aircraft. The input device may be used to send commands through a series of selections from preset commands displayed on the control menu.

[0028] Several illustrative embodiments will now be described with reference to the accompanying drawings, which form a part of these illustrative embodiments. The following description provides only the (one or more) embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of the (one or more) embodiments will provide those skilled in the art with an enabling description for implementing one or more embodiments. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present disclosure. In the following description, specific details are set forth for the purpose of explanation in order to provide a thorough understanding of certain inventive embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and description are not intended to be restrictive. The words "example" or "exemplary" are used herein to mean "serving as an example, instance, or illustration". Any embodiment or design described herein as "exemplary" or "example" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0029] An embodiment provides a flight supervision platform that includes a server computer and an animated graphical user interface (GUI) to supervise, monitor, and / or communicate with one or more autonomous vehicles. For example, a human operator may be able to use the flight supervision platform to supervise multiple autonomous vehicles simultaneously, and the flight supervision platform includes an animated GUI that is displayed on a display device on the ground (e.g., a monitoring center remote from the autonomous vehicle). A supervisor can transmit commands to change one or more of the flight parameters (e.g., altitude, attitude, airspeed, heading) of the autonomous vehicle.

[0030] According to various embodiments, the autonomous vehicle is configured to implement, among other technologies, self-flight technology that implements an automatic piloting function, precise navigation, detection, and avoidance capabilities. The flight supervision platform described herein provides surveillance of the autonomous vehicle and provides the ability to intervene when needed. The flight supervision platform (e.g., a multi-vehicle supervision platform) can monitor the flight from start to finish. During the flight, if the autonomous vehicle takes steps to avoid potential hazards outside the assigned flight plan, the flight supervision platform receives a notification or alert, which is then displayed on the GUI. The flight supervision platform allows a supervisor (e.g., a human supervisor) to intervene at any point during the flight by sending new commands for the autonomous vehicle to execute. For example, the supervisor can use the flight supervision platform to send a command to the vehicle to redirect it to an alternative landing point.

[0031] Figure 1 FIG. shows a block diagram of a flight supervision platform 1000 that a supervisor 1030 (e.g., a human supervisor 1030) can use to individually and / or jointly monitor one or more autonomous vehicles 1040 and interact with the one or more autonomous vehicles 1040. According to various embodiments, the supervisor 1030 can use an input device 1050 to interact with the flight supervision platform 1000. For example, the supervisor 1030 can select an autonomous vehicle and / or use the input device 1050 to provide commands to the selected autonomous vehicle. Details of the input device 1050 are described in more detail below in conjunction with Figures 10 - 13 the details of the input device 1050 are described in more detail.

[0032] The flight supervision platform 1000 may include a server computer 1020, which includes one or more processors 1004, a system memory 1002 (which may include any combination of volatile and / or non-volatile memories, such as, for example, buffer memory, RAM, DRAM, ROM, flash memory, or any other suitable memory device), and a network interface (e.g., an external communication interface) 1006. In some embodiments, one or more of the modules may be provided within one or more of the components of the system memory 1002, or may be provided externally. Figure 1 The software and hardware modules shown are provided for illustrative purposes only, and the configuration is not intended to be restrictive. The processor 1004, the system memory 1002, and / or the external communication interface 1006 may implement the techniques and / or methods described herein.

[0033] The network interface 1006 may be configured or programmed to receive and generate electronic messages that include information transmitted to or from a plurality of autonomous vehicles 1040 via the flight supervision platform 1000.

[0034] The flight supervision platform 1000 may also include at least one display device 1010 for displaying a graphical user interface (GUI) 1012. When an electronic message is received by the flight supervision platform 1000 via the external communication interface 1006 of the server computer 1020, the electronic message may be processed, and the relevant information may be displayed on the display device 1010 via the GUI 1012. When an input is received from the supervisor 1030 via the GUI 1012, the input may be processed, and the relevant information may be transmitted to the corresponding autonomous vehicle 1040. According to various embodiments, the flight supervision platform 1000 may also be configured to receive supplementary information from a third party, such as air traffic control, weather, other vehicles (e.g., vehicles monitoring one or more autonomous vehicles 1040 in the air). The supplementary information may be processed by the flight supervision platform 1000 and displayed on the GUI 1012.

[0035] Figure 2A An exemplary animated graphical user interface (GUI) 100 of a flight supervision platform 1000 for supervising and interacting with a plurality of autonomous vehicles according to various embodiments is illustrated. In the exemplary GUI 100, a supervisor (e.g., Figure 1The supervisor (1030) illustrated in the figure is monitoring five autonomous aircraft, which are illustrated using five aircraft icons on the GUI 100. The GUI 100 may include a first pane 102 that illustrates each autonomous aircraft, where the corresponding aircraft icons 116, 118 are overlaid on a map of the terrain on which the autonomous aircraft are flying, hovering, and / or waiting on the ground for standby. Human factors can be used to determine the amount of detail to include on the map, and these human factors include, for example, an assessment of how much information a human can safely process on a screen. Some embodiments may use artificial intelligence (AI) as the supervisor, or a combination of AI and a human supervisor. The amount of detail to include on the map may vary depending on the capabilities of the supervising entity. In some embodiments, the supervisor

[0036] The GUI 100 may also include a second pane 104 that includes sections 106, 108 corresponding to each of the aircraft icons 116, 118 illustrated in the first pane 102. That is, each section 106, 108 provides information associated with a different autonomous aircraft under the supervision of the supervisor. In some embodiments, when the supervisor selects an autonomous aircraft by, for example, clicking on the desired aircraft icon 116, a visual cue may associate the selected aircraft icon 116 displayed on the first pane 102 with the corresponding information section 106 displayed on the second pane 104. For example, the selected aircraft icon 116 and the corresponding information section 106 may be displayed in a color, shade, and / or font different from the rest of the graphical elements of the GUI. The selected aircraft icon 116 and the information section 106 represent the same autonomous aircraft (e.g., Figure 1 the autonomous aircraft 1040) illustrated in the figure. According to various embodiments, the GUI 100 may display only the active tasks (e.g., tasks with an active status) supervised by the supervisor. There may be additional tasks assigned to the supervisor (e.g., tasks that are not yet active or tasks in a ready-to-fly state), and after the supervisor accepts the tasks or after the tasks assume an active status, their information may be displayed on the appropriate panes 102, 104 of the GUI 100.

[0037] In some embodiments, selecting the aircraft icon 116 on the first pane 102 may cause the corresponding information section 106 to be automatically selected on the second pane 104. Alternatively, selecting the corresponding information section 106 on the second pane 104 may cause the aircraft icon to be automatically selected on the first pane 102. When coupled to the GUI, a server computer (e.g., Figure 1When the server computer 1020 shown in the figure receives a selection of the aircraft icon 116 among the multiple icons on the first pane 102 of the GUI 100 or a selection of the information section 106 among the multiple sections on the second pane 104 of the GUI 100, the server computer identifies the aircraft icon 116 or another one in the first section 106 on the GUI 100. Then, one or more visual cues are used to display the aircraft icon 116 and the information section 106 to distinguish the aircraft icon 116 and the information section 106 from the remaining graphical elements.

[0038] Displaying the aircraft icon and the information section at separate locations on the GUI enables accurate identification and selection of the intended autonomous aircraft represented by the aircraft icon and the information section. For example, if two autonomous aircraft are flying one above the other, their representations on the map (e.g., the aircraft icons) may be displayed overlapping each other. Thus, it may be impossible or difficult for a supervisor to obtain, for example, using the first pane 102 to select one of the overlapping autonomous aircraft. In such a case, the supervisor can use the corresponding section on the second pane 104 to select the autonomous aircraft they want to interact with. In this way, the selection of the intended aircraft is ensured.

[0039] As shown in the exemplary embodiment illustrated in the accompanying drawings, the first pane 102 is allocated more space on the screen than the second pane 104. This allocation of space allows the aircraft icon to be displayed on a larger map (e.g., while also being able to display associated information on a separate section of the screen (e.g., the second pane 104)) without cluttering the display on the map. The relative positioning and size of the first pane 102 and the second pane 104 can vary according to various embodiments.

[0040] The flight supervision platform 1000 allows the supervisor to interact with all autonomous aircraft illustrated thereon with the corresponding aircraft icons 116, 118 or information sections 106, 108 through the GUI 100. Before takeoff, the autonomous aircraft can autonomously perform a pre - flight review, including checking one or more of the power system, control surfaces, avionics, passenger equipment, cabin systems, and / or the environment around the autonomous aircraft. The autonomous aircraft can also continuously monitor these systems and / or items during flight. When the inspection is completed and the autonomous aircraft and the passengers (where applicable) are ready to take off, the autonomous aircraft can notify (e.g., transmit data and / or messages to) the flight supervision platform 1000 (and thus notify the supervisor 1030).

[0041] Upon receiving a notification that the autonomous aircraft is ready to take off and / or a route has been assigned to the autonomous aircraft (discussed below in connection with Figure 3After further discussion), the supervisor can first select the aircraft icon 116 corresponding to the autonomous aircraft on the first pane 102, and send a takeoff command to the autonomous aircraft by, for example, selecting the first (e.g., takeoff) button 110 on the second pane 104 (e.g., as shown in Figure 2A and shown, using a visual cue such as bold font to display the selected takeoff button 110 to indicate that the button has been selected) to indicate that the button has been selected), and then select the second (e.g., execute) button 112 on the second pane 104 (e.g., as shown in Figure 2B and shown, using a visual cue such as bold font to display the selected execute button 112 to indicate that the button has been selected) to confirm that the takeoff command has been sent to the selected aircraft. The two-step selection of including selecting two separate buttons to send a command to the autonomous aircraft can ensure that the command is sent intentionally. Thus, the server computer 1020 can receive the selection of the aircraft icon 116 on the first pane 102 of the GUI 100, and can receive the takeoff command from the second pane 104 of the GUI 100, followed by the execute command. Then, the server computer 1020 can transmit the takeoff command to the autonomous aircraft represented by the aircraft icon 116 on the GUI 100.

[0042] In Figure 2A and Figure 2B In the illustrated embodiments, when the supervisor selects an icon on the first pane of the GUI or an information section on the second pane of the GUI, no additional pop-up window is displayed. This eliminates clutter on the GUI and reduces interference to the supervisor. In some embodiments, the pop-up window can be blocked or the layout of the GUI can be changed to reduce or eliminate interference to the supervisor.

[0043] In some embodiments, when the autonomous aircraft follows the assigned route, the supervisor can receive instructions from air traffic control (ATC) that need to be relayed to the supervised autonomous aircraft. Optimize the display of the GUI for the supervisor to quickly identify the correct autonomous aircraft and relay the ATC communication to the correct autonomous aircraft.

[0044] In some embodiments, the supervisor can locate the autonomous aircraft on the map displayed on the first pane 102 and determine whether the ATC communication is applicable. For example, if the map shows the autonomous aircraft in mountainous terrain that is unsafe for the maneuver proposed in the ATC communication, the supervisor can confirm the maneuver with the ATC before relaying the maneuver to the autonomous aircraft, or can use the map displayed on the GUI 100 to determine based on the information provided to the supervisor that the communication is incorrect and should be discarded.

[0045] Return to reference Figure 2A, when four aircraft in the fleet may already be following a route in the air, a fifth aircraft represented by aircraft icon 116 may be waiting on the ground for route assignment. According to various embodiments, a supervisor may transmit instructions to the aircraft, such as assigning a route, giving a takeoff instruction, etc. The autonomous aircraft executes a flight plan to follow the route upon receiving the route assignment.

[0046] Figure 3 Illustrated is an exemplary technique for assigning a route to an autonomous aircraft using an exemplary animated GUI of a flight supervision platform according to various embodiments. GUI 100 may also be used to assign a flight path to an autonomous aircraft. For example, the autonomous aircraft may be waiting on the ground for flight assignment. The supervisor may select the autonomous aircraft (as shown in Figure 2A and 2B ) by selecting the aircraft icon 116 or the information section 106 representing the aircraft on the ground. Selection of the autonomous aircraft on the ground may trigger GUI 100 to display a list of available routes 202 using menu 200. Those of ordinary skill in the art will appreciate that the graphical elements shown in the figures for providing information or receiving input from the supervisor are not limited to the means shown in the figures, and alternative means may be used to convey the same information or receive the same input. For example, menu 200 may include a scroll-down menu, a drop-down menu, etc. The supervisor may select one of the routes by, for example, clicking on the desired route in the list of available routes 202. The supervisor may be required to confirm the selection by, for example, selecting the accept button 204. The accept button 204 for confirming the selection may be displayed on the second pane 104, while the menu 200 with the routes 202 may be displayed on the first pane 102. Thus, the confirmation action may require the supervisor to travel from the first pane 102 to the second pane 104 to intentionally select the accept button 204. This configuration may avoid accidentally executing the selection.

[0047] When route 206 is selected for assignment to the aircraft, a visual cue, such as using a first color, can be used to display the selected route 206 on the map as the displayed route 210 from the departure point to the arrival point. This will also allow the supervisor to visually confirm that the intended route is assigned to the autonomous aircraft. The departure and arrival points 208 of the selected route can be displayed on the information section corresponding to the selected autonomous aircraft. For example, if the intention is to assign a route from Los Angeles to San Francisco, but the displayed route is from San Diego to Los Angeles, the supervisor can be aware of the error before executing the route assignment command. Once the supervisor confirms that the displayed route 210 is the intended route, the supervisor can execute the assignment by selecting the accept button 204. Execution of the command can result in the displayed assigned route being shown on the map from the departure point to the arrival point using a different visual cue (such as using a second color) to indicate that the route is now the confirmed route. That is, the GUI 100 relays the transition from the selected route to the confirmed route.

[0048] The server computer 1020 coupled to the GUI 100 can continuously receive the flight parameters and position location signals of multiple autonomous aircraft monitored using the GUI 100. As the autonomous aircraft progresses along the displayed route 210, the server computer can use the flight data and position location signals of the autonomous aircraft to periodically update the GUI 100 to display the relative position of the first icon 116 with respect to the displayed route 210 assigned to the autonomous aircraft.

[0049] According to various embodiments, the supervisor can select the amount of detail to be shown on the map. For example, the GUI 100 can have information layers 250 that can be displayed in an overlapping manner. The supervisor can select one or more of the available layers (e.g., radius information 252 around the aircraft, wind information 254, telemetry information 256 received from one or more aircraft, any additional information 258) to have the desired level of detail on the map.

[0050] Figures 4A - 4B Illustrates various levels of detail displayed on the GUI of the flight supervision platform according to various embodiments. Specifically, Figure 4A Illustrates adding a layer to the displayed exemplary animated GUI, and Figure 4B Illustrates removing a layer from the displayed animated GUI. Figure 4AThe visual aid 302 is shown for illustrating the environment of the autonomous vehicle represented by the vehicle icon 116. For example, the visual aid 302 can be in the form of concentric rings, with each ring corresponding to a predetermined distance from the autonomous vehicle. In this way, the supervisor can quickly see and evaluate whether there are other vehicles or hazards in the immediate vicinity of the monitored autonomous vehicle. Similarly, the visual aid 302 can also convey information about the distance between the monitored autonomous vehicle and geographical elements (such as mountains) near the monitored autonomous vehicle.

[0051] If the supervisor prefers a cleaner display, the supervisor can remove the visual aid 302 from the GUI, as Figure 4B shown. For example, when the radius layer 252 ( Figure 3 shown) is selected, the visual aid 302 can be displayed, and when the radius layer 252 is not selected on the GUI 100, the visual aid 302 can be removed. Although some of the elements in the GUI may be subject to masking (for example, some elements are always displayed and may not be removed by toggling layers), other elements can be displayed according to the selected options provided on the GUI (for example, based on Figure 3 the selection of the map layer 250 illustrated).

[0052] As shown in FIG. 4- Figure 5 the vehicle icons 116, 118 can have additional information 400 displayed in association with the vehicle icons on the map. For example, flight parameters (such as heading, altitude, and / or speed information) associated with the autonomous vehicles represented by the respective vehicle icons 116, 118 can be displayed next to the vehicle icons 116, 118. In some embodiments, the information 400 can be stuck to the vehicle icon, and the information 400 can move along with the vehicle icon along the displayed path.

[0053] As Figure 5 shown, the animated GUI can display the vehicle icon 116 moving along the route 120 assigned to the vehicle. When the autonomous vehicle advances along the assigned route (e.g., flight path), the corresponding vehicle icon 116 moves along the route 120 on the map. The displayed information 400 moves along with the vehicle icon 116. By having all the information on the same GUI, the techniques provided herein ensure that the supervisor evaluates all relevant information associated with the autonomous vehicle and selects the correct autonomous vehicle when activating a route or when communicating a command to the autonomous vehicle. In some embodiments, the supervisor can switch between displaying and hiding the displayed information 400. That is, the flight parameters associated with the autonomous vehicle can be selectively displayed near the first icon representing the autonomous vehicle on the GUI.

[0054] According to various embodiments, an autonomous aircraft supervised by an animated GUI 100 flies autonomously using instrument flight rules (IFR). If the ATC instructs the supervisor to transmit new parameters (e.g., altitude, heading, speed, etc.) to the autonomous aircraft, the supervisor can do so by selecting the aircraft icon 116 and sending a command with the new parameters. In some embodiments, the autonomous aircraft can utilize messages transmitted back to the server computer and displayed on the GUI 100 to confirm the receipt and / or execution of the command.

[0055] In some embodiments, the GUI can be displayed at various zoom levels as selected by the supervisor or based on the selected action. For example, when assigning a flight path to the autonomous aircraft, the GUI can be displayed at a zoom level that shows both the starting and ending points of the route. For example, when entering a flight parameter change command, the GUI can be displayed at a zoom level that better illustrates the surrounding environment of the autonomous aircraft to ensure that the flight parameter modification does not pose an imminent threat to the autonomous aircraft.

[0056] Figure 6 An enlarged view of an exemplary animated GUI of a flight supervision platform according to various embodiments is illustrated. As Figure 6 shown, a route can be formed by connecting multiple waypoints 552, 554, 556. In some embodiments, when the autonomous aircraft has flown over a segment 560 (e.g., portion) between waypoints, the segment can change color. For example, in the Figure 6 exemplary embodiment illustrated, the segment 560 (e.g., the completed portion of the route) is shown in a first color (e.g., white), indicating that the segment 560 of the route has been completed, while the segment 562 (e.g., the remaining portion of the route) is shown in a second color (e.g., purple), indicating that the segment has not been completed. These visual cues can help the supervisor collect various types of information simply by looking at or observing the GUI. Using the visual cues, the supervisor can track multiple autonomous aircraft under their supervision.

[0057] According to various embodiments, when the aircraft icon 116 is selected on the GUI, the supervisor can select a waypoint on the planned route of the aircraft represented by the aircraft icon 116 and route the aircraft directly to the selected waypoint. For example, if the aircraft is configured to fly from waypoint A to waypoint B and then to waypoint C, the supervisor can select waypoint C and instruct the aircraft to fly directly from waypoint A to waypoint C, skipping waypoint B. Like the various commands described herein, one or more selection inputs on the GUI are transmitted by the server computer coupled to the GUI to the autonomous aircraft.

[0058] When the autonomous vehicle has completed its flight plan (e.g., reached the destination location), the vehicle icon 116 may show one or more visual cues that indicate that the autonomous vehicle has completed its flight and has landed.

[0059] Figure 7 Another view of an exemplary animated GUI including a flight guidance pane is illustrated in accordance with various embodiments. Figure 7 The exemplary GUI 600 shown in includes a first pane 602 having a plurality of icons 601, 603 (similar to Figure 2A the first pane 102), each icon representing an autonomous vehicle monitored using the GUI 600. The exemplary GUI 600 also includes a second pane 604 (similar to Figure 2A the second pane 104), which has a plurality of information sections, each information section corresponding to one of the icons displayed on the first pane 602. The first pane 602 and the second pane 604 may illustrate tasks (e.g., flights) that are active (e.g., have an active status or are executing a flight plan). The exemplary GUI 600 also includes a third pane 606 that displays a task list 650 assigned to the supervisor of the GUI 600. Each task assignment represents an additional autonomous flight that is being assigned to the supervisor for monitoring by the GUI. That is, the third pane 606 illustrates autonomous vehicles that are being assigned but are not currently being supervised using the GUI 600. For example, the task list 650 may include tasks waiting for a supervisor (e.g., an operator) to accept responsibility for their execution. Once the supervisor accepts and assigns the tasks, the final preparations before departure can be completed. In some embodiments, the tasks on the list 650 may not include all assigned tasks, but only those in a pre-preparation state, as described above. Accepting assigned tasks (e.g., autonomous flights) is described below in connection with Figure 8 Accepting an assigned task (e.g., an autonomous flight) is described below in connection with

[0060] According to various embodiments, the flight supervision platform 1000 may display a message reporting one or more upcoming tasks that may require approval or acceptance by the supervisor 1030. For example, an upcoming task reminder may be displayed in a prioritized manner within a predetermined amount of time along with the information necessary to avoid human error or human performance degradation (e.g., delays in response to time-critical and / or safety-critical events). In some embodiments, the reminder message may have a countdown displayed therein to indicate the remaining time for the supervisor 1030 to act on the reminder.

[0061] Continuing with the description of the exemplary GUI 600, a fourth pane 608 is provided to illustrate a zoomed - in map view of the selected autonomous vehicle. For example, if an icon 601 is selected on the first pane 602, the corresponding information section 605 is identified on the second pane 604 using a visual cue (as described above in connection with Figure 2A and Figure 2B ), and the immediate area around the selected autonomous vehicle is shown in pane 608 using an enlarged map. The exemplary GUI 600 also includes a flight guidance pane 610. The GUI 600 can receive modifications on the flight guidance pane 610 that correspond to desired changes in one or more flight parameters associated with the selected autonomous vehicle (e.g., the vehicle represented by 601 and information section 605 in the exemplary GUI of Figure 7 ). The server computer transmits commands to the autonomous vehicle to effect the desired changes. According to various embodiments, the flight guidance pane 610 can include one or more of a control dial 612 or 614, a drop - down menu, a text entry field, or a widget 620.

[0062] For example, the first control dial 612 can enable a supervisor to enter or modify the heading setting of the selected autonomous vehicle. The GUI can display possible headings in a predetermined increment. When the supervisor makes a selection, a visual cue 618 can be used to indicate the selected heading. The second control dial 614 can enable a supervisor to enter or modify the altitude setting of the selected autonomous vehicle. The GUI can display possible altitudes in a predetermined increment. When the supervisor makes a selection, a visual cue 616 can be used to indicate the selected altitude. In some embodiments, flight parameter settings or modifications can be entered in a text input field. When one or more of the flight parameter settings or modifications are complete, the supervisor can select the widget 620 (e.g., an "Execute" widget) to cause the server computer to transmit commands to the autonomous vehicle to effect one or more of the flight parameter settings or modifications.

[0063] Figure 8 Another view of an exemplary animated GUI of a flight supervision platform after receiving a task assignment is illustrated according to various embodiments. As described above, Figure 7 the exemplary GUI 600 shown includes a third pane 606 that displays a task list 650 assigned to the supervisor of the GUI 600. When a supervisor accepts one of the tasks, the task moves from the third pane to the second pane, which displays a list of active autonomous flights being supervised by the supervisor. As Figure 8As shown, the supervisor receives task 702, which is removed from the third pane 705 and moved to the second pane 740. The second pane 740 displays all autonomous vehicles supervised and / or managed by the supervisor, regardless of whether the autonomous vehicle is in flight or on the ground (e.g., waiting to start a task or having recently landed as part of its task).

[0064] When the supervisor accepts a new task, the flight guidance pane 610 temporarily switches to the configuration view 710. The configuration view 710 provides additional information associated with the newly accepted task, including, for example, the starting point and destination location, the estimated departure time, the charging status of the vehicle, the flight inspection status, etc. The configuration view 710 may include a text area 704 for displaying additional information associated with the newly accepted task, and one or more command window widgets 708. When the takeoff command is selected on the GUI, the server computer sends a takeoff command to the autonomous vehicle, and the configuration view 710 switches back to the flight guidance pane 610, as Figure 7 shown.

[0065] Autonomous flights managed by the supervisor can be displayed using icons on the first pane 730 of the GUI. According to various embodiments, the icon 732 representing an autonomous vehicle in flight may have a different representation from the icon 734 representing an autonomous vehicle on the ground (e.g., an autonomous vehicle ready to fly or having recently landed). The icon representing the newly accepted task may not yet be associated with a flight plan (e.g., a route) on the map. When the execute takeoff command is transmitted by the server computer to the autonomous vehicle, the icon can be associated with the flight plan or route on the GUI.

[0066] As Figure 8 shown, the magnified map pane 720 can provide a close-up bird's-eye view around the newly accepted task. The map pane 720 can also indicate the current status of the autonomous vehicle. For example, since the selected autonomous vehicle corresponds to the newly accepted task 702, the status of the autonomous vehicle is indicated as "Transition Level 1, Transition - Climb", meaning the autonomous vehicle is transitioning to the vertical flight mode to take off or climb.

[0067] According to various embodiments, the server computer coupled to the GUI can receive an input for selecting a task assignment displayed in the selection pane 705. For each selected task assignment, a separate icon 734 is created in the first pane 730, and a corresponding separate section 702 representing information associated with the selected task assignment is created in the second pane 740. Then, the accepted task is removed from the third pane 705 of the GUI.

[0068] According to various embodiments, an entity (e.g., an ATC, a third party, another aircraft) may send an alert to a supervisor. In some embodiments, the alert may be received at a flight supervision platform and / or at a server computer communicating with the GUI, and a visual alert may be displayed on the GUI. The supervisor may choose to ignore or take action on the displayed alert. In some embodiments, the GUI may also display proposed changes or modifications to the flight parameters of the monitored autonomous aircraft. The supervisor may accept the proposed changes and transmit a command to change the flight parameters to the autonomous aircraft. In some embodiments, in response to an alert, the supervisor may use the flight guidance pane of the GUI to manually enter desired changes to the flight parameters. The server computer may then transmit a command to the autonomous aircraft using the GUI.

[0069] In some embodiments, the autonomous aircraft is configured to change its route to avoid a collision with a hazard or to avoid mission failure due to, for example, an on-board electromechanical problem. The autonomous aircraft may change its route without receiving approval from the supervisor. In some embodiments, the autonomous aircraft may be configured to wait for a predetermined amount of time for approval before implementing a change to the flight plan.

[0070] According to various embodiments, a dedicated area of the GUI may be reserved for displaying alert messages. For example, feedback or data indicating an aircraft problem received from the sensors or other devices of the autonomous aircraft may be displayed as an alert in the dedicated area. In some embodiments, depending on the importance of the alert, the alert may be displayed as a pop-up screen on the main pane (e.g., the first pane) of the GUI.

[0071] Figure 9 Exemplary warnings using an exemplary animated GUI display are illustrated. In some embodiments, the autonomous aircraft may detect a hazard 800 on its route 802 and may deviate from its route 802 to avoid a collision with the hazard 800. For example, the autonomous aircraft may turn left or right and change its route to an alternative route 804 that avoids a collision with the hazard 800. In some embodiments, due to a problem identified by the autonomous aircraft, such as a propeller (e.g., a fan) failure, the autonomous aircraft may need to change its route. The alternative route 804 and a collision avoidance alert (e.g., a warning) 806 are displayed on the GUI to notify the supervisor. In some embodiments, after transmitting the hazard 800 to the flight supervision platform, the autonomous aircraft may request a trajectory update. The request for a trajectory update may be displayed on the GUI. The supervisor may provide a response to the request for a trajectory update by sending a change to the flight parameters or flight plan of the autonomous aircraft via the server computer.

[0072] When warning 806 is displayed on the GUI, the GUI may switch the selected aircraft icon to the aircraft icon associated with warning 806. For example, if a warning is received from the autonomous aircraft represented by icon 805 and section 822, or the warning is otherwise associated with that autonomous aircraft, the GUI may display icon 805 and the corresponding section 822 as the selected autonomous aircraft. In some embodiments, if an alert is displayed for a change that has been implemented by the autonomous aircraft, the alert may be displayed for a predetermined amount of time before the supervisor switches back to the last selected aircraft.

[0073] In some embodiments, ATC may suggest a flight plan change, which may be displayed as an alert on the GUI. The alert may also include widgets displayed on the GUI for the supervisor to accept or reject the suggested flight plan change. If the supervisor accepts the suggested flight plan change, selecting the widget that accepts the suggested flight plan change may cause the server computer to transmit a command to the autonomous aircraft to implement the suggested flight plan change (e.g., change course as indicated in the suggested flight plan change). The autonomous aircraft executes the received command and begins following the changed flight plan.

[0074] Embodiments provide a flight supervision platform (e.g., Figure 1 flight supervision platform 1000 illustrated in) to perform the techniques described herein. The server computer (e.g., server computer 1020) analyzes and / or processes information received from the autonomous aircraft being monitored or any other source and transmits it to the GUI (e.g., GUI 1012) for display. Similarly, any input or selection received at the GUI is analyzed and / or processed by the server computer and transmitted to the intended autonomous aircraft via a secure communication channel between the server computer and the autonomous aircraft.

[0075] Figure 10Illustrates an exemplary input device according to various embodiments. The exemplary input device 1050 can be communicatively coupled to a server computer 1020. The input device 1050 can communicate with the server computer 1020 via a wired or wireless connection. The input device 1050 can include a plurality of input mechanisms 1052, 1054, 1056 associated with unique flight parameters. The input device 1050 can be controlled by a supervisor 1030 and can send a signal based on the selection (e.g., activation) of one or more of the input mechanisms 1052, 1054, 1056. Exemplary input mechanisms can include, but are not limited to, any combination of (one or more) buttons, (one or more) switches, (one or more) rollers, (one or more) touchpads, (one or more) touch sensors. Each input mechanism 1052, 1054, 1056 can be assigned to a unique flight parameter (e.g., heading, true airspeed, altitude). Selecting (or activating) the input mechanisms 1052, 1054, 1056 sends a signal from the input device 1050 to the server computer 1020.

[0076] For example, the input device 1050 can include a multi-button mouse (e.g., a three-button mouse). The first input mechanism (e.g., the first button, switch, touchpad, pointer) 1052 can be dedicated to commands associated with a first flight parameter (e.g., heading), the second input mechanism (e.g., the second button, switch, touchpad, pointer) 1054 can be dedicated to commands associated with a second flight parameter (e.g., airspeed, true airspeed), and the third input mechanism (e.g., the third button, switch, touchpad, pointer) 1056 can be dedicated to commands associated with a third flight parameter (e.g., altitude) of the aircraft. According to various embodiments, the input mechanisms can be of the same type (e.g., all input mechanisms are buttons or rollers, etc.). Alternatively, each input mechanism can be of a different type (e.g., the first input mechanism associated with the first parameter can be a button, the second input mechanism associated with the second parameter can be a roller, etc.). In some embodiments, the input device 1050 can optionally include one or more additional input mechanisms. For example, the input device 1050 can include optional side buttons 1058 to provide additional input (e.g., input other than selecting or modifying flight parameters).

[0077] In some embodiments, each input mechanism can be associated with a sensory cue. For example, each input mechanism can be color-coded. The same color can be used to illustrate commands associated with flight parameters on the GUI 100. In other embodiments, each input mechanism can have a unique tactile texture. The same texture (e.g., hashed, dot texture in a first direction) can be used to illustrate flight parameters on the GUI 100.

[0078] One of ordinary skill in the art will appreciate that an input device including multiple input mechanisms is not limited to Figure 10 the structure illustrated in

[0079] and can take any shape or form, as the input device includes dedicated input mechanisms for each unique parameter that a supervisor can vary and / or control. Figure 1 According to various embodiments, supervisor 1030 may hover over an icon (e.g.,

[0080] the icon 116 illustrated in

[0081] that represents an actual aircraft displayed on GUI 100). Server computer 1020 may detect that the cursor has hovered over the icon for a predetermined amount of time. When the predetermined amount of time has elapsed, server computer 1020 may interpret the hover as an input to select the icon among the multiple icons displayed on the GUI. Figure 11 Using input device 1050 and the techniques described herein, supervisor 1030 may send commands to the selected aircraft to change flight parameters (e.g., airspeed, heading, or altitude). For example, supervisor 1030 may send a command to the aircraft to slow down, have a new heading, or assume a new altitude.

[0082] According to an exemplary embodiment, a cursor controlled via input device 1050 may be displayed on GUI 100. An exemplary cursor 1108 is illustrated in Figure 11As shown in. For example, activating the first input mechanism 1052 can trigger the display of the first radial menu 1102 on the GUI, activating the second input mechanism 1054 can trigger the display of the second radial menu 1104 on the GUI, and activating the third input mechanism 1056 can trigger the display of the third radial menu 1106 on the GUI.

[0083] The menus 1102, 1104, 1106 can allow the supervisor 1030 to modify the flight of the aircraft. Each menu 1102, 1104, 1106 can be associated with one or more preset commands to modify one or more flight parameters of the selected aircraft. Each menu 1102, 1104, 1106 can include a plurality of preset commands 1106. According to various embodiments, the menus 1102, 1104, 1106 can be in the form of a rosette (e.g., having a circular format), including a central element and a plurality of preset commands arranged around the central element. Figure 11 The exemplary menus 1102, 1104, and 1106 shown above are radial menus. However, the menus can be provided in other forms that allow the implementation of the features described herein.

[0084] When the menus 1102, 1104, or 1106 are displayed on the GUI, the supervisor 1030 can use the input device 1050 to control the cursor to select the desired command. As Figure 11 shown in, each menu 1102, 1104, or 1106 includes a plurality of preset commands. According to various embodiments, the preset commands can be displayed in a circular format around a central element that identifies the flight parameter. The placement of the preset commands can be based on at least one of the orientation or magnitude of each preset command.

[0085] Return reference Figure 11 , the menu 1102 is associated with the aircraft heading as a flight parameter. According to various embodiments, each input mechanism and the displayed menu pair are associated with the same haptic cue. Here, the central element 1122 identifies the heading as the flight parameter. As Figure 11 shown in, the central element 1122 can use the input mechanism 1052 associated with the same flight parameter (e.g., heading) (as Figure 10The same sensory cues (e.g., visual cues such as color) as those illustrated in are used to represent. The preset commands 1105 are arranged around the central element 1122. Preset commands related to allocating a heading angle to the right are provided on the right side of the menu 1102. Preset commands related to allocating a heading angle to the left are provided on the left side of the menu 1102. The preset commands for the right heading are arranged in a clockwise increasing order. The preset commands for the left heading are arranged in a counterclockwise increasing order. Thus, the placement of the preset commands 1105 is based on at least one of the orientation or magnitude of each preset heading command. The "Continue Flight Plan" preset command 1124 is also presented in the menu 1102. According to various embodiments, the menu 1102 may also include preset commands based on the direction of the desired heading and allow the supervisor 1030 to enter the desired heading vector. For example, the preset command 1126 is a preset command for a right heading that allows the supervisor to enter the desired heading in the preset command 1126. For example, the supervisor 1030 can use the input mechanism of the input device 1050 to provide the desired heading. In some embodiments, a roller or continuous tapping on the selected input mechanism may be used to enter the desired input. Selecting one of the preset commands 1105 via the input element 1050 using the cursor 1108 (e.g., the "360 Degrees Left" command as shown in Figure 11 will send the "360 Degrees Left" command to the selected aircraft. The control computer of the aircraft will then execute the received command.

[0086] The menu 1104 is associated with the true airspeed of the aircraft as a flight parameter. The central element 1132 identifies the true airspeed as the flight parameter. As Figure 11 shown, the central element 1132 can use the same sensory cues (e.g., visual cues such as color) as those associated with the input mechanism 1054 (as illustrated in Figure 10 and related to the same flight parameter (e.g., true airspeed)) to represent. The preset commands 1110 are arranged around the central element 1132. Preset commands related to increasing the true airspeed are provided in the upper half of the menu 1104. Preset commands related to decreasing the true airspeed are provided in the lower half of the menu 1104. The absolute value of the preset commands 1110 increases from left to right. Thus, the placement of the preset commands 1110 is based on at least one of the magnitude and value (or absolute value) of each preset true airspeed command. Selecting one of the preset commands 1116 via the input element 1050 using the cursor 1108 (e.g., the "+5 KTS" command as shown in Figure 11 will send the "Increase Airspeed by 5 Knots" command to the selected aircraft. The control computer of the aircraft will then execute the received command.

[0087] The menu 1106 is associated with the altitude of the aircraft as a flight parameter. The central element 1142 identifies the altitude as the flight parameter. AsFigure 11 As shown in, the central element 1142 can be represented using the same sensory cues (e.g., visual cues such as color) as the input mechanism 1056 associated with the same flight parameters (e.g., true airspeed) as Figure 10 shown in. The preset commands 1112 are arranged around the central element 1142. Preset commands related to increasing altitude are provided in the upper half of the menu 1106. Preset commands related to decreasing altitude are provided in the lower half of the menu 1106. The absolute value of the preset commands 1110 increases from left to right. Thus, the placement of the preset commands 1110 is based on at least one of the magnitude and value (or absolute value) of each preset altitude command. Selecting one of the preset commands 1116 (e.g., the "+100 feet" command as Figure 11 shown in) via the input element 1050 using the cursor 1108 will send a "Increase altitude by 100 feet" command to the selected aircraft. The control computer of the aircraft will then execute the received command.

[0088] One of ordinary skill in the art will appreciate that the arrangement of the preset commands is not limited to Figure 11 the layout shown, and any layout can be selected for any of the above menus (e.g., a layout that is intuitive to the supervisor 1030).

[0089] According to various embodiments, once the supervisor 1030 has finished transmitting the desired commands to the aircraft, the menus 1102, 1104, or 1106 disappear from the GUI. This reduces clutter on the GUI 100 and enables the supervisor 1030 to focus on the aircraft under their supervision.

[0090] In some embodiments, when activating the input mechanisms 1052, 1054, or 1056, the supervisor 1030 can press or hold a key (e.g., the "Ctrl" key) provided on a keyboard coupled to the input device 1050. Keys can also be provided on the input device 1050 (e.g., on the surface of the input device 1050, such as the side surface). Simultaneous activation of multiple buttons and / or keys can ensure that the commands sent to the aircraft are intentional and not accidental inputs.

[0091] In some embodiments, the most common commands can be placed on the menus 1102, 1104, 1106 where they are easily accessible to the supervisor 1030. In some embodiments, the central elements 1122, 1132, 1142 indicating the selected parameters (e.g., heading, airspeed, altitude) can be located at the center of the aircraft icon that the supervisor 1030 has selected. Thus, the supervisor 1030 does not move the cursor 1108 away from the aircraft icon, thereby increasing the visual feedback that the command will be sent to the selected aircraft represented by the aircraft icon.

[0092] Figure 12 Illustrates another exemplary display of multiple aircraft simultaneously controlled by a controller. Each aircraft icon 1202, 1204 may be associated with an autonomous aircraft controlled by the same supervisor 1030. As Figure 12 shown, the aircraft icons 1202, 1204 may be minimalist representations of the aircraft without incorporating detailed design features to further reduce clutter on the GUI. A line with a first visual cue (e.g., a first color) may be used to show the flight path (e.g., the flight route) 1206 of the aircraft. One or more predetermined flight parameters may be displayed in the form of a legend 1208 near the aircraft icon 1202. For example, heading, airspeed, and altitude information may be displayed using colors associated with the input mechanisms 1052, 1054, 1056 of the input device 1050 and menus 1102, 1104, 1106 associated with the respective unique flight parameters.

[0093] Using consistent sensory cues (e.g., using the same color for unique parameters such as a first color for heading, a second color for airspeed, and a third color for altitude) reduces human error (e.g., sending the wrong command or sending a command to the wrong aircraft).

[0094] Commands sent to the aircraft may also be displayed on the GUI. As Figure 12 shown, the legend 1210 indicates the current heading (e.g., heading 315°) and the target parameter (e.g., heading 045°) of the aircraft associated with the icon 1204. The legend 1210 illustrates the old command or current flight parameter, as well as the target flight parameter. That is, the GUI indicates the maneuver that the aircraft is about to perform. Once the target is reached, the old flight parameter disappears and the achieved target (which has now become the current parameter) is displayed. In Figure 12 the illustrated example, once the aircraft represented by the icon 1204 has a heading of 045°, the number "315°" will disappear and the legend will display "HDG 045" (e.g., using the executed command sent to the aircraft by the flight supervision platform 1000 described herein).

[0095] According to various embodiments, supervisor 1030 may wish to display the terrain around the aircraft. The GUI may include an option for supervisor 1030 to turn the terrain display on or off. For example, the GUI may include widget 1212, which can be selected to display or obscure the terrain. In some embodiments, to eliminate clutter in the GUI, the terrain may not be displayed by default. The GUI described herein simultaneously illustrates multiple aircraft. Since each aircraft may be at a different altitude from the rest of the aircraft, permanently displaying the terrain may increase the likelihood of error by the supervisor. However, supervisor 1030 may wish to see the terrain and can use widget 1212 to turn the terrain display on / off. Input device 1050 can be used to control (activate, deactivate) widget 1212. For example, clicking, or clicking and holding, or hovering over widget 1212 can activate the widget and display the terrain on the GUI. Releasing the widget or hovering away, or re-clicking the widget with the input device can remove the terrain from the GUI. For example, additional input mechanism 1058 of input device 1050 (e.g., an input mechanism not assigned to a specific flight parameter) can be used to select widget 1212.

[0096] Figure 13 is a flowchart of example process 1300. In some implementations, Figure 13 one or more of the process blocks can be performed by the flight supervision platform.

[0097] At step 1302, a graphical user interface (GUI) is displayed on a display device using a server computer. In Figure 2A exemplary GUI 100 is illustrated.

[0098] At step 1304, multiple autonomous aircraft are represented on the GUI by multiple icons. For example, referring back to Figure 2A , each of icons 116, 118 represents a different autonomous aircraft on GUI 100. In some embodiments, information regarding one or more flight parameters associated with each of the multiple autonomous aircraft can be displayed near the icon representing the autonomous aircraft on the GUI using a unique haptic cue for each flight parameter. For example, as shown in Figure 12 , each flight parameter can be assigned a color and displayed near the corresponding icon using the assigned color.

[0099] At step 1306, the server computer receives an input to select a first icon among the multiple icons on the GUI. For example, the server computer can receive an input (e.g., a signal transmitted from an input device to the server computer) to select Figure 2AThe icon 116 thereon. In some embodiments, the server computer may detect that the cursor hovers over the first icon for a predetermined amount of time. The cursor is controlled by an input device communicatively coupled to the server computer. When the predetermined amount of time has elapsed, the server computer may interpret the icon under the cursor as being selected.

[0100] At step 1308, the server computer receives a first signal from an input device that includes a plurality of input mechanisms. Each input mechanism is assigned a unique flight parameter. The first signal is generated by selecting a first input mechanism of the input device, wherein a first component is associated with the first flight parameter. Figure 10 An exemplary input device is illustrated therein. The input device 1050 includes input mechanisms 1052, 1054, 1056, which are respectively assigned to control the heading, airspeed, and altitude of the aircraft. Selecting the input mechanism 1052, for example, assigned to the true airspeed generates a first signal and transmits it to the server computer to display a menu associated with the true airspeed. According to various embodiments, the input mechanism of the input device associated with the first flight parameter is identified on the input device using a unique haptic cue assigned to the first flight parameter. For example, if a first color is assigned to the true airspeed, the input mechanism associated with the true airspeed is marked with the first color on the input device.

[0101] At step 1310, in response to the server computer receiving the first signal, a first menu among a plurality of menus is displayed on the GUI. The first menu is associated with a first flight parameter of the autonomous aircraft represented by the first icon. The first menu includes a plurality of first preset commands. For example, the first menu includes Figure 11 One of the illustrated menus 1102, 1104, 1106. According to the input mechanism activated on the input device, a menu associated with the flight parameter assigned to the activated input mechanism is displayed on the GUI. According to various embodiments, the menu may be in a rosette shape, including a central element and a plurality of presentation commands arranged around the central element. In some embodiments, the first preset commands are displayed in a circular format around the central element that identifies the first flight parameter. The placement of the first preset commands may be based on at least one of the orientation or amplitude of each preset command.

[0102] At step 1312, the server computer receives a selection of a first command among the plurality of first preset commands. The supervisor may use the input device to select one of the preset commands of the menu.

[0103] At step 1314, the server computer transmits a first command to the first autonomous vehicle represented by the first icon on the GUI. The control computer of the first autonomous vehicle receives and implements the first command. In some embodiments, information regarding one or more flight parameters associated with each of the plurality of autonomous vehicles may be displayed near the icon representing the autonomous vehicle on the GUI. For example, the information includes the values of one or more current flight parameters, as well as the value of the first flight parameter identified in the first command. For example, in view of the first command, the current value of the first flight parameter and the target value of the flight parameter may be displayed near the first icon.

[0104] The embodiments also provide a system that includes a display screen, one or more processors, and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the steps described herein. The system may also include an input device that includes a plurality of input mechanisms. Each input mechanism is assigned to a unique flight parameter, where the flight parameters include one of heading, airspeed, and altitude.

[0105] In the foregoing specification, embodiments of the present disclosure have been described with reference to numerous specific details, which may vary according to different implementations. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive in nature. The sole and exclusive indicator of the scope of the present disclosure, and what the applicant intends the scope of the present disclosure to be, is the literal and equivalent scope of the set of claims issued from this application, in the specific form in which such claims are issued, including any subsequent amendments. Without departing from the spirit and scope of the embodiments of the present disclosure, the specific details of particular embodiments may be combined in any suitable manner.

[0106] Additionally, spatially relative terms, such as "bottom", "top", or "side", etc., may be used to describe the relationship of an element and / or feature to another element and / or feature(s), as illustrated in the figures. It should be understood that, in addition to the orientation depicted in the figures, spatially relative terms are intended to encompass different orientations of the device in use and / or operation. For example, if the device in the figures is flipped, an element described as the "bottom" surface may be oriented "above" other elements or features. The device may be oriented in other ways (e.g., rotated 90 degrees or other orientations), and the spatially relative descriptors used herein are to be interpreted accordingly.

[0107] The methods, systems, and devices discussed in this document are examples. Various embodiments may appropriately omit, substitute, or add various processes or components. For example, the features described with respect to certain embodiments may be combined in various other embodiments. Different aspects and elements of the embodiments may be combined in a similar manner. Additionally, technology is evolving, and thus, many of the elements are examples and do not limit the scope of the disclosure to those specific examples.

[0108] The terms “and,” “or,” and “one or more” as used herein may include a variety of meanings that are also expected to depend, at least in part, upon the context in which such terms are used. Typically, “or” if used to associate a list, such as A, B, or C, is intended to mean A, B, and C (herein used in the inclusive sense) as well as A, B, or C (herein used in the exclusive sense). In addition, the term “one or more” as used herein may be used to describe any feature, structure, or characteristic in the singular or may be used to describe some combination of features, structures, or characteristics. However, it should be noted that this is merely illustrative and the claimed subject matter is not limited to this example. Further, the term “at least one” if used to associate a list, such as A, B, or C, may be interpreted to mean any combination of A, B, and / or C, such as A, B, C, AB, AC, BC, AA, AAB, ABC, AABBCCC, etc.

[0109] Throughout this specification, references to “one example,” “an example,” “certain examples,” or “exemplary implementations” mean that a particular feature, structure, or characteristic described in connection with the feature and / or example may be included in at least one feature and / or example of the claimed subject matter. Thus, the appearances of the phrases “in one example,” “an example,” “in certain examples,” “in certain implementations,” or other similar phrases throughout this specification are not necessarily all referring to the same feature, example, and / or limitation. Further, the particular features, structures, or characteristics may be combined in one or more examples and / or features.

[0110] In the foregoing detailed description, numerous specific details have been set forth in order to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter may be practiced without these specific details. In other instances, methods and apparatuses known to those of ordinary skill in the art have not been described in detail so as not to obscure the claimed subject matter. Accordingly, it is intended that the claimed subject matter not be limited to the specific examples disclosed, but that such claimed subject matter may also include all aspects within the scope of the appended claims and their equivalents.

Claims

1. A computer-implemented method for monitoring and interacting with a plurality of autonomous vehicles, the method comprising: Displaying a graphical user interface (GUI) on a display device using a server computer; Representing the plurality of autonomous vehicles by the server computer using a plurality of icons displayed on the GUI; Receiving, by the server computer, an input for selecting a first icon among the plurality of icons displayed on the GUI; Receiving, by the server computer, a first signal from an input device including a plurality of input mechanisms, wherein each input mechanism is assigned to a unique flight parameter; In response to receiving the first signal, displaying, by the server computer, a first menu among a plurality of menus on the GUI, wherein the first menu is associated with a first flight parameter of the autonomous vehicle represented by the first icon, and wherein the first menu includes a plurality of first preset commands; Receiving, by the server computer, a selection of a first command among the plurality of first preset commands; And Transmitting, by the server computer, the first command to a first autonomous vehicle represented by the first icon on the GUI.

2. The method according to claim 1, wherein the first signal is generated by activating a first input mechanism of the input device, and wherein the first input mechanism is associated with the first flight parameter.

3. The method according to claim 1, wherein the first flight parameter is one of heading, airspeed, or altitude.

4. The method according to claim 1, further comprising: Displaying, near an icon representing the autonomous vehicle on the GUI, information associated with one or more flight parameters using a unique haptic cue for each flight parameter, the one or more flight parameters being associated with each autonomous vehicle among the plurality of autonomous vehicles, wherein the one or more flight parameters include one or more of altitude, heading, or speed of the autonomous vehicle represented by each icon among the plurality of icons.

5. The method according to claim 4, wherein the input mechanism of the input device associated with the first flight parameter is identified on the input device using the unique haptic cue assigned to the first flight parameter.

6. The method according to claim 4, wherein the information includes values of one or more current flight parameters and a value of the first flight parameter identified in the first command.

7. The method according to claim 1, wherein receiving an input for selecting the first icon further comprises: Detecting that a cursor hovers over the first icon for a predetermined amount of time, wherein the cursor is controlled using the input device communicatively coupled to the server computer.

8. The method according to claim 1, further comprising: Displaying, near the first icon, a current value of the first flight parameter and a target value of the flight parameter in consideration of the first command.

9. The method according to claim 1, wherein the first preset command is displayed in a circular format around a central element that identifies the first flight parameter, and wherein the placement of the first preset command is based on at least one of the orientation or amplitude of each preset command.

10. A system, comprising: a display device; one or more processors; an input device including a plurality of input mechanisms, wherein the input device is communicatively coupled to the one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform steps including: displaying a graphical user interface (GUI) on the display device; representing a plurality of autonomous aircraft using a plurality of icons displayed on the GUI; receiving an input selecting a first icon from among the plurality of icons displayed on the GUI; receiving a first signal from the input device; in response to receiving the first signal, displaying a first menu from among a plurality of menus, wherein the first menu is associated with a first flight parameter of the autonomous aircraft represented by the first icon, and wherein the first menu includes a plurality of first preset commands; receiving a selection of a first command from among the plurality of first preset commands; and transmitting the first command to a first autonomous aircraft represented by the first icon on the GUI.

11. The system according to claim 10, wherein the first signal is generated by activating a first input mechanism of the input device, and wherein the first input mechanism is associated with the first flight parameter.

12. The system according to claim 10, wherein when the instructions are executed by the one or more processors, the instructions cause the one or more processors to perform steps including: displaying information associated with one or more flight parameters, the one or more flight parameters being associated with each of the plurality of autonomous aircraft, near an icon representing the autonomous aircraft on the GUI using a unique haptic cue for each flight parameter, wherein the one or more flight parameters include one or more of altitude, heading, or speed of the autonomous aircraft represented by each of the plurality of icons.

13. The system according to claim 12, wherein the input mechanism of the input device associated with the first flight parameter is identified on the input device using the unique haptic cue assigned to the first flight parameter.

14. The system according to claim 12, wherein the information includes values of one or more current flight parameters and a value of the first flight parameter identified in the first command.

15. The system according to claim 10, wherein receiving an input selecting the first icon further includes: detecting that a cursor hovers over the first icon for a predetermined amount of time, wherein the cursor is controlled using the input device.

16. The system according to claim 10, wherein the first preset command is displayed in a circular format around a central element identifying the first flight parameter, wherein the placement of the first preset command is based on at least one of the orientation or amplitude of each preset command.

17. The system according to claim 10, wherein each input mechanism and the displayed menu are associated with the same sensory cue.

18. The system according to claim 17, wherein the sensory cue is color such that a first input mechanism assigned to the first flight parameter is provided in a first color and the first menu is displayed on the GUI using the first color.

19. One or more non-transitory computer-readable storage media storing instructions that, when executed on a server computer to remotely monitor and interact with a plurality of autonomous vehicles, cause the server computer to: display a graphical user interface (GUI) on a display device; represent the plurality of autonomous vehicles using a plurality of icons displayed on the GUI; receive an input selecting a first icon from among the plurality of icons displayed on the GUI; receive a first signal from an input device, the input device including a plurality of input mechanisms, wherein each input mechanism is assigned to a unique flight parameter; in response to receiving the first signal, display a first menu among a plurality of menus, wherein the first menu is associated with a first flight parameter of the autonomous vehicle represented by the first icon, wherein the first menu includes a plurality of first preset commands; receive a selection of a first command among the plurality of first preset commands; and transmit the first command to a first autonomous vehicle represented by the first icon on the GUI.

20. The one or more non-transitory computer-readable storage media according to claim 19, wherein the first preset command is displayed in a circular format around a central element identifying the first flight parameter, wherein the placement of the first preset command is based on at least one of the orientation or amplitude of each preset command.

Images