Remote supervision of multiple autonomous aircraft

Abstract

Embodiments provide a flight supervision platform comprising a server computer and an animated graphical user interface for simultaneously monitoring a plurality of autonomous aircraft. More specifically, the technology disclosed herein provides a flight supervision platform that allows one-to-many supervision in which a remote supervisor (e.g., human or artificial intelligence) can continuously monitor and / or interact with multiple autonomous aircraft. For example, the autonomous aircraft may include one or more autonomous electric vertical takeoff landing (eVTOL) aircraft. The flight supervision platform may display information via the GUI and receive input, and transmit a command based on the received input to the autonomous aircraft.

Description

[0001] Cross-references to other applications

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 408,088, filed on September 19, 2022, entitled “Remote Supervision of Multiple Autonomous Aircraft,” the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0003] This application is related to a U.S. design patent application entitled “Multi Vehicle Supervisor Interface” filed on September 19, 2022, with attorney case number 105984-1346953-010100US. Summary of the invention

[0004] Embodiments provide techniques for safely and efficiently monitoring and supervising many autonomous aircraft simultaneously. According to various embodiments, a graphical user interface (GUI) is provided for simultaneously monitoring, supervising and / or communicating (e.g., interacting) with multiple autonomous aircraft. The GUI can be an animated GUI including a graphical representation of aircraft and related information displayed on a map (e.g., superimposed on a map), the map representing an area where the autonomous aircraft is flying. The GUI may also include: a flight guidance pane for sending commands to modify flight parameters of the monitored autonomous aircraft. In some embodiments, using the GUI, the flight guidance pane may always be displayed in conjunction with a particular supervisor.

[0005] According to various embodiments, a computer-implemented method is provided for monitoring and interacting with a plurality of autonomous aircraft. The method includes: using a computer device, displaying a graphical user interface (GUI) on a display device; and representing a plurality of autonomous aircraft using a plurality of icons on a first pane of the GUI. Information associated with each of the plurality of autonomous aircraft is displayed in a plurality of sections on a second pane of the GUI. The method also includes: receiving a selection of a first icon among the plurality of icons on the first pane of the GUI or a first section among the plurality of sections on the second pane of the GUI. The method also includes: identifying the first icon among the plurality of icons on the first pane of the GUI or another of the first section among the plurality of sections on the second pane of the GUI. The first icon and the first section are displayed using one or more visual cues to distinguish the first icon and the first section from the remaining graphical elements. The method also includes: transmitting a command or message to the autonomous aircraft represented by the first icon on the GUI.

[0006] Various embodiments provide a system (e.g., a flight monitoring platform) for monitoring and interacting with multiple autonomous aircraft. The system includes: a display screen; and a server computer including one or more processors; and a memory storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform the above method.

[0007] Some embodiments provide one or more computer-readable storage media storing instructions that, when executed on a flight surveillance platform having a server computer, cause the server computer to perform the above-described method.

[0008] In order to better understand the nature and advantages of the present disclosure, reference should be made to the following description and drawings. However, it should be understood that each of the figures in the drawings is provided for illustration purposes only and is not intended to be used as a definition of the limitations of the scope of the present disclosure. Unless otherwise apparent from the description, when the same reference numerals are used for elements in different drawings, the elements are generally the same or at least similar in function or purpose. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram illustrating an exemplary flight supervision platform for monitoring and interacting with a plurality of autonomous aerial vehicles in accordance with various embodiments;

[0010] Figure 2Aillustrates an exemplary animated graphical user interface (GUI) of a flight supervision platform for supervising and interacting with a plurality of autonomous aerial vehicles in accordance with various embodiments;

[0011] Figure 2B Another view illustrating an exemplary animated GUI of a flight supervision platform in accordance with various embodiments;

[0012] Figure 3 illustrates an exemplary technique for assigning a route to an autonomous aerial vehicle using an exemplary animated GUI of a flight supervision platform in accordance with various embodiments;

[0013] Figure 4A illustrates adding layers to an exemplary animated GUI displayed by a flight monitoring platform in accordance with various embodiments;

[0014] Figure 4B illustrates removing layers from an animated GUI displayed by a flight surveillance platform in accordance with various embodiments;

[0015] Figure 5 illustrates tracking an autonomous aerial vehicle during flight using an exemplary animated GUI of a flight supervision platform in accordance with various embodiments;

[0016] Figure 6 illustrates a zoomed-in view of an exemplary animated GUI of a flight supervision platform in accordance with various embodiments;

[0017] Figure 7 illustrating another view of an exemplary animated GUI including a flight guide pane in accordance with various embodiments;

[0018] Figure 8 illustrates another view of an exemplary animated GUI after a mission assignment is accepted in accordance with various embodiments;

[0019] Fig. 9 illustrates an exemplary alert displayed using an exemplary animated GUI of a flight supervision platform in accordance with various embodiments; and

[0020] Fig.10 is a flow chart of an exemplary process performed by a flight surveillance platform in accordance with various embodiments. DETAILED DESCRIPTION

[0021] The technology disclosed herein generally relates to an animated graphical user interface for simultaneously monitoring multiple autonomous aircraft. More specifically, the technology disclosed herein provides a platform (e.g., a flight supervision platform) that allows one-to-many supervision, wherein a remote supervisor (e.g., a human or artificial intelligence) can continuously monitor and / or interact with multiple autonomous aircraft. The interaction may include: sending commands to the autonomous aircraft using a GUI component of the platform. For example, the autonomous aircraft may include one or more autonomous electric vertical take-off and landing (eVTOL) aircraft. Various embodiments of the present invention are described herein, including methods, processes, systems, devices, etc.

[0022] Several illustrative embodiments will now be described with respect to the accompanying drawings, which form a part of the embodiments. The subsequent description only provides (one or more) embodiments, and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of (one or more) embodiments will provide those skilled in the art with an effective description for implementing one or more embodiments. It should be understood that various changes can be made in the functions and arrangements of elements without departing from the spirit and scope of the present disclosure. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments of the present invention. However, it will be clear that various embodiments can be implemented without these specific details. The drawings and descriptions are not intended to be restrictive. The words "example" or "exemplary" are used herein to mean "used as an example, instance or illustration". Any embodiment or design described herein as "exemplary" or "example" may not be interpreted as being preferred or superior to other embodiments or designs.

[0023] An embodiment provides a flight supervision platform, comprising: a server computer; and an animated graphical user interface (GUI) for supervising, monitoring, and / or communicating with one or more autonomous aircraft. For example, a human operator may be able to supervise multiple autonomous aircraft simultaneously using a flight supervision platform, the flight supervision platform comprising an animated GUI displayed on a display device on the ground (e.g., a monitoring center located remotely relative to the autonomous aircraft). The supervisor may transmit commands to change one or more of the flight parameters of the autonomous aircraft (e.g., altitude, attitude, airspeed, heading).

[0024] According to various embodiments, the autonomous aircraft is configured to implement self-flight technology that implements autopilot functions, precision navigation, detection and avoidance capabilities, etc. The flight supervision platform described herein provides care for the autonomous aircraft and provides the ability to intervene when necessary. A flight supervision platform (e.g., a multi-vehicle supervision platform) can oversee the flight from start to finish. During the flight, if the autonomous aircraft takes measures to avoid potential hazards outside the assigned flight plan, the flight supervision platform receives a notification or warning, which is then displayed on the GUI. The flight supervision platform allows a supervisor (e.g., a human supervisor) to intervene at any point during the flight by sending new commands for the autonomous aircraft to execute. For example, a supervisor can use the flight supervision platform to send a command to the aircraft to redirect the aircraft to an alternative landing site.

[0025] Figure 1 A block diagram of a flight surveillance platform 1000 is illustrated, which can be used by a supervisor 1030 (e.g., a human supervisor 1030) to monitor and interact with one or more autonomous aerial vehicles 1040 individually and / or collectively. The flight surveillance platform 1000 may include: a server computer 1020, including one or more processors 1004, a system memory 1002 (the system memory 1002 may include any combination of volatile and / or non-volatile memory, such as, for example, buffer memory, RAM, DRAM, ROM, flash memory, or any other suitable memory device), and a network interface (e.g., an external communication interface) 1006. In some embodiments, one or more of the modules may be housed within one or more of the components of the system memory 1002, or may be housed externally. For illustrative purposes only, Figure 1 The software and hardware modules shown in are provided, and the configuration is not intended to be limiting. The processor 1004, the system memory 1002, and / or the external communication interface 1006 can implement the techniques and / or methods described herein.

[0026] The network interface 1006 may be configured or programmed to receive and generate electronic messages transmitted to or from the plurality of autonomous aerial vehicles 1040 via the flight surveillance platform 1000 , the electronic messages including information.

[0027] The flight surveillance platform 1000 may also include: at least one display device 1010 for displaying a graphical user interface (GUI) 1012. When an electronic message is received by the flight surveillance platform 1000 via the external communication interface 1006 of the server computer 1020, it may be processed and the relevant information may be displayed on the display device 1010 via the GUI 1012. When input is received from the supervisor 1030 via the GUI 1012, it may be processed and the relevant information may be transmitted to the corresponding autonomous aircraft 1040. According to various embodiments, the flight surveillance platform 1000 may also be configured to receive supplementary information from a third party, such as air traffic control, weather, other aircraft (e.g., aircraft monitoring the one or more autonomous aircraft 1040 in the air). The supplementary information may be processed by the flight surveillance platform 1000 and displayed on the GUI 1012.

[0028] Figure 2A An exemplary animated graphical user interface (GUI) 100 of a flight supervisor platform 1000 for supervising and interacting with multiple autonomous aircraft is illustrated in accordance with various embodiments. In the exemplary GUI 100, a supervisor (e.g., Figure 1 A supervisor 1030 illustrated in FIG. 1 is overseeing five autonomous aerial vehicles illustrated on the GUI 100 using five aerial vehicle icons. The GUI 100 may include a first pane 102 illustrating each autonomous aerial vehicle with corresponding aerial vehicle icons 116, 118 superimposed on a map of terrain over which the autonomous aerial vehicles are flying, hovering, and / or waiting on the ground. The amount of detail to be included on the map may be determined using human factors including, for example, an assessment of how much information can be safely processed by a human on a screen. Some embodiments may use artificial intelligence (AI) as a supervisor or a combination of AI and human supervisors. The amount of detail to be included on the map may vary depending on the capabilities of the supervisory entity. In some embodiments, the supervisor

[0029] The GUI 100 may also include a second pane 104 that includes segments 106, 108 corresponding to each of the aircraft icons 116, 118 illustrated in the first pane 102. That is, each segment 106, 108 provides information associated with a different autonomous aircraft under the supervision of the supervisor. In some embodiments, when the supervisor selects an autonomous aircraft by, for example, clicking on the desired aircraft icon 116, a visual cue may associate the selected aircraft icon 116 displayed on the first pane 102 with the corresponding information segment 106 displayed on the second pane 104. For example, the selected aircraft icon 116 and the corresponding information segment 106 may be displayed in a different color, shading, and / or font than the rest of the graphical elements of the GUI. The selected aircraft icon 116 and the information segment 106 represent the same autonomous aircraft (e.g., Figure 1 1040). According to various embodiments, the GUI 100 may only display active missions (e.g., missions with an active status) that are supervised by the supervisor. There may be additional missions assigned to the supervisor (e.g., missions that are not yet active or missions that are in a state of being ready to fly), and their information may be displayed on the appropriate panes 102, 104 of the GUI 100 after the supervisor accepts the mission or after the mission presents an active status.

[0030] In some embodiments, selecting an aircraft icon 116 on the first pane 102 may cause the corresponding information segment area 106 to be automatically selected on the second pane 104. Alternatively, selecting a corresponding information segment area 106 on the second pane 104 may cause the aircraft icon to be automatically selected on the first pane 102. Figure 1 When the server computer 1020 illustrated in FIG. 1 receives a selection of the aircraft icon 116 among the plurality of icons on the first pane 102 of the GUI 100 or a selection of the information segment area 106 among the plurality of segment areas on the second pane 104 of the GUI 100, the server computer identifies the other of the aircraft icon 116 or the first segment area 106 on the GUI 100. The aircraft icon 116 and the information segment area 106 are then displayed using one or more visual cues to distinguish the aircraft icon 116 and the information segment area 106 from the remaining graphical elements.

[0031] Displaying the aircraft icon and the information segment area at separate locations on the GUI enables accurate identification and selection of the desired autonomous aircraft represented using the aircraft icon and the information segment area. For example, if two autonomous aircraft are flying above each other, their representations on the map (e.g., aircraft icons) may be displayed as superimposed directly above each other. Therefore, the supervisor may not be able or cannot easily use the first pane 102 to select one of the superimposed autonomous aircraft. In this case, using the corresponding segment area on the second pane 104, the supervisor can select the autonomous aircraft they want to interact with. In this way, selection of the desired aircraft is ensured.

[0032] As shown in the exemplary embodiment illustrated in the figures, the first pane 102 is allocated greater real estate on the screen than the second pane 104. This allocation of real estate allows the aircraft icon to be displayed on a larger map (e.g., while also being able to display associated information on a separate segment of the screen (e.g., the second pane 104) without cluttering the display on the map. According to various embodiments, the relative positioning and size of the first pane 102 and the second pane 104 may vary.

[0033] Through the GUI 100, the flight supervision platform 1000 allows the supervisor to interact with all autonomous aircraft illustrated on the GUI 100 using the corresponding aircraft icons 116, 118 or information segment areas 106, 108. Before takeoff, the autonomous aircraft can autonomously perform pre-flight inspections, including checking one or more of the power system, control surfaces, avionics, passenger equipment, cabin systems, and / or the environment around the autonomous aircraft. The autonomous aircraft can also continuously monitor these systems and / or items during flight. When the inspection is completed, the autonomous aircraft can notify (e.g., transmit data and / or messages to) the flight supervision platform 1000 (and thereby, the supervisor 1030), and the autonomous aircraft and passengers (if applicable) are ready for takeoff.

[0034] Upon receiving a signal that the autonomous aircraft is ready to take off and / or a route is assigned to the autonomous aircraft (hereinafter referred to as Figure 3 After receiving a notification (discussed further), the supervisor may first select the aircraft icon 116 corresponding to the autonomous aircraft on the first pane 102, and then initiate the flight by, for example, selecting the first (e.g., take-off) button 110 (e.g., as shown in FIG. Figure 2A , using a visual cue (such as bold font), the selected liftoff button 110 is displayed to indicate that the button has been selected), and then a second (e.g., execute) button 112 is selected on the second pane 104 (e.g., as shown in FIG. Figure 2B100, the lift command is sent to the autonomous aircraft. The server computer 1020 may receive a selection of the aircraft icon 116 on the first pane 102 of the GUI 100, and may receive a lift command from the second pane 104 of the GUI 100, and then receive a execute command. The server computer 1020 may then transmit the lift command to the autonomous aircraft represented using the aircraft icon 116 on the GUI 100.

[0035] exist Figure 2A and 2B In the embodiment illustrated in FIG, when the supervisor selects an icon on the first pane of the GUI or an information segment area on the second pane of the GUI, no additional pop-up window is displayed. This eliminates clutter on the GUI and reduces distraction for the supervisor. In some embodiments, pop-up windows or changes to the layout of the GUI may be prevented to reduce or eliminate distraction for the supervisor.

[0036] In some embodiments, while the autonomous aircraft is following the assigned route, the supervisor may receive instructions from air traffic control (ATC) that need to be forwarded to the autonomous aircraft under supervision. The display of the GUI is optimized to enable the supervisor to quickly identify the correct autonomous aircraft and forward the ATC communication to the correct autonomous aircraft.

[0037] In some embodiments, the supervisor may locate the autonomous aircraft on the map displayed on the first pane 102 and determine whether the ATC communication is applicable. For example, if the map shows an autonomous aircraft in mountainous terrain that is unsafe for the maneuver suggested in the ATC communication, the supervisor may confirm the maneuver with ATC before forwarding it to the autonomous aircraft, or may determine that the communication is erroneous and should be discarded based on the information provided to the supervisor using the map displayed on the GUI 100.

[0038] Return to reference Figure 2A , while four of the aircraft may already be in the air following the route, a fifth aircraft, represented by aircraft icon 116, may be on the ground, awaiting a route assignment. According to various embodiments, the supervisor may transmit instructions to the aircraft, such as assigning a route, giving lift instructions, etc. Upon receiving the route assignment, the autonomous aircraft executes the flight plan to follow the route.

[0039] Figure 3An exemplary technique for assigning a route to an autonomous aircraft using an exemplary animated GUI of a flight supervisor platform according to various embodiments is illustrated. The GUI 100 may also be used to assign a flight path to an autonomous aircraft. For example, an autonomous aircraft may be on the ground awaiting flight assignment. By selecting an aircraft icon 116 or an information segment area 106 representing an aircraft on the ground, a supervisor may select an autonomous aircraft (e.g., Figure 2A and 2B ). The selection of an autonomous aircraft on the ground may trigger the GUI 100 to display a list of available routes 202 using a menu 200. It will be appreciated by those of ordinary skill in the art that the graphical elements illustrated in the accompanying drawings for providing information or receiving input from a supervisor are not limited to the tools shown in the accompanying drawings, and alternative tools may be used to convey the same information or receive the same input. For example, the menu 200 may include a scroll-down menu, a drop-down menu, etc. The supervisor may select one of the routes by, for example, clicking on a desired route among the list of available routes 202. The supervisor may be asked to confirm the selection by, for example, selecting an accept button 204. An accept button 204 for confirming the selection may be displayed on the second pane 104, while a menu 200 with the route 202 may be displayed on the first pane 102. Therefore, the confirmation action may require the supervisor to travel from the first pane 102 to the second pane 104 to intentionally select the accept button 204. This configuration may avoid performing a selection in error.

[0040] When route 206 is selected to be assigned to an aircraft, the selected route 206 may be displayed on the map as a displayed route 210 from the departure point to the arrival point using a visual cue (such as using a first color). This will also allow the supervisor to visually confirm that the intended route is assigned to the autonomous aircraft. The departure point and arrival point 208 of the selected route may be displayed on the information segment area corresponding to the selected autonomous aircraft. For example, if the intention is to assign a route from Los Angeles to San Francisco, but the displayed route is from San Diego to Los Angeles, the supervisor may be aware of the error before executing the route assignment command. Once the supervisor confirms that the displayed route 210 is the intended route, the supervisor may execute the assignment by selecting the accept button 204. The execution of the command may result in the display of the assigned route on the map from the departure point to the arrival point using a different visual cue (such as using a second color) to illustrate that the route is now a confirmed route. That is, the GUI 100 forwards the transition from the selected route to the confirmed route.

[0041] The server computer 1020 coupled to the GUI 100 may continuously receive flight parameters and location positioning signals of the plurality of autonomous aircraft monitored using the GUI 100. As the autonomous aircraft proceed along the displayed route 210, using the flight data and location positioning signals of the autonomous aircraft, the server computer may periodically update the GUI 100 to display the relative position of the first icon 116 with respect to the displayed route 210 assigned to the autonomous aircraft.

[0042] According to various embodiments, the supervisor may select the amount of detail to be shown on the map. For example, the GUI 100 may have information layers 250 that can be displayed in an overlay. The supervisor may select one or more of the available layers (e.g., radius information 252 around the aircraft, wind information 254, telemetry information received from one or more aircraft 256, any additional information 258) to have a desired level of detail on the map.

[0043] Figure 4A-4B The various levels of detail displayed on the GUI of the flight monitoring platform according to various embodiments are illustrated. Specifically, Figure 4A The diagram adds layers to the exemplary animation GUI shown, and Figure 4B Illustration of removing a layer from the displayed animation GUI. Figure 4A Shown is a visual aid 302 for illustrating the environment of an autonomous aircraft represented using the aircraft icon 116. For example, the visual aid 302 may have the form of concentric rings, each ring corresponding to a predetermined distance relative to the autonomous aircraft. In this way, a supervisor can quickly see and assess whether there are other aircraft or hazards in the vicinity of the monitored autonomous aircraft. Similarly, the visual aid 302 may also convey information about the distance between the monitored autonomous aircraft and a geographical element (e.g., a mountain) in the vicinity of the monitored autonomous aircraft.

[0044] If the supervisor prefers a cleaner display, the supervisor can remove the visual aid 302 from the GUI, such as Figure 4B For example, when the radius layer 252 ( Figure 3 ) is selected, and the visual aid 302 may be removed when the radius layer 252 is not selected on the GUI 100. Although some of the elements of the GUI may not be masked (e.g., some elements are always displayed and may not be removed by switching layers), other elements may be displayed based on the selected options provided on the GUI (e.g., based on the selected options provided on the GUI). Figure 3 A selection of the map layer 250 illustrated in FIG. 1 is displayed.

[0045] 4-5 , the aircraft icons 116, 118 may have additional information 400 displayed on the map in conjunction with the aircraft icons. For example, flight parameters associated with the autonomous aircraft represented by the respective aircraft icons 116, 118 (such as heading, altitude, and / or speed information) may be displayed next to the aircraft icons 116, 118. In some embodiments, the information 400 may be affixed to the aircraft icons and may move with the aircraft icons along a displayed path.

[0046] like Figure 5 As shown in FIG. 1 , the animated GUI may show an aircraft icon 116 moving along a route 120 assigned to the aircraft. As the autonomous aircraft advances along the assigned route (e.g., flight path), the corresponding aircraft icon 116 moves along the route 120 on the map. The displayed information 400 moves with the aircraft icon 116. By having all the information on the same GUI, the techniques provided herein ensure that the supervisor evaluates the relevant information associated with the autonomous aircraft and selects the correct autonomous aircraft when activating a route or when communicating a command to the autonomous aircraft. In some embodiments, the supervisor can toggle between showing and hiding the displayed information 400. That is, flight parameters associated with the autonomous aircraft can be selectively displayed near the first icon representing the autonomous aircraft on the GUI.

[0047] According to various embodiments, the autonomous aircraft supervised using the animated GUI 100 flies autonomously using instrument flight rules (IFR). If ATC directs the supervisor to transmit new parameters (e.g., altitude, heading, speed, etc.) to the autonomous aircraft, the supervisor can do so by selecting the aircraft icon 116 and sending a command with the new parameters. In some embodiments, the autonomous aircraft can confirm receipt of the command and / or execution of the command with a message transmitted back to the server computer and displayed on the GUI 100.

[0048] In some embodiments, the GUI may be displayed at various zoom levels, as selected by the supervisor, or based on the selected action. For example, when a flight path is assigned to an autonomous aircraft, the GUI may be displayed at a zoom level that shows both the start and end points of the route. For example, when a flight parameter change command is entered, the GUI may be displayed at a zoom level that better illustrates the surroundings of the autonomous aircraft to ensure that the flight parameter modification does not pose an imminent threat to the autonomous aircraft.

[0049] Figure 6 FIG. 1 illustrates an enlarged view of an exemplary animated GUI of a flight monitoring platform according to various embodiments. Figure 6As shown in , by connecting multiple waypoints 552, 554, 556, a route can be formed. In some embodiments, when the autonomous aircraft has flown over a section 560 (e.g., a portion) between waypoints, the section can change color. For example, in Figure 6 In the exemplary embodiment illustrated in FIG. 5 , segment 560 (e.g., the completed portion of the route) is shown in a first color (e.g., white), illustrating that segment 560 of the route has been completed, while segment 562 (e.g., the remainder of the route) is shown in a second color (e.g., purple), indicating that the segment has not yet been completed. These visual cues can help a supervisor gather various types of information simultaneously simply by looking at or observing the GUI. Using the visual cues, a supervisor can track multiple autonomous aerial vehicles under his or her supervision.

[0050] According to various embodiments, when the aircraft icon 116 is selected on the GUI, the supervisor may select a waypoint on the planned route of the aircraft represented by the aircraft icon 116 and route the aircraft directly to the selected waypoint. For example, if the aircraft is configured to fly from waypoint A to waypoint B and then to waypoint C, the supervisor may select waypoint C and direct the aircraft to fly directly from waypoint A to waypoint C, skipping waypoint B. As with the various commands described herein, the one or more selection inputs on the GUI are transmitted to the autonomous aircraft by a server computer coupled to the GUI.

[0051] When the autonomous aerial vehicle completes its flight plan (eg, reaches a destination point), the aircraft icon 116 may be shown with one or more visual cues indicating that the autonomous aerial vehicle has completed its flight and has landed.

[0052] Figure 7 Illustrated is another view of an exemplary animated GUI including a flight guide pane in accordance with various embodiments. Figure 7 The exemplary GUI 600 shown in FIG. 6 includes a first pane 602 (similar to Figure 2A The exemplary GUI 600 also includes a second pane 604 (similar to the first pane 602 of FIG. 600 ), the first pane 602 having a plurality of icons 601, 603, each icon representing an autonomous aircraft supervised using the GUI 600. Figure 2AThe exemplary GUI 600 further includes a third pane 606 that displays a list of missions 650 assigned to a supervisor of the GUI 600. Each mission assignment represents an additional autonomous flight that is being assigned to the supervisor for monitoring by the GUI. That is, the third pane 606 illustrates an autonomous aircraft that is being assigned to the GUI 600 but is not currently being supervised using the GUI 600. For example, the list of missions 650 may include missions that are waiting for a supervisor (e.g., an operator) to accept responsibility for their execution. Once the supervisor accepts the assigned mission, final preparations before departure can be completed. In some embodiments, the missions 650 on the list may not include all of the assigned missions, but may only include assigned missions that are in a pre-prepared state, as described above. The following is combined with Figure 8 Describes the acceptance of an assigned mission (e.g., autonomous flight).

[0053] According to various embodiments, the flight supervision platform 1000 may display a message reporting one or more upcoming tasks that may require approval or acceptance by the supervisor 1030. For example, during a predetermined amount of time, according to a prioritized manner, along with information needed to avoid human error or degradation of human performance (e.g., delays in responding to time and / or safety critical events), the upcoming task reminder may be displayed. In some embodiments, the reminder message may have a countdown displayed within it to indicate the remaining time for the supervisor 1030 to act on the reminder.

[0054] Continuing with the description of the exemplary GUI 600, a fourth pane 608 is provided to illustrate a zoomed map view of the selected autonomous aircraft. For example, if icon 601 is selected on the first pane 602, a visual cue (as described above in conjunction with Figure 2A and 2B ), a corresponding information segment area 605 is identified on the second pane 604, and using the zoomed-in map, the vicinity around the selected autonomous aircraft is shown in the pane 608. The exemplary GUI 600 also includes a flight guidance pane 610. The GUI 600 can receive a modification on the flight guidance pane 610 corresponding to a change related to the selected autonomous aircraft (e.g., Figure 7The server computer transmits commands to the autonomous aircraft to implement the desired changes. According to various embodiments, the flight guidance pane 610 may include one or more of a control dial 612 or 614, a drop-down menu, a text input field, or a widget 620.

[0055] For example, a first control dial 612 may enable a supervisor to enter a heading setting or modification for a selected autonomous aircraft. The GUI may display possible headings in predetermined increments. When the supervisor makes a selection, the selected heading may be indicated using a visual cue 618. A second control dial 614 may enable a supervisor to enter an altitude setting or modification for a selected autonomous aircraft. The GUI may display possible altitudes in predetermined increments. When the supervisor makes a selection, the selected heading may be indicated using a visual cue 616. In some embodiments, flight parameter settings or modifications may be entered on a text input field. When one or more of the flight parameter settings or modifications are completed, the supervisor may select a widget 620 (e.g., an "execute" widget) to cause the server computer to transmit a command to the autonomous aircraft to implement the one or more flight parameter settings or modifications.

[0056] Figure 8 Another view of an exemplary animated GUI of a flight monitoring platform after a mission assignment is accepted is illustrated in accordance with various embodiments. As described above, Figure 7 The exemplary GUI 600 shown in FIG. 6 includes a third pane 606 that displays a list of missions 650 assigned to the supervisor of the GUI 600. When the supervisor accepts one of the missions, the mission is moved from the third pane to the second pane, which displays a list of active autonomous flights being supervised by the supervisor. Figure 8 As shown in FIG. 7 , the supervisor has accepted mission 702, which is removed from third pane 705 and moved to second pane 740. Second pane 740 displays all autonomous aircraft that are supervised and / or managed by the supervisor, regardless of whether the autonomous aircraft is in flight or on the ground (e.g., waiting to begin a mission or recently landed as part of its mission).

[0057] When the supervisor accepts a new mission, the flight guidance pane 610 temporarily switches to the configuration view 710. The configuration view 710 provides additional information associated with the newly accepted mission, including, for example, the origin and destination points, estimated departure time, the charge status of the aircraft, flight check status, etc. The configuration view 710 may include: a text area 704 for displaying additional information associated with the newly accepted mission; and one or more command widgets 708. When the launch command is selected on the GUI, the server computer sends the launch command to the autonomous aircraft, and the configuration view 710 switches back to the flight guidance pane 610, as shown in FIG. Figure 7 as shown in .

[0058] On the first pane 730 of the GUI, autonomous flights managed by the supervisor may be displayed using icons. According to various embodiments, an icon 732 representing an autonomous aircraft in flight may have a different representation than an icon 734 representing an autonomous aircraft on the ground (e.g., an autonomous aircraft ready to fly or recently landed). An icon representing a newly accepted mission may not yet be associated with a flight plan (e.g., a route) on the map. When an execution launch command is transmitted to the autonomous aircraft by the server computer, the icon may be associated with a flight plan or route on the GUI.

[0059] like Figure 8 As shown in , zooming in on the map pane 720 can provide a close-up bird's-eye view of the surroundings of the newly accepted mission. The map pane 720 can also indicate the current status of the autonomous aircraft. For example, since the selected autonomous aircraft corresponds to the newly accepted mission 702, the status of the autonomous aircraft is indicated as "Transition Level 1, Transition-Climb", meaning that the autonomous aircraft is transitioning to a vertical flight mode to become airborne or climb.

[0060] According to various embodiments, a server computer coupled to the GUI may receive input selecting a mission assignment displayed in the pane 705. For each selected mission assignment, an individual icon 734 is created in the first pane 730, and a corresponding individual section 702 representing information associated with the selected mission assignment is created in the second pane 740. The accepted missions are then removed from the third pane 705 of the GUI.

[0061] According to various embodiments, an entity (e.g., ATC, a third party, another aircraft) may send a warning to a supervisor. In some embodiments, at a flight supervision platform and / or a server computer in communication with the GUI, a warning may be received and a visual warning may be displayed on the GUI. The supervisor may choose to ignore the displayed warning, or take action on the displayed warning. In some embodiments, the GUI may also display proposed changes or modifications to the flight parameters of the monitored autonomous aircraft. The supervisor may accept the proposed changes and transmit commands to change the flight parameters to the autonomous aircraft. In some embodiments, in response to the warning, the supervisor may use the flight guidance pane of the GUI to manually input the desired changes to the flight parameters. The server computer may then use the GUI to transmit the commands to the autonomous aircraft.

[0062] In some embodiments, the autonomous aircraft is configured to change its course to avoid a collision with a hazard or to avoid mission failure due to, for example, an electromechanical problem on board. The autonomous aircraft may change its course without receiving approval from a supervisor. In some embodiments, the autonomous aircraft may be configured to wait for approval during a predetermined amount of time before implementing a change in the flight plan.

[0063] According to various embodiments, a dedicated area of ​​the GUI may be reserved for displaying warning messages. For example, feedback or data received from sensors or other equipment of the autonomous aircraft indicating a problem with the aircraft may be displayed as a warning in this dedicated area. In some embodiments, given the importance of the warning, the warning may be displayed as a pop-up screen on a primary pane (e.g., the first pane) of the GUI.

[0064] Fig. 9 An exemplary alert displayed using an exemplary animated GUI according to various embodiments is illustrated. In some embodiments, an autonomous aircraft may detect a hazardous object 800 on its route 802 and may divert from its route 802 to avoid colliding with the hazardous object 800. For example, the autonomous aircraft may turn left or right and change its route to an alternative route 804 that avoids colliding with the hazardous object 800. In some embodiments, the autonomous aircraft may need to change its route due to problems identified by the autonomous aircraft, such as thruster (e.g., propeller) failure. Alternative routes 804 and collision avoidance warnings (e.g., alarms) 806 are displayed on the GUI to notify the supervisor. In some embodiments, the autonomous aircraft may ask for a trajectory update when the hazardous object 800 is transmitted to the flight supervision platform. A request for a trajectory update may be displayed on the GUI. The supervisor may provide a response to the trajectory update request by sending changes to the flight parameters or flight plan of the autonomous aircraft via the server computer.

[0065] When alert 806 is displayed on the GUI, the GUI may switch the selected aircraft icon to the aircraft icon associated with alert 806. For example, if an alert is received from or is otherwise associated with the autonomous aircraft represented by icon 805 and segment 822, the GUI may display icon 805 and corresponding segment 822 as the selected autonomous aircraft. In some embodiments, if a warning is displayed for a change that has been implemented by the autonomous aircraft, the warning may be displayed for a predetermined amount of time before switching back to the aircraft last selected by the supervisor.

[0066] In some embodiments, ATC may recommend a flight plan change that can be displayed as an alert on the GUI. The alert may also include a widget displayed on the GUI to allow the supervisor to accept or reject the recommended flight plan change. If the supervisor accepts the recommended flight plan change, selecting the widget to accept the recommended flight plan change may cause the server computer to transmit a command to the autonomous aircraft to implement the recommended flight plan change (e.g., change heading as indicated in the recommended flight plan change). The autonomous aircraft executes the received command and begins following the modified flight plan.

[0067] Embodiments provide a flight monitoring platform (e.g., Figure 1 The flight supervision platform 1000 illustrated in FIG. 1000 is used to perform the techniques described herein. Information received from the autonomous aircraft being monitored or any other source is analyzed and / or processed by a server computer (e.g., server computer 1020) and transmitted to a GUI (e.g., GUI 1012) for display. Similarly, any input or selection received at the GUI is analyzed and / or processed by the server computer and transmitted to the intended autonomous aircraft via a secure communication channel between the server computer and the autonomous aircraft.

[0068] Fig.10 is a flow chart of an exemplary process 900. In some implementations, Fig.10 One or more process blocks may be executed by a flight supervision platform.

[0069] like Fig.10 As shown in FIG. 9 , process 900 may include: using a server computer, displaying a graphical user interface (GUI) on a display device (block 902). Using a plurality of icons, a plurality of autonomous aircraft are represented on a first pane of the GUI (block 904). Fig.10As further shown in FIG. 9 , process 900 may include displaying information associated with each of the plurality of autonomous aircraft in a plurality of sections on a second pane of the GUI (block 906). The server computer may receive input that selects one of (1) a first icon among the plurality of icons on the first pane of the GUI or (2) a first section among the plurality of sections on the second pane of the GUI (block 908). The server computer may then identify another of (1) a first icon among the plurality of icons on the first pane of the GUI or (2) a first section among the plurality of sections on the second pane of the GUI (block 910). The first icon and the first section are displayed on the GUI using one or more visual cues to distinguish the first icon and the first section from the remaining graphical elements (block 912). The server computer transmits a command or message to the autonomous aircraft represented by the first icon on the GUI (block 914).

[0070] In some embodiments, flight parameters associated with the autonomous aircraft are displayed on the GUI near a first icon representing the autonomous aircraft, wherein the flight parameters include one or more of the altitude, heading, or speed of the autonomous aircraft. According to various embodiments, the method may include: receiving flight data messages from the plurality of autonomous aircraft. Each flight data message includes one or more flight parameters associated with one of the plurality of autonomous aircraft. The plurality of icons representing the plurality of autonomous aircraft may be displayed on the GUI together with the corresponding one or more flight parameters.

[0071] Although Fig.10 Example blocks of process 900 are shown, but in some implementations, Fig.10 Compared to the blocks described in , process 900 may include additional blocks, fewer blocks, different blocks, or blocks arranged in a different manner. Additionally or alternatively, two or more blocks in the blocks of process 900 may be performed in parallel.

[0072] The embodiment may also provide one or more computer-readable storage media, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a flight monitoring platform including a server computer, the instructions cause the server computer to execute the above method. The embodiment may also provide a system (e.g., a flight monitoring platform), wherein the system includes a display screen and a server computer, wherein the server computer includes: one or more processors; and a memory, wherein the instructions are stored, and when the instructions are executed by the one or more processors, the instructions cause the one or more processors to execute the above method.

[0073] In the foregoing description, embodiments of the present disclosure have been described with reference to many specific details that can vary depending on the implementation. Therefore, the description and drawings should be viewed in an illustrative rather than a restrictive sense. The only specific indicator of the scope of the present disclosure and what the applicants intend to be the scope of the present disclosure is the text and equivalents of the set of claims issued from this application in the specific form in which such claims are issued, including any subsequent corrections. The specific details of specific embodiments can be combined in any suitable manner without departing from the spirit and scope of the embodiments of the present disclosure.

[0074] Additionally, spatially relative terms, such as "bottom," "top," or "side," etc., can be used to describe the relationship of one element and / or feature to another (one or more) element and / or (one or more) feature, as shown, for example, in the accompanying drawings. It will be understood that the spatially relative terms are intended to include different orientations of the device in use and / or operation in addition to the orientation described in the accompanying drawings. For example, if the device in the accompanying drawings is turned over, the element described as the "bottom" surface can then be oriented "above" other elements or features. The device can be oriented differently (e.g., rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein are interpreted accordingly.

[0075] The methods, systems and devices discussed herein are examples. Various embodiments may omit, replace or add various procedures or parts as needed. For example, features described with respect to certain embodiments may be combined in various other embodiments. Different aspects and elements of the embodiments may be combined in a similar manner. In addition, technology evolves, and therefore, many elements in the elements are examples, which do not limit the scope of the present disclosure to those specific examples.

[0076] As used herein, the terms "and", "or" and "and / or" may include various meanings, which are also expected to depend at least in part on the context in which such terms are used. In general, if "or" is used for an associated list (such as, A, B, or C), it is intended to mean: A, B, and C, used here in an inclusive sense; and A, B, or C, used here in an exclusive sense. In addition, as used herein, the term "one or more" may be used to describe any feature, structure, or characteristic in the singular, or may be used to describe a certain combination of features, structures, or characteristics. However, it should be noted that this is merely an illustrative example, and the subject matter claimed is not limited to this example. In addition, if the term "at least one of..." is used for an associated list (such as, A, B, or C), it can be interpreted to mean any combination of A, B, and / or C, such as A, B, C, AB, AC, BC, AA, AAB, ABC, AABBCCC, etc.

[0077] References throughout this specification to "one example," "example," "some examples," or "exemplary implementations" mean that a particular feature, structure, or characteristic described in conjunction with that feature and / or example may be included in at least one feature and / or example of the claimed subject matter. Thus, the appearances of the phrases "in one example," "example," "in some examples," "in some implementations," or other similar phrases in various places throughout this specification are not necessarily all referring to the same feature, example, and / or limitation. In addition, particular features, structures, or characteristics may be combined in one or more examples and / or features.

[0078] In the foregoing detailed description, many specific details have been set forth to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter can be implemented without these specific details. In other examples, methods and devices that will be known to those of ordinary skill in the art are not described in detail to avoid obscuring the claimed subject matter. Therefore, the claimed subject matter is not intended to be limited to the specific examples disclosed, but such claimed subject matter may also include all aspects and equivalents thereof that fall within the scope of the appended claims.

Claims

1. A computer-implemented method for monitoring and interacting with a plurality of autonomous aerial vehicles, the method comprising: using the server computer, displaying a graphical user interface (GUI) on a display device; representing, by the server computer, a plurality of autonomous aerial vehicles using a plurality of icons on a first pane of the GUI; displaying, by the server computer, information associated with each of the plurality of autonomous aerial vehicles in a plurality of sections on a second pane of the GUI; receiving, by the server computer, an input selecting one of a first icon among the plurality of icons on the first pane of the GUI or a first section area among the plurality of sections on the second pane of the GUI; identifying, by the server computer, another one of the first icon among the plurality of icons on the first pane of the GUI or the first section area among the plurality of sections on the second pane of the GUI; Displaying the first icon and the first segment region by the server computer using one or more visual cues to distinguish the first icon and the first segment region from other graphical elements; as well as A command or message is transmitted by the server computer to the autonomous aerial vehicle represented by the first icon on the GUI.

2. The method according to claim 1, further comprising: displaying one or more mission assignments on a third pane of the GUI, each mission assignment representing an additional autonomous flight to be monitored by the GUI; receiving an input, the input selecting at least one of the one or more mission assignments; For each selected mission assignment: generating an individual icon and an individual segment representing information associated with the selected mission assignment; displaying the individual icons on the first pane of the GUI; displaying the individual segment area on the second pane of the GUI; as well as The selected mission is removed from the third pane of the GUI.

3. The method according to claim 1, further comprising: displaying a flight guide pane on the GUI; receiving, on the flight guidance pane, a modification corresponding to a desired change in one or more flight parameters associated with the autonomous aerial vehicle represented by the first icon; as well as A command is transmitted to the autonomous aerial vehicle represented by the first icon to implement the desired change. 4 . The method of claim 3 , wherein the flight guide pane comprises one or more of a control dial, a drop-down menu, a text entry field, or a widget. 5 . The method of claim 1 , wherein the plurality of icons are superimposed on a map displayed on the first pane, wherein the first pane is larger than the second pane on the display device.

6. The method according to claim 1, further comprising: displaying a list of one or more routes on the GUI; receiving a selection of a route from the list; assigning the route to the autonomous aerial vehicle represented by the first icon on the GUI, wherein the autonomous aerial vehicle executes a flight plan to follow the route upon receiving the route assignment; as well as As the autonomous aerial vehicle advances along the route, the GUI is periodically updated to display a relative position of the first icon with respect to the route assigned to the autonomous aerial vehicle.

7. The method according to claim 6, further comprising: On the GUI, a completed portion of the route is displayed using a first visual cue, and a remaining portion of the route is displayed using a second visual cue.

8. The method according to claim 1, further comprising: Flight parameters associated with the autonomous aircraft are selectively displayed adjacent to the first icon representing the autonomous aircraft on the GUI, wherein the flight parameters include one or more of an altitude, a heading, or a speed of the autonomous aircraft.

9. The method according to claim 1, further comprising: receiving a message from the autonomous aerial vehicle indicating a hazard detected by the autonomous aerial vehicle along a route and an alternate route to be followed by the autonomous aerial vehicle; as well as A collision avoidance warning and the alternative route are displayed on the GUI.

10. The method according to claim 1, further comprising: receiving location positioning signals from the plurality of autonomous aerial vehicles; as well as The plurality of icons representing the plurality of autonomous aerial vehicles are displayed on a map based on the location positioning signal.

11. The method according to claim 1, further comprising: receiving flight data messages from the plurality of autonomous aerial vehicles, wherein each flight data message includes one or more flight parameters associated with one of the plurality of autonomous aerial vehicles; as well as The plurality of icons representing the plurality of autonomous aerial vehicles and corresponding one or more flight parameters are displayed on the GUI.

12. A system for monitoring and interacting with a plurality of autonomous aerial vehicles, the system comprising: Display, one or more processors, and A memory storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform steps, the steps comprising: displaying a graphical user interface (GUI) on a display device; Representing a plurality of autonomous aerial vehicles using a plurality of icons on a first pane of the GUI; displaying information associated with each of the plurality of autonomous aerial vehicles in a plurality of sections on a second pane of the GUI; receiving an input that selects one of a first icon among the plurality of icons on the first pane of the GUI or a first segment area among the plurality of segments on the second pane of the GUI; identifying another one of the first icon among the plurality of icons on the first pane of the GUI or the first segment area among the plurality of segment areas on the second pane of the GUI; displaying the first icon and the first segment region using one or more visual cues to distinguish the first icon and the first segment region from other graphical elements; and A command or message is transmitted to the autonomous aerial vehicle represented by the first icon on the GUI.

13. The system of claim 12, wherein when the instructions are executed by the one or more processors, the instructions cause the one or more processors to perform steps, the steps further comprising: displaying one or more mission assignments on a third pane of the GUI, each mission assignment representing an additional autonomous flight to be monitored by the GUI; receiving an input, the input selecting at least one of the one or more mission assignments; For each selected mission assignment: generating an individual icon and an individual segment representing information associated with the selected mission assignment; displaying the individual icons on the first pane of the GUI; displaying the individual segment area on the second pane of the GUI; as well as The selected mission is removed from the third pane of the GUI.

14. The system of claim 12, wherein when the instructions are executed by the one or more processors, the instructions cause the one or more processors to perform steps, the steps further comprising: displaying a list of one or more routes on the GUI; receiving a selection of a route from the list; assigning the route to the autonomous aerial vehicle represented by the first icon on the GUI; As the autonomous aerial vehicle advances along the route, periodically updating the GUI to display a relative position of the first icon relative to a route assigned to the autonomous aerial vehicle; as well as On the GUI, a completed portion of the route is displayed using a first visual cue, and a remaining portion of the route is displayed using a second visual cue.

15. The system of claim 12, wherein when the instructions are executed by the one or more processors, the instructions cause the one or more processors to perform steps, the steps further comprising: displaying a flight guide pane on the GUI; receiving, on the flight guidance pane, a modification corresponding to a desired change in one or more flight parameters associated with the autonomous aerial vehicle represented by the first icon; as well as A command is transmitted to the autonomous aerial vehicle represented by the first icon to implement the desired change.

16. The system of claim 12, wherein when the instructions are executed by the one or more processors, the instructions cause the one or more processors to perform steps, the steps further comprising: Flight parameters associated with the autonomous aircraft are displayed adjacent to the first icon representing the autonomous aircraft on the GUI, wherein the flight parameters include one or more of an altitude, a heading, or a speed of the autonomous aircraft.

17. The system of claim 12, wherein when the instructions are executed by the one or more processors, the instructions cause the one or more processors to perform steps, the steps further comprising: receiving a message from the autonomous aerial vehicle indicating a hazard detected by the autonomous aerial vehicle along a route and an alternate route to be followed by the autonomous aerial vehicle; as well as A collision avoidance warning and the alternative route are displayed on the GUI.

18. The system of claim 12, wherein when the instructions are executed by the one or more processors, the instructions cause the one or more processors to perform steps, the steps further comprising: receiving location positioning signals from the plurality of autonomous aerial vehicles; as well as The plurality of icons representing the plurality of autonomous aerial vehicles are displayed on a map based on the location positioning signal.

19. The system of claim 12, wherein when the instructions are executed by the one or more processors, the instructions cause the one or more processors to perform steps, the steps further comprising: receiving flight data messages from the plurality of autonomous aerial vehicles, wherein each flight data message includes one or more flight parameters associated with one of the plurality of autonomous aerial vehicles; as well as The plurality of icons representing the plurality of autonomous aerial vehicles and corresponding one or more flight parameters are displayed on the GUI.

20. One or more computer-readable storage media storing instructions that, when executed on a flight surveillance platform having a server computer, cause the server computer to perform steps comprising: displaying a graphical user interface (GUI) on a display device; Representing a plurality of autonomous aerial vehicles using a plurality of icons on a first pane of the GUI; displaying information associated with each of the plurality of autonomous aerial vehicles in a plurality of sections on a second pane of the GUI; receiving an input that selects one of a first icon among the plurality of icons on the first pane of the GUI or a first segment area among the plurality of segments on the second pane of the GUI; identifying another one of the first icon among the plurality of icons on the first pane of the GUI or the first segment area among the plurality of segment areas on the second pane of the GUI; displaying the first icon and the first segment region using one or more visual cues to distinguish the first icon and the first segment region from other graphical elements; as well as A command or message is transmitted to the autonomous aerial vehicle represented by the first icon on the GUI.

Images