Systems and methods for autonomous monitoring of highly automated vehicle operations

Abstract

The present invention is entitled "System and method for autonomously monitoring the operation of highly automated vehicles". The present invention discloses a system and method for monitoring the operation of autonomous vehicles. For example, a monitoring device receives subsystem data from one or more subsystems of the autonomous vehicle. The device receives context data of the vehicle, which includes one or more situational conditions of the vehicle during operation. The device receives control data from the control system of the vehicle and analyzes the subsystem data, the context data and the control data to determine a control result. The device provides an assessment of the overall control status of the vehicle to a management system.

Description

Technical Field

[0001] Various embodiments of the present disclosure relate generally to monitoring operations and controls in a vehicle, and more particularly, to monitoring control systems and data systems in a vehicle to facilitate making safe, effective, and efficient decisions for vehicle operation. Background Art

[0002] Vehicles of all types are becoming increasingly complex as they are equipped with network connectivity, automated operating components (e.g., "auto-plotting" or "autopilot features"), connected safety and security features, and the like. As processing systems and subsystems make vehicle operation increasingly automated, control and navigation systems make complex decisions that would otherwise be made by a human navigator. Often, these systems are designed to mimic human-machine interaction and decision making. For example, some auto-plotting features of aircraft vehicles are designed to replicate onboard human pilot and operator procedures. As these automated systems become increasingly complex, failures become possible, particularly if the systems are unable to adequately simulate human control responses in complex and unexpected circumstances.

[0003] The present disclosure relates to systems and methods for achieving these goals and interests. Accordingly, the technology discussed herein discloses systems and methods for autonomously monitoring highly automated passenger aircraft operations. Summary of the Invention

[0004] The present disclosure relates to technology including systems, methods, computer-readable media, and system devices for autonomous or near-autonomous monitoring of highly automated vehicle operations.

[0005] For example, the method may include receiving subsystem data from one or more subsystems of an autonomous vehicle by a monitoring device in communication with the autonomous vehicle; receiving context data for the vehicle, the context data including data on one or more situational conditions of the vehicle during operation; receiving control data from a control system of the vehicle via a network; analyzing the subsystem data, the context data, and the control data to determine a control result; and providing an assessment of the overall control state of the vehicle to a management system based on the analysis.

[0006] The system for monitoring the operation of the autonomous vehicle may include a monitoring device having an interface unit configured to receive information, a data storage device (e.g., a memory) storing instructions associated with the vehicle data system, and a processor. The processor may be configured to execute the instructions to: receive subsystem data from one or more subsystems of the autonomous vehicle; receive context data for the vehicle, wherein the context data includes data about one or more contextual conditions of the vehicle during operation; receive control data from a control system of the vehicle; analyze the subsystem data, the context data, and the control data to determine a control result; and provide an assessment of the overall control state of the vehicle to a management system.

[0007] The non-transitory computer-readable medium stores instructions for causing the monitoring system to perform operations. The instructions may cause the processor to receive subsystem data from one or more subsystems of the autonomous vehicle; receive context data for the vehicle, wherein the context data includes data of one or more situational conditions of the vehicle during operation; receive control data from a control system of the vehicle; analyze the subsystem data, the context data, and the control data to determine a control result; and provide an assessment of the overall control state of the vehicle to a management system based on the analysis.

[0008] Additional objects and advantages of the disclosed embodiments will be set forth in part in the following description and in part will be obvious from the description, or may be learned by practice of the disclosed embodiments. The objects and advantages of the disclosed embodiments will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.

[0009] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

[0011] Figure 1 A network environment including a system for aircraft communications and automatic aircraft control monitoring according to techniques presented herein is shown.

[0012] Figures 2A to 2C An exemplary embodiment of a system for automatic aircraft control monitoring according to the techniques presented herein is shown.

[0013] Figure 3An exemplary flow chart illustrating a method for monitoring operation of an autonomous vehicle according to the techniques presented herein is shown.

[0014] Figure 4 An exemplary apparatus for monitoring the operation of an autonomous vehicle according to the techniques presented herein is shown. DETAILED DESCRIPTION

[0015] Various embodiments of the present disclosure relate generally to monitoring autonomous vehicles and, more particularly, to monitoring operations in an aircraft to assess aircraft control performance.

[0016] Generally speaking, the present disclosure relates to the use of automated aircraft systems acting as pilot monitoring systems to provide a level of supervisory redundancy for flight control and operations. Traditionally, airborne operations are divided into components to be performed by various crew members with different responsibilities. Typical tasks, responsibilities, and delegation of authority (e.g., in a commercial airline environment) are divided into a pilot-in-flight / pilot-in-command (hereinafter referred to as "PF" and "PIC") component and a pilot-in-monitoring (hereinafter referred to as "PM") component. Regardless of the crew member's title or rank (e.g., captain or first officer), there is a clear division of responsibilities between the crew members. Typically, the designated PF assumes primary responsibility for aircraft handling, while the PM monitors aircraft and subsystem performance, handles communications, and generally oversees appropriate procedures. The PM is generally expected to alert the PF in the event of a system malfunction or failure, query the PF in the event of an erroneous or unexpected control action, and recommend or work with the PF to determine the next appropriate action in a given scenario. Thus, the PM provides redundancy for the PF, with the primary purpose of combining and integrating the situational awareness of both the PF and PM into a shared mental model. This is intended to facilitate safe and effective decision-making regarding aircraft operations.

[0017] In modern aviation passenger systems, the responsibilities of the crew are becoming increasingly automated. For example, automated vehicle systems are used to perform operations that are typically performed by the PF. These systems may be referred to as automated flight or "AF" systems in this article. AF systems can utilize artificial intelligence (AI) and machine learning (ML) data analysis and processes to control the aircraft. In such scenarios, the human pilot / aircraft operator may have a more passive role than the human PM in a two-person crew. However, a shared mental model between the AF and monitoring systems is still desirable for the efficient operation of the aircraft. For example, in an urban air mobility (UAM) environment, a software architecture that supports automated crews that are restricted to no human supervision is needed. An independent automation architecture that mimics current manned deck operations provides a certain degree of redundancy in awareness and decision-making, which will enhance the safety and efficiency of semi-automated and highly automated flights.

[0018] Therefore, an automated system is needed to act as a PM to supplement the AF system. This automated system may be referred to herein as an automated monitoring or "AM" system. Similar to the AF system, the AM system may utilize AI and ML analytics and processes to provide monitoring capabilities. When the AM is implemented to act as a PM, the AM may be configured to provide the same degree of supervisory redundancy as in current commercial flight operating procedures. The technology herein presents an example of an AF / AM system. In one example, the AF / AM system consists of a hierarchical pervasive monitoring system to evaluate individual aircraft subsystems, the decision state of the AF, aircraft attitude and environmental factors, and overall system-level intelligent data integration to provide advice and recommendations to the AF as needed.

[0019] The terms used below are to be interpreted in their broadest reasonable manner, even when used in conjunction with certain specific exemplary embodiments of the present disclosure. Indeed, certain terms may even be emphasized below; however, any term intended to be interpreted in any limited manner will be explicitly and specifically defined in this detailed description. The foregoing general description and the following detailed description are merely exemplary and illustrative and are not intended to limit the features protected by the claims.

[0020] As used herein, the terms "comprises," "includes," "has," "contains" or variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements may not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0021] In this disclosure, relative terms such as, for example, "about," "substantially," "generally," and "approximately" are used to indicate a possible variation of ±10% from the specified value.

[0022] The term “exemplary” is used in the sense of an “example” rather than an “ideal.” As used herein, the singular forms “a,” “an,” and “the” include plural references unless the context dictates otherwise.

[0023] Although the present disclosure describes the systems and methods with reference to aircraft, it should be understood that the systems and methods of the present disclosure are applicable to the security of any vehicle management system, including the security of drones, automobiles, trains (locomotives), or any other autonomous and / or internet-connected vehicle management systems.

[0024] Now refer to Figure 1 , which illustrates an exemplary networking environment and systems / infrastructure for aircraft communications and automatic aircraft control monitoring according to the techniques presented herein. Figure 1 An infrastructure 100 for monitoring aircraft systems is shown. Figure 1 An aircraft 102 is shown. Figure 1 Also shown are several systems in infrastructure 100. Some of the systems are onboard aircraft 102, and some are external to aircraft 102. Figure 1 1 , the systems on the aircraft include a communication system 104, a control system 106, a monitoring system 108, and a subsystem 110. The aircraft 102 also has onboard user equipment 112 and onboard users 114 (eg, a pilot or crew member). Figure 1 Aircraft 102 is shown in communication with network 116 via communication system 104 . Figure 1 Also shown are off-board systems 118 , off-board user equipment 120 , and off-board users 122 (eg, an off-board pilot, crew member, or controller).

[0025] It should be understood that although Figure 1 Certain systems are shown as being on board aircraft 102, but one or more of these systems may be external to aircraft 102. For example, one or more of control system 106, monitoring system 108, and subsystem 110 may be on board aircraft 102. Figure 1 102, one or more (or even all) of the control system 106, monitoring system 108, and subsystem 110 may be located outside of the aircraft 102. It should be understood that any of the systems described herein may be stored on one or more physical devices ( Figure 1 The logic and / or computer-readable instructions (e.g., software or firmware) reside on one or more physical devices (not shown). The logic and / or computer-readable instructions can be executed by a processor on the one or more physical devices to perform the functions of the system (including the systems and methods described herein).

[0026] exist Figure 1 , the control system 106, the monitoring system 108, and the subsystems 110 communicate with each other and with the communication system 104. The monitoring system 108 also communicates with an onboard user device 112, which is configured to communicate with an onboard user 114. The onboard user device 112 can be any type of computer device configured to communicate with the onboard user 114 directly or via a network (e.g., a local area network, a wide area network, etc.). The onboard user device 112 can have one or more displays and / or user interfaces (UIs) to enable interaction with the onboard user 114. For example, the onboard user device 112 can be a known or heretofore contemplated computing device, smartphone, tablet computer, panel display, etc.

[0027] The communication system 104 is configured to send and receive data with an off-board system 118 via the network 116. The off-board system 118 is configured to process these communications and to send and receive data with an off-board user device 120. The off-board user device 120 may have one or more displays and / or UIs to enable interaction with an off-board user 122 and may be a similar computing device as described with respect to the on-board user device 112.

[0028] As described above, the automated vehicle system can perform control operations that are typically performed by the PF in traditional flight. Figure 1 In the present invention, these control operations are represented by the control system 106. The control system 106 may also be referred to herein as an AI control system or an AI vehicle control system. In one example, the control system 106 is intended to perform as the AF system described above to control the operation of the aircraft 102. Similarly, the monitoring system 108 may be referred to herein as an AI monitoring system or an AI vehicle monitoring system. In one example, the monitoring system 108 is intended to perform as an AM system to provide the same degree of supervisory redundancy as human commercial flight / vehicle operations. The techniques described herein describe how the monitoring system 108 and the control system 106 operate to provide an appropriate degree of redundancy and shared mental state between the AF system and the AM system.

[0029] Now see Figure 2A . Figure 2A A first exemplary embodiment of a system 200 for aircraft communication is shown. The system 200 shows a system having the above-mentioned Figure 1 The control system 106 and the monitoring system 108 of the aircraft 202 are shown. Figure 2A Also shown is a subsystem 110 having a plurality of subsystem modules, which are shown with reference numerals 110(1) through 110(7). It should be understood that the subsystem modules 110(1) through 110(7) are not an exhaustive list of the systems / modules that make up the subsystem 110. Additionally, it should be understood that the subsystem modules 110(1) through 110(7) may be logic or computer-readable instructions (e.g., software or firmware) stored on one or more physical devices and executed by a processor to perform the functionality of the modules. Figure 2A Also shown above combined Figure 1 The user device 112 and the onboard user 114, which may be a human crew member (such as a skilled air pilot or an unskilled air pilot). Figure 2A Shown are an autonomous management system 204 and an emergency / non-nominal handling system (herein "emergency system") 206. Autonomous management system 204 and emergency system 206 communicate with monitoring system 108. Autonomous management system 204 and emergency system 206 may be referred to herein, individually or in combination, as management systems.

[0030] Referring again to subsystems 110(1) to 110(7), Figure 2A An air traffic control (ATC) interaction module 110(1), a terrain avoidance module 110(2), a flight management module 110(3), a vehicle situational awareness module 110(4), a traffic avoidance module 110(5), a weather avoidance module 110(6), and a health and status monitoring module 110(7) are shown. The ATC interaction module 110(1) provides information about communications or instructions exchanged between ATC and the aircraft. The terrain avoidance module 110(2) provides information about the ground terrain along the flight path of the aircraft. The flight management module 110(3) provides information about the current flight status. The visual situational awareness module 110(4) provides information about visibility conditions along the flight path. The traffic avoidance module 110(5) provides information about air traffic along the flight path of the aircraft and in the vicinity. The weather avoidance module 110(6) provides information about weather conditions near the aircraft and in the flight path. The health and status monitoring module 110(7) provides information about the health of the crew. It should be understood that these are merely examples and that the modules may provide additional functionality.

[0031] Modules 110(1) through 110(7) send data and information to control system 106 to help control system 106 make appropriate control decisions for navigation and control of aircraft 202. Modules 110(1) through 110(7) may collect and send information about the situational conditions of aircraft 202 during aircraft operation. For example, control system 106 may receive data from weather avoidance module 110(6), and based on the data, control system 106 may adjust the flight path or other controls of aircraft 202.

[0032] Monitoring system 108 also receives data and information from modules 110(1) through 110(7) of subsystem 110. Monitoring system 108 analyzes information from subsystem 110 and control information received from control system 106 to determine control results and provide an assessment of the overall control state of the aircraft.

[0033] The interaction between the monitoring system 108, the subsystems 110, and the control system 106 enables the monitoring system to operate as a hierarchical pervasive monitoring system (e.g., AM) to evaluate the states and decision making of the various aircraft subsystems, the control system 106 (e.g., AF), the aircraft attitude, and environmental factors, and the intelligent integration of this data at the overall system level to provide suggestions and recommendations to the control system 106 as needed (e.g., suggestions to modify one or more control operations of the aircraft). The monitoring system 108 operates by integrating information from the components of the aircraft system 200 to develop a unique understanding of the aircraft and can be activated or triggered when the control system 106 makes an unexpected control action, or to refrain from taking action when it is appropriate to do so. For example, the control system 106 may steer or deviate from the planned aircraft flight path in response to information obtained from the traffic avoidance module 110(5). The steering may activate the monitoring system 108 to collect data from the modules 110(1)-110(7) in the subsystem 110 and from the control system 106 to verify and evaluate the appropriateness of the control system's actions. In addition, the monitoring system 108 may provide recommendations to the control system 106 and the onboard user 114 (e.g., via the user device 112). The monitoring system 108 may monitor one or more modules 110(1)-110(7) of the subsystem 110 and other contextual flight information to determine whether the control system 106 should take corrective action. For example, the monitoring system 108 may determine that the control system 106 should take corrective action when the monitoring system 108's analysis of a control result deviates from an expected control result based on information received from one or more of the modules 110(1)-110(7) or the control system 106, or both.

[0034] In one example, the monitoring system 108 receives a continuous stream of diagnostic information from one or more of the modules 110(1) to 110(7) and environmental data from aircraft sensors and external sources (some of which may not be represented by the modules 110(1) to 110(7)). The monitoring system 108 converts the data into a standard format and may aggregate information and trends from the data so that the monitoring system 108 can be configured to sense concurrent trends and anomalies in multiple subsystems and reliably predict the probability of future faults and failures. The monitoring system 108 can then combine this information with empirical data based on flight history, a database of known and predicted risks associated with concurrent and cascading faults and failures, and preloaded troubleshooting guidance to determine one or more courses of action to avoid the fault or failure. The possible actions can be prioritized and provided to the control system 106 (e.g., via the autonomy management system 204 or the emergency system 206) in a standardized format. The control system 106 makes a final decision based on the context and scenario-specific knowledge and executes the determined action. This process is consistent with current operating practices, where a human PM collates multiple diagnostic data sources and presents knowledge recommendations to the PF for final decision and subsequent control action.

[0035] Monitoring system 108 can utilize these determinations, evaluations, and recommendations to update autonomous management system 204 and emergency response system 206. In one example, autonomous management system 204 is a supervisory monitor that utilizes machine learning and / or decision logic to determine the root cause of a failure and output a diagnosis and recommendations regarding an appropriate course of action. Autonomous management system 204 is aware of monitoring system 108 by aggregating input from diagnostics of subsystems 110. Thus, autonomous management system 204 is used to support the evaluation of control system 106 and the decision logic employed in the event of non-nominal system conditions determined by emergency response system 206. Thus, autonomous management system 204, together with monitoring system 108, provides supervisory situational awareness.

[0036] In one example, the monitoring system 108 communicates with the control system 106 when any of the following information exists: an informed inference about the overall condition of the aircraft 202, a detection of an abnormal control activity performed by the control system 106, a prompt to take flight control in response to a detected physical, communication, or environmental clue, and / or a recommendation to address a non-nominal condition or situation. The monitoring system 108 transmits necessary and relevant information about the overall system condition, environmental condition, and / or current control system 106 state to the autonomy management system 204 and for display by a device (e.g., user device 112) to an onboard user 114, or by an onboard user device 120 to an offboard user 122 (e.g., Figure 1). Autonomy management system 204 sends the information to emergency system 206. In one example, monitoring system 108 may send the information to autonomy management system 204, emergency system 206, and / or control system 106 to indicate a modification to one or more control operations of aircraft 202.

[0037] Now refer to Figure 2B , which shows a second exemplary embodiment of a system 250 for aircraft communications. The system 250 shows a similar Figure 2A The modified parts. Figure 2B In FIG, an aircraft 252 has onboard components including a control system 106, a monitoring system 108, and a subsystem 110 including modules 110(1) to 110(7). Figure 2B In FIG, autonomy management system 204 and emergency system 206 are located outside of aircraft 252 and communicate with monitoring system 108 via network 116 and communication system 104 on aircraft 252 . Figure 2B Also shown is an off-board user 122 (e.g. Figure 1 ) and the onboard user equipment 120 (such as Figure 1 The external user may be a ground pilot or ground controller of the aircraft.

[0038] exist Figure 2B In FIG, communication between the monitoring system 108 and the autonomous management system 204 and the emergency system 206 occurs via network communication. Figure 2B The monitoring system 108 supports the use of a combination of Figure 2A The same or similar methods described above evaluate the control system 106 and share this information with off-board components of the system 250 via the network 116. This enables a "ground" controller to receive an evaluation of the monitoring system 108 (eg, via off-board user equipment 120).

[0039] Now refer to Figure 2C , which shows a third exemplary embodiment of a system 270 for aircraft communications. System 270 shows a plurality of aircraft 272(a) through 272(n). Each of aircraft 272(a) through 272(n) has corresponding onboard subsystems and control systems (e.g., subsystem 110(a) and control system 106(a) for aircraft 272(a), subsystem 110(b) and control system 106(b) for aircraft 272(b), etc.). As described herein, each aircraft also has a communication system, Figure 2C Not shown in the figure. Figure 2CThe monitoring system 108 in FIG. 1 is shown as being external to the aircraft 272(a) through 272(n), but it should be understood that each aircraft 272(a) through 272(n) may have its own monitoring system that performs the functions described herein. Figure 2C As shown, the monitoring system 108 is offboard and is configured to perform the same or similar methods for evaluating the plurality of control systems 106(a) to 106(n). The monitoring system 108 can be configured to communicate with the autonomy management system 204 and the emergency system 206, and can ultimately provide information and updates to offboard users 122 (e.g., ground dispatch 203) via user devices 120, as described in the techniques herein. Figure 2C The system 270 in enables "ground" dispatch to receive control assessments from the monitoring systems 108 of a plurality of aircraft 272(a) to 272(n) for efficient control monitoring and decision making for one or more autonomous aircraft.

[0040] exist Figures 2A to 2C In the embodiment of the present invention, the monitoring system can communicate to a user (e.g., an onboard user or an offboard user) via an existing diagnostic or navigation display or a separate display with an automated assembly component. The diagnostic or navigation display or separate display can be provided by Figures 2A to 2C 122 in the user device 112 / 122. For example, a diagnostic or navigation display may have a multimodal component (tactile or voice) to prompt the operator / user to view the display. The operator / user will need to approve certain control actions and navigation changes, and should be alerted if an impending fault or failure is unlikely to be resolved before it occurs. Decision logic will be used to determine when and whether to alert the operator. As an example of this feature, when the battery is depleting at a faster-than-expected rate, the monitoring system 108 may alert the control system 106 to shut down non-essential electronics in the aircraft's cockpit. If this action is expected to mitigate the condition with sufficient reserve power before landing, the operator may not be alerted unless the operator is affected by the change. If battery power continues to deplete rapidly or if depletion accelerates, the operator may be alerted that the aircraft may need to find an alternative suitable heliport or airport for landing and maintenance. Thus, the monitoring device 108 may enable the transmission of display messages to the display device of the aircraft's user, and these display messages may include information regarding control evaluations and recommendations for modifying control operations.

[0041] Now refer to Figure 3, which shows a flowchart 300 of an exemplary method for monitoring the operation of an autonomous vehicle. At step 302, a monitoring device 108 in communication with an autonomous vehicle (e.g., aircraft 102, 202, 252 and / or 272(a) to 272(n)) receives subsystem data from one or more subsystems of the vehicle. The subsystem data can be data from one or more of modules 110(1) to 110(7) or other components of subsystem 110. At step 304, the monitoring device 108 receives context data for the vehicle. The context data includes data of one or more situational conditions of the vehicle during operation. At step 306, the monitoring device 108 receives control data from the control system of the vehicle (e.g., control system 106), and at step 308, the monitoring device analyzes the subsystem data, context data, and control data to determine control results. At step 310, an assessment of the overall state of the vehicle is provided to the management system. The management system may be any one or more of the autonomous management system 204 , the emergency system 206 , and / or the control system 106 .

[0042] Figure 4 is a simplified functional block diagram of a device 400 according to an exemplary embodiment of the present disclosure, which may be configured as Figure 1 and Figures 2A to 2C Any of the systems in to perform the techniques described herein.

[0043] Specifically, in one embodiment, any of the aircraft systems (including communication system 104, control system 106, monitoring system 108, subsystem 110, offboard system 118, autonomy management system 204, and emergency system 206) may be a component of device 400. In one example, Figure 4The device 400 in FIG. 4 shows an interface unit 402, a processor 404, and a memory unit 406. The interface unit 402 is configured to send and receive data to one or more systems. For example, when the device 400 represents a device that executes logic for monitoring the functions of the device 108, the interface unit can send and receive messages to the control system 106, the subsystem 110, and the communication system 102. The processor 404 can be a central processing unit in the form of one or more processors for executing program instructions. The device 400 may include an internal communication bus and a memory 406 configured for program storage and data storage of various data files to be processed and / or transmitted, although the device 400 may receive programming and data via network communications. The memory 406 includes control and monitoring logic 450, which includes instructions that, when executed by the processor 404, cause the processor to perform operations of the monitoring system 108 or other systems described herein. The device 400 may also include input and output ports for connecting to input and output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Of course, various system functions can be implemented in a distributed manner on multiple similar platforms to distribute the processing load. Alternatively, the system can be implemented by appropriate programming of one computer hardware platform.

[0044] Any suitable system infrastructure can be implemented to allow assessment model monitoring equipment. In one embodiment, any one in disclosed system, method and / or graphical user interface can be performed or realized by the consistent or similar computing system with the specification sheets of this paper. Although not necessary, in the background of computer executable instructions, various aspects of the present disclosure are described, such as by data processing equipment, the routine that server computer, wireless device and / or personal computer are performed. Those skilled in the art will appreciate that various aspects of the present disclosure can be put into practice using other communications, data processing or computer system configurations, including internet devices, handheld devices (including personal digital assistants (" PDAs ")), wearable computers, various cellular phones or mobile phones (including voice over IP (" VoIP ") phones), dumb terminals, media players, gaming devices, virtual reality equipment, multiprocessor systems, based on microprocessor or programmable consumer electronics, set-top boxes, network PCs, microcomputers, mainframe computers etc. In fact, terms "computer", "server" etc. are usually used interchangeably herein, and refer to any of the above-mentioned devices and systems and any data processor.

[0045] Aspects of the present disclosure may be implemented in a special-purpose computer and / or data processor that is specifically programmed, configured, and / or constructed to perform one or more computer-executable instructions described in detail herein. Although aspects of the present disclosure, such as certain functions, are described as being performed only on a single device, the present disclosure may also be practiced in a distributed environment where functions or modules are shared between different processing devices linked by a communication network, such as a local area network ("LAN"), a wide area network ("WAN"), and / or the Internet. Similarly, the technology presented herein as involving multiple devices may be implemented in a single device. In a distributed computing environment, program modules may be located in a local memory storage device and / or a remote memory storage device.

[0046] Aspects of the present disclosure may be stored and / or distributed on non-transitory computer-readable media, including magnetic or optically readable computer disks, hard-wired or pre-programmed chips (e.g., EEPROM semiconductor chips), nanotechnology memories, biological memories, or other data storage media. Alternatively, computer-implemented instructions, data structures, screen displays, and other data under various aspects of the present disclosure may be distributed over a period of time on propagation signals on a propagation medium (e.g., one or more electromagnetic waves, acoustic waves, etc.) over the Internet and / or over other networks (including wireless networks), and / or they may be provided over any analog or digital network (packet switching, circuit switching, or other schemes).

[0047] The procedural aspects of the technology can be considered as a "product" or "article of manufacture," typically in the form of executable code and / or associated data, which is carried or embodied in a type of machine-readable medium. "Storage" type media include any or all tangible memories of a computer, processor, etc., or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which can readily provide non-transitory storage for software programming. All or part of the software can sometimes be communicated via the Internet or various other telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another, such as from a management server or host of a mobile communication network to a server's computer platform and / or from a server to a mobile device. Therefore, another type of medium that can carry software elements includes optical waves, radio waves, and electromagnetic waves, such as those used on physical interfaces between local devices, through wired and optical ground networks, and through various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be considered as media that carry software. As used herein, unless restricted to non-transitory, tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0048] Although the methods, devices, and systems disclosed herein are described with exemplary reference to transmitting data, it should be understood that the embodiments disclosed herein are applicable to any environment, such as desktop or laptop computers, car entertainment systems, home entertainment systems, etc. In addition, the embodiments disclosed herein are applicable to any type of Internet protocol.

[0049] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

Claims

1. A method for monitoring the operation of an autonomous vehicle, the method comprising: receiving, by a monitoring device, subsystem data from one or more subsystems of the autonomous vehicle; receiving, by the monitoring device, contextual data of the vehicle, wherein the contextual data includes data of one or more situational circumstances of the vehicle during operation; transmitting, by the monitoring device, data related to the one or more possible actions to a management system, wherein the management system comprises an autonomous management system and an emergency system, and wherein the management system formats the data related to the one or more possible actions and transmits the formatted data to the control system; receiving, by the monitoring device, control data from a control system of the vehicle based on the formatted data; analyzing the subsystem data, the background data, and the control data by the monitoring device to determine a control result; as well as Based on the determined control result, the monitoring device provides an assessment of the overall control status of the vehicle to a management system. 2 . The method of claim 1 , further comprising providing one or more recommendations to the management system indicating modifications to one or more control operations of the vehicle. 3 . The method of claim 2 , further comprising sending a display message to a display device of an operator of the vehicle, wherein the display message includes information regarding the evaluation and the recommendation to modify the control operation. The method of claim 1 , wherein the vehicle is an autonomous flying vehicle. The method of claim 1 , wherein receiving the context data comprises receiving environmental data from a sensor of the vehicle and external data of the operation of the vehicle as the context data. 6 . The method of claim 5 , wherein the external data includes incoming control commands for the vehicle, weather information, deviations from a planned path of the vehicle, and traffic. 7 . The method of claim 1 , wherein analyzing comprises analyzing the subsystem data, the context data, and the control data when the control result deviates from an expected control result.

8. A monitoring device for monitoring the operation of an autonomous vehicle, the monitoring device comprising: an interface unit configured to receive information; a data storage device storing instructions associated with a vehicle data system; as well as a processor configured to execute the instructions to: receiving subsystem data from one or more subsystems of the autonomous vehicle; receiving contextual data for the vehicle, wherein the contextual data includes data of one or more situational circumstances of the vehicle during operation; sending data related to one or more possible actions to a management system, wherein the management system includes an autonomous management system and an emergency system, and wherein the management system formats the data related to the one or more possible actions and sends the formatted data to the control system; receiving control data from a control system of the vehicle based on the formatted data; analyzing the subsystem data, the background data, and the control data to determine a control result; as well as Based on the determined control results, an evaluation of the overall control status of the vehicle is provided to the management system.

9. The monitoring device of claim 8, wherein the processor is further configured to execute the instructions to provide one or more recommendations to the management system indicating modifications to one or more control operations of the vehicle.

10. The monitoring device of claim 9, wherein the processor is further configured to execute the instructions to send a display message to a display device of an operator of the vehicle, wherein the display message includes information regarding the evaluation and the recommendation to modify the control operation.

Images