System and method for visualizing automated actions using holographic agents

Abstract

A system and method are provided for visualizing automated actions using holographic agents. [Solution] A selection of the holographic format of a holographic agent is received. An auto-control notification associated with an auto-control action is received in the first time. The auto-control action is scheduled to be performed by the vehicle's autopilot system in the second time. An agent action is generated based at least in part on the auto-control action. A command is sent to the hologram generation system in the first time to generate a holographic agent in the selected holographic format involved in the agent action. The agent action is a visual representation of the autopilot system performing the auto-control action prior to the performance of the first auto-control action in the second time.

Description

Technical Field

[0001] The present invention generally relates to the operation of an aircraft, and more particularly, to a system and method for visualizing automatic actions using a holographic agent.

Background Art

[0002] Face-to-face conversations among crew members typically use constant gestures, expressions, and vocalizations to communicate the intent and execution of vehicle operation actions. Automatic systems often communicate the intent to perform an automatic action via individual mode notifications and flat displays. The performance of automatic actions by the cockpit and avionics control systems can be difficult to trust, monitor, and interact with under difficult vehicle operating conditions. Notifications associated with automatic actions are sometimes troubled by fundamental situations that pose problems due to the lack of information associated with the state of the vehicle system and crew intervention that stops or interferes with the automatic action.

[0003] Therefore, there is a need for a system and method for visualizing automatic actions using a holographic agent.

Summary of the Invention

[0004] This summary is provided to present selected concepts in a simplified form that are further described in the "Detailed Description of the Invention" below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In various embodiments, a system for visualizing automated actions using a holographic agent includes at least one processor and at least one memory communicably coupled to the at least one processor. The at least one memory includes instructions, which, when executed by the at least one processor, cause the at least one processor to receive a selection of a holographic form of a holographic agent via a user interface, receive a first auto-control notification associated with a first automated control action at a first time, the first automated control action being scheduled to be performed by the vehicle's autopilot system at a second time, the second time being after the first time, receive a first agent action based at least in part on the first automated control action, and send a first command to a hologram generation system to generate a holographic agent in the selected holographic form involved in the first agent action at the first time, the first agent action being a visual representation of the autopilot system performing the first automated control action prior to the performance of the first automated control action at the second time.

[0006] In various embodiments, a method for visualizing an automated action using a holographic agent includes receiving a selection of a holographic form of a holographic agent via a user interface; receiving a first automated control notification associated with a first automated control action at a first time, the first automated control action being scheduled to be performed by the vehicle's autopilot system at a second time, the second time being after the first time; generating a first agent action at least in part with the first automated control action; and transmitting a first command to a hologram generation system to generate a holographic agent in the selected holographic form involved in the first agent action at the first time, the first agent action being a visual representation of the autopilot system performing the first automated control action before the second time.

[0007] In various embodiments, an aircraft including a system for visualizing automated actions using a holographic agent includes at least one processor and at least one memory communicably coupled to the at least one processor. The at least one memory includes instructions, which, when executed by the at least one processor, cause the at least one processor to receive a selection of a holographic form of a holographic agent via a user interface, receive a first auto-control notification associated with a first automated control action at a first time, the first automated control action being scheduled to be performed by the vehicle's autopilot system at a second time, the second time being after the first time, receive a first agent action based at least in part on the first automated control action, and send a first command to a hologram generation system to generate a holographic agent in a selected holographic form involved in the first agent action at the first time, the first agent action being a visual representation of the autopilot system performing the first automated control action prior to the performance of the first automated control action at the second time.

[0008] Furthermore, other desirable functions and features of systems and methods for visualizing automated actions using holographic agents will become apparent from the subsequent detailed description and the attached claims, in conjunction with the attached drawings and the aforementioned background. [Brief explanation of the drawing]

[0009] The present invention may be described hereafter in conjunction with the figures shown below, where similar numbers represent similar elements. [Figure 1] This is a block diagram representation of a system configured to perform visualization of automated actions using a holographic agent, according to at least one embodiment. [Figure 2]This is a block diagram representation of an aircraft including an automated action visualization system, according to at least one embodiment. [Figure 3] This is a flowchart representation of a method for performing visualization of automated actions using a holographic agent, according to at least one embodiment. [Figure 4] This is an illustrative diagram of a holographic agent in the form of a holographic hand in the cockpit of an aircraft, according to at least one embodiment. [Figure 5] This is an illustrative diagram of a holographic agent in the form of a holographic symbol in the cockpit of an aircraft, according to at least one embodiment. [Modes for carrying out the invention]

[0010] The following detailed descriptions are essentially illustrative. As used herein, the term “exemplary” means “serving as an example, case, or illustration.” Therefore, any embodiment described herein as “exemplary” should not necessarily be construed as being preferable or advantageous to other embodiments. All embodiments described herein are exemplary embodiments provided to enable those skilled in the art to construct or use the invention and do not limit the scope of the invention as defined by the claims. Furthermore, it is not intended to be bound by any express or implied theories presented in the preceding “Technical Field,” “Background Art,” “Summary of the Invention,” or the following “Modes for Carrying Out the Invention.”

[0011] Referring to Figure 1, a block diagram representation of a system 10 (hereinafter abbreviated as "System" 10) configured to perform visualization of automated actions using a holographic agent, according to at least one embodiment. System 10 may be mounted and utilized on a mobile platform 5 as described herein. In various embodiments, the mobile platform is an aircraft that mounts or is equipped with System 10. As schematically shown in Figure 1, System 10 includes the following components or subsystems, each of which may take the form of a single device or a plurality of interconnected devices, and includes a controller circuit 12 operably coupled to at least one display device 14, a computer-readable storage medium or memory 16, an optional input interface 18, and an aircraft data source 20 including, for example, a flight management system (FMS) 21 and an array of flight system status and geospatial sensors 22.

[0012] In various embodiments, System 10 may be separate from or integrated with the Flight Management System (FMS) 21 and / or the Flight Control System (FCS). Although schematically shown as a single unit in Figure 1, the individual elements and components of System 10 may be implemented in a distributed manner using any practical number of physically separate, operably interconnected hardware or devices. When System 10 is used as described herein, the various components of System 10 are typically all mounted and arranged on the Mobile Platform 5.

[0013] The term “controller circuit” (and its abbreviated form “controller”) broadly encompasses the components used to perform or support the processing functions of System 10. Thus, the controller circuit 12 may include, or be associated with, any number of individual processors, a flight control computer, navigation equipment, computer-readable memory (including or in addition to memory 16), a power supply, a storage device, an interface card, and other standardized components. In various embodiments, the controller circuit 12 embodies one or more processors operably coupled to a data storage device storing at least one firmware or software program (generally computer-readable instructions embodying algorithms) for performing the various process tasks, calculations, and control / display functions described herein. During operation, the controller circuit 12 is programmed with, for example, program 30, at least one firmware or software program, if the mobile platform 5 is an aircraft, which embodies the algorithms described herein to perform visualization of automated actions using a holographic agent according to at least one embodiment on the mobile platform 5, and can execute the program, thereby performing various process steps, tasks, calculations, and control / display functions described herein.

[0014] In this embodiment, the controller circuit 12 can exchange data, including real-time wireless data, with one or more external sources 50 to support the operation of the system 10. In this case, bidirectional wireless data exchange may take place over a communication network such as a public or private network implemented according to the Transmit Control Protocol / Internet Protocol Architecture or other conventional protocol standards. Encryption and mutual authentication techniques may be applied as appropriate to ensure data security.

[0015] Memory 16 is a data storage device that can contain any number and type of storage media suitable for storing computer-readable code or instructions, such as the aforementioned software program 30, and other data that generally supports the operation of the system 10. Memory 16 can also store the values ​​of one or more thresholds 34 used by algorithms incorporated into the software program 30. One or more databases 28 are other forms of storage media. These may be integrated with memory 16 or may be separate from memory 16.

[0016] In various embodiments, aircraft-specific parameters and information of an aircraft may be stored in memory 16 or database 28 and referenced by program 30. Non-limiting examples of aircraft-specific information include the aircraft's weight and dimensions, performance, and configuration options.

[0017] The flight parameter sensor and geospatial sensor 22 supply various types of data or measurements to the controller circuit 12 during the aircraft's flight. In various embodiments, the geospatial sensor 22 supplies, without limitation, one or more of the following: inertial reference system measurements providing position, flight path angle (FPA) measurements, airspeed data, ground speed data (including ground speed direction), vertical speed data, vertical acceleration data, altitude data, attitude data including pitch and roll measurements, yaw data, heading information, sensed atmospheric condition data (including wind speed and wind direction data), flight path data, flight trajectory data, radar altitude data, and geometric altitude data.

[0018] Continuing to refer to Figure 1, the display device 14 may include any number and type of image generating devices capable of generating one or more avionics displays 32. When the system 10 is used in a manned aircraft, the display device 14 may be mounted on the static structure of the aircraft cockpit, for example, as a head-down display (HDD) or head-up display (HUD) unit. In various embodiments, the display device 14 may take the form of a movable display device (e.g., a pilot-worn display device) or a portable display device such as an electronic flight bag (EFB), laptop, or tablet computer brought into the aircraft cockpit by the pilot.

[0019] At least one avionics display 32 is generated on the display device 14 during the operation of the system 10. The term “avionics display” is synonymous with the terms “aircraft-related display” and “cockpit display” and includes displays generated in text, graphics, maps, and other formats. The system 10 can generate various types of lateral and vertical avionics displays 32 on which map views and symbols, text notices, and other graphics related to flight plans are presented for the pilot to view. The display device 14 is configured to continuously render at least a lateral display showing the aircraft's current position in map data. The avionics displays 32 generated and controlled by the system 10 may include graphical user interface (GUI) objects and alphanumeric input displays of the type typically presented on the screens of multifunction control display units (MCDUs) and control display units (CDUs) in general. Specifically, embodiments of the avionics display 32 include one or more two-dimensional (2D) avionics displays, such as a horizontal (i.e., lateral) navigation display or a vertical navigation display (i.e., a vertical situation display, VSD), and / or one or more three-dimensional (3D) avionics displays, such as a Primary Flight Display (PFD) or an exocentric 3D avionics display.

[0020] In various embodiments, the human-machine interface is implemented as an integration of a pilot input interface 18 and a display device 14. In various embodiments, the display device 14 is a touch screen display. In various embodiments, the human-machine interface also generally includes a separate pilot input interface 18 (such as a keyboard, cursor control device, or voice input device) operably coupled to the display device 14. Through various display and graphics system processes, the controller circuit 12 commands and controls the touch screen display device 14 to generate various graphical user interface (GUI) objects or elements described herein, including, for example, buttons, sliders, etc., used to prompt a user to interact with the human-machine interface to provide user input, and the controller circuit 12 can activate their respective functions and provide user feedback in response to user input received in the GUI elements. In at least one embodiment, the human-machine interface is implemented via a holographic agent generated by a hologram generation system.

[0021] In various embodiments, system 10 may also include a dedicated communication circuit 24 configured to provide real-time two-way wired and / or wireless data exchange for the controller 12 to communicate with external sources 50 (including each of traffic, air traffic control (ATC), satellite weather sources, terrestrial stations, etc.). In various embodiments, the communication circuit 24 may include a public or private network implemented according to Transmission Control Protocol / Internet Protocol architecture and / or other conventional protocol standards. Encryption and mutual authentication techniques may be applied as appropriate to ensure data security. In some embodiments, the communication circuit 24 is integrated within the controller circuit 12, and in other embodiments, the communication circuit 24 is external to the controller circuit 12. When the external source 50 is "traffic", the communication circuit 24 can incorporate software and / or hardware for communication protocols as needed for traffic collision avoidance system (TCAS), automatic dependent surveillance-broadcast (ADS-B), and enhanced vision system (EVS).

[0022] In certain embodiments of system 10, the controller circuit 12 and other components of system 10 may be integrated into or cooperate with any number and type of systems that are typically deployed and installed on an aircraft, for example, including FMS 21.

[0023] The disclosed algorithms are embodied in a hardware program or a software program (e.g., program 30 within the controller circuit 12) and are configured to operate regardless of the flight phase of the aircraft.

[0024] In various embodiments, the provided controller circuit 12, and thus its program 30, receives a selection of a holographic form of a holographic agent via a user interface, receives a first automatic control notification associated with a first automatic control action scheduled to be performed at a first time by an autopilot system of a vehicle, generates a first agent action at least partially based on the first automatic control action, and transmits a first command to a hologram generation system to generate a holographic agent in a selected holographic form that participates in the first agent action at the first time, wherein the first agent action is a visual representation of the performance of the first automatic control action by the autopilot system, and can incorporate programming instructions for performing the foregoing.

[0025] Referring to FIG. 2, a block diagram representation of an aircraft 5 including an automatic action visualization system 200 according to at least one embodiment is shown. The aircraft 5 includes a controller 202. The controller 202 includes at least one processor 204 and at least one memory 206. The at least one memory 206 includes the automatic action visualization system 200. In various embodiments, the controller 202 may include additional components that facilitate the operation of the controller 202.

[0026] The controller 202 is configured to be communicatively coupled to the pilot interface unit 18, the flight management system (FMS) 21, the operator monitoring system 208, the autopilot system 210, and the hologram generation system 212. The pilot interface unit 18 is the same as the pilot interface unit 18 described with reference to Figure 1. The FMS 21 is the same as the FMS 21 described with reference to Figure 1. In at least one embodiment, the operator monitoring system 208 includes one or more cameras. In at least one embodiment, the autopilot system 210 is configured to perform partial automation of the aircraft's operation. In at least one embodiment, the autopilot system 210 is configured to perform full automation of the aircraft's operation.

[0027] In at least one embodiment, the hologram generation system 212 is an augmented reality (AR) system. In at least one embodiment, the hologram generation system 212 is a mixed reality (MR) system. In at least one embodiment, the hologram generation system 212 is a reflective hologram system. In at least one embodiment, the hologram generation system 212 is a transmitting hologram system. In at least one embodiment, the hologram generation system 212 is a hybrid hologram system. The operation of the automated action visualization system 200 is described in more detail below.

[0028] Referring to Figure 3, a flowchart representation of Method 300 for performing visualization of automated actions using a holographic agent is shown. Method 300 is described with reference to an exemplary implementation of the automated action visualization system 200. As can be understood in light of this disclosure, the sequence of operations within Method 300 is not limited to the sequential execution shown in Figure 3, but may be performed in one or more different sequences as appropriate in accordance with this disclosure.

[0029] In 302, the automated action visualization system 200 receives a holographic agent activation command. In at least one embodiment, the automated action visualization system 200 receives a holographic agent activation command from the pilot via the pilot interface unit 18 of the aircraft 5.

[0030] In 304, the automated action visualization system 200 receives a selection of the holographic form of the holographic agent via a user interface. In at least one embodiment, the automated action visualization system 200 receives a selection of the form of the holographic agent via the pilot interface unit 18 of the aircraft 5. Examples of holographic forms include, but are not limited to, anthropomorphic forms, human head forms, human hand forms, text forms, numerical forms, symbolic forms, and instructional image forms.

[0031] In 306, the automatic action visualization system 200 receives time. In at least one embodiment, the automatic action visualization system 200 receives time from the clock of the aircraft 5. In at least one embodiment, the automatic action visualization system 200 receives time from the FMS 21.

[0032] In 308, the automated action visualization system 200 receives the geographic location of the aircraft 5. In at least one embodiment, the automated action visualization system 200 receives the geographic location from the FMS 21. In at least one embodiment, the automated action visualization system 200 receives the geographic location from the automated dependent monitoring broadcast (ADS-B) system. In at least one embodiment, the automated action visualization system 200 receives the geographic location from the geospatial sensor 22.

[0033] In 310, the automated action visualization system 200 receives weather conditions. In at least one embodiment, the automated action visualization system 200 receives weather conditions from the FMS 21. In at least one embodiment, the FMS 21 receives weather conditions from a satellite weather source.

[0034] In 312, the automated action visualization system 200 determines the workload of the vehicle operator. In at least one embodiment, the vehicle operator is the pilot of the aircraft 5. In at least one embodiment, the automated action visualization system 200 receives images of the vehicle operator from the operator monitoring system 208. In at least one embodiment, the operator monitoring system 208 includes one or more cameras mounted on the aircraft 5. One or more cameras transmit images of the vehicle operator, such as the pilot, to the automated action visualization system 200. The automated action visualization system 200 determines the workload of the vehicle operator based on the received images of the vehicle operator. In at least one embodiment, the automated action visualization system 200 receives the flight stage of the aircraft 5 from the FMS 21. The automated action visualization system 200 determines the workload of the vehicle operator based on the flight stage of the aircraft 5.

[0035] In 314, the automated action visualization system 200 determines the state of the vehicle operator. In at least one embodiment, the vehicle operator is the pilot of the aircraft 5. In at least one embodiment, the automated action visualization system 200 receives images of the vehicle operator from the operator monitoring system 208. In at least one embodiment, the operator monitoring system 208 includes one or more cameras mounted on the aircraft 5. One or more cameras transmit images of the vehicle operator, such as the pilot, to the automated action visualization system 200. The automated action visualization system 200 determines the state of the vehicle operator based on the received images of the vehicle operator. Examples of the state of the vehicle operator include the vehicle operator's level of alertness and the vehicle operator's gaze direction.

[0036] In 316, the automatic action visualization system 200 receives the automatic state of vehicle operation from the vehicle's autopilot system. In at least one embodiment, the vehicle is an aircraft 5. Examples of automatic states include, but are not limited to, partially automatic and fully automatic vehicle operation.

[0037] In 318, the automatic action visualization system 200 receives an automatic control notification from the vehicle's autopilot system 210 associated with an automatic control action during a first time. In at least one embodiment, the vehicle is an aircraft 5. The automatic control action is scheduled to be performed by the vehicle's autopilot system 210 during a second time. The second time is after the first time. In at least one embodiment, the automatic control action is an automatic aircraft operation control action scheduled to be performed by the aircraft 5's autopilot system during the second time.

[0038] In 320, the automatic action visualization system 200 generates agent actions based on the automatic control actions received from the autopilot system 210. Based on the automatic control actions in the first time, the automatic action visualization system 200 generates agent actions to be performed via the holographic agent.

[0039] In 322, the automated action visualization system 200 sends a command to the hologram generation system 212 to generate a holographic agent of a selected holographic form involved in the agent action in the first time. The holographic agent provides a visual representation of the execution of the automated control action by the autopilot system 210 via the agent action, prior to the execution of the automated control action by the autopilot system 210 in the second time. By presenting a visual representation of the automated control action using the holographic agent before the actual execution of the automated control action, the pilot is provided with prior notification that the automated control action is scheduled to be executed.

[0040] In at least one embodiment, the automated action visualization system 200 sends a command to the hologram generation system 212 to generate a holographic form of the holographic agent, based in part on one or more of the time received in 306, the geographical location of the aircraft 5 received in 308, the weather conditions received in 310, and the automated state received in 316.

[0041] In at least one embodiment, the automatic action visualization system 200 sends a command to the hologram generation system 212 to generate a holographic agent at a location within the vehicle, based on one or more of the vehicle operator's workload determined in 312 and the vehicle operator's state determined in 314. For example, if the automatic action visualization system 200 determines that the vehicle operator's line of sight is over the clipboard, the automatic action visualization system 200 sends a command to the hologram generation system 212 to generate a holographic agent on the clipboard. The hologram generation system 212 generates a holographic agent that provides a visual representation of the execution of an automated control action via an agent action in a first time. For example, if the automatic action visualization system 200 determines that the vehicle operator has a high workload, the automatic action visualization system 200 sends a command to the hologram generation system 212 to generate a holographic agent at a location in the vehicle that does not hinder the vehicle operator's ability to concentrate on tasks associated with the high workload. Method 300 returns to 306.

[0042] Although the automated action visualization system 200 in Method 300 is described with reference to the vehicle operator of aircraft 5, the automated action visualization system 200 may be used to implement Method 300 for vehicle operators of ground vehicles, vehicle operators of underwater vehicles, and vehicle operators of surface vehicles.

[0043] The automatic action visualization system 200 is configured to receive a first automatic control notification associated with a first automatic control action in a first time period. The automatic action visualization system 200 is configured to receive a first automatic control notification from the autopilot system 210. The first automatic control action is scheduled to be performed by the autopilot system 210 in a second time period. The second time period is after the first time period. The automatic action visualization system 200 is configured to generate a first agent action based on the first automatic control action and to send a first command to the hologram generation system 212 to generate a holographic agent involved in the first agent action in the first time period. The first agent action is a visual representation of the autopilot system's performance of the first automatic control action prior to its performance in the second time period.

[0044] In at least one embodiment, the pilot performs a manual control action via a control interface following the execution of a first automatic control action. A second automatic control action is generated by the autopilot system 210 in response to the manual control action. The automatic action visualization system 200 is configured to receive a second automatic control notification associated with the second automatic control action from the autopilot system 210 at a third time. The second automatic control action is scheduled to be performed by the autopilot system 210 at a fourth time, which is after the third time. The automatic action visualization system 200 is configured to generate a second agent action based on the second automatic control action and to send a second command to the hologram generation system 212 to generate a holographic agent involved in the second agent action at the third time. The second agent action is a visual representation of the autopilot system 210's execution of the second automatic control action prior to its execution at the fourth time.

[0045] In at least one embodiment, the pilot performs a manual control action via the control interface before the execution of the first automatic control action, thereby overriding the execution of the first automatic control action. A second automatic control action is generated by the autopilot system 210 in response to the manual control action. The automatic action visualization system 200 is configured to receive a second automatic control notification associated with the second automatic control action from the autopilot system 210 at a third time. The second automatic control action is scheduled to be performed by the autopilot system 210 at a fourth time, which is after the third time. The automatic action visualization system 200 is configured to generate a second agent action based on the second automatic control action and to send a second command to the hologram generation system 212 to generate a holographic agent involved in the second agent action at the third time. The second agent action is a visual representation of the execution of the second automatic control action by the autopilot system 210 prior to the execution of the second automatic control action at the fourth time. By generating a holographic agent involved in an agent action representing an automated control action before the automated control action is performed, the vehicle operator is given the opportunity to override the automated control action if guaranteed.

[0046] In at least one embodiment, the automatic action visualization system 200 is configured to receive a second automatic control notification associated with a second automatic control action in a first time. The automatic action visualization system 200 is configured to receive a second automatic control notification from the autopilot system 210. The second automatic control action is scheduled to be performed by the autopilot system 210 in a second time. The automatic action visualization system 200 is configured to send a second command to the hologram generation system 212 to generate a second agent action based on the second automatic control action and to generate another holographic agent involved in the second agent action in the first time. The second agent action is a visual representation of the autopilot system's performance of the second automatic control action prior to its performance in the second time. The automatic action visualization system 200 is configured to send a command to the hologram generation system 212 to generate multiple holographic agents involved in agent actions that reflect multiple automatic actions scheduled to be performed simultaneously by the autopilot system 210.

[0047] In at least one embodiment, the automated action visualization system 200 is configured to receive images of the vehicle operator from the operator monitoring system 208. The automated action visualization system 200 is configured to determine the state of the vehicle operator based on the images of the vehicle operator and to activate the operation of the automated action visualization system 200 based on the state of the vehicle operator.

[0048] Referring to Figure 4, an exemplary diagram of a holographic agent in the form of a holographic hand 402 in the cockpit of an aircraft 5 is shown according to at least one embodiment. The vehicle operator is the pilot 404 of aircraft 5. The automatic action visualization system 200 receives a pilot selection for a holographic agent in the form of a holographic hand. The automatic action visualization system 200 receives an automatic control notification from the autopilot system 210 associated with an automatic control action. The automatic control action involves adjusting a control knob in the cockpit. The automatic action visualization system 200 generates an agent action based on the automatic control action. The agent action is a visual representation of the adjustment of the control knob. The automatic action visualization system 200 sends a command to the hologram generation system 212 to generate a holographic agent involved in the agent action of adjusting the control knob. The holographic agent is a holographic hand 402 that provides a visual representation of the adjustment of the control knob by the autopilot system 210 before the autopilot system 210 performs the adjustment of the control knob.

[0049] Referring to Figure 5, an exemplary diagram of a holographic agent in the form of a holographic symbol in the cockpit of aircraft 5 is shown according to at least one embodiment. The automatic action visualization system 200 received a pilot selection of the holographic agent in the form of a holographic symbol. The automatic action visualization system 200 received an automatic control notification from the autopilot system 210 associated with an automatic control action. The automatic control action is adjusting the air / fuel mixture. The automatic action visualization system 200 generated an agent action based on the automatic control action. The agent action is a visual representation of adjusting the air / fuel mixture. The automatic action visualization system 200 sent a command to the hologram generation system 212 to generate a holographic agent involved in the agent action of adjusting the air / fuel mixture. The holographic agent is a holographic symbol 502 that provides a visual representation of the air / fuel mixture adjustment by the autopilot system 210 before the autopilot system 210 performs the air / fuel mixture adjustment. Holographic symbol 502 is a circle around the manual air / fuel mixture control unit and an arrow indicating the direction of movement of the manual air / fuel mixture control unit.

[0050] Those skilled in the art will understand that the various exemplary logic blocks, modules, circuits, and algorithmic processes described in relation to the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. Some embodiments and implementations are described above in terms of functional and / or logical block components (or modules) and various processing steps. However, it should be understood that such block components (or modules) can be realized by any number of hardware, software, and / or firmware components configured to perform a particular function. To clearly demonstrate this compatibility between hardware and software, various exemplary components, blocks, modules, circuits, and processes have been described above in general terms of their function. Whether such functions are implemented as hardware or as software depends on the specific application and the design constraints imposed on the overall system. A skilled technician may implement the described functions in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention. For example, an embodiment of the system or component may use various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., that can perform various functions under the control of one or more microprocessors or other control devices. Furthermore, those skilled in the art will understand that the embodiments described herein are merely illustrative implementations.

[0051] Various exemplary logic blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or run by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, individual gate or transistor logic, individual hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working with a DSP core, or any other such configuration.

[0052] The steps of the methods or algorithms described in relation to the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. The exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. The processor and storage medium may reside in an ASIC.

[0053] Techniques and technologies can be described herein in terms of functions and / or logic block components, with reference to symbolic representations of operations, processing tasks, and functions that can be performed by various computing components or devices. Such operations, tasks, and functions may be referred to as being performed by a computer, computerized, implemented in software, or implemented by a computer. In fact, one or more processor devices can perform the aforementioned operations, tasks, and functions by manipulating electrical signals representing data bits at memory locations in system memory, as well as by processing other signals. The memory locations where data bits are maintained are physical locations having specific electrical, magnetic, optical, or organic properties corresponding to the data bits. It should be understood that the various block components shown in the figures can be realized by any number of hardware, software, and / or firmware components configured to perform a specified function. For example, an embodiment of a system or component may use various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices.

[0054] When implemented in software or firmware, the various elements of the system described herein are essentially code segments or instructions that perform various tasks. Programs or code segments may be stored in a processor-readable medium or transmitted by computer data signals embodied on a carrier wave via a transmission medium or communication channel. “Computer-readable medium,” “processor-readable medium,” or “machine-readable medium” may include any medium capable of storing or transferring information. Examples of processor-readable mediums include electronic circuits, semiconductor memory devices, ROMs, flash memory, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, and high-frequency (RF) links. Computer data signals may include any signals that can propagate via a transmission medium such as an electronic network channel, fiber optics, air, an electromagnetic path, or an RF link. Code segments may be downloaded via computer networks such as the Internet, intranets, or LANs.

[0055] Some of the functional units described herein are referred to as “modules” to particularly emphasize their implementation independence. For example, a function referred to as a module herein may be implemented as a whole or in part as a hardware circuit including a custom VLSI circuit or a gate array, logic chip, transistor, or other individual components, including commercially available semiconductors. A module may also be implemented as a programmable hardware device, such as a field-programmable gate array, programmable array logic, or programmable logic unit. A module may also be implemented as software for execution on various types of processors. For example, an identified module of executable code may include one or more physical or logical modules of computer instructions, which can be organized, for example, as objects, procedures, or functions. However, the executables of an identified module may include heterogeneous instructions stored in different locations that do not need to be physically located together but, when logically combined, constitute a module and achieve the purpose defined for the module. In fact, a module of executable code may be a single instruction or a number of instructions, and may even be distributed across several different code segments, different programs, and several memory devices. Similarly, operational data may be embodied in any preferred form and organized in any preferred type of data structure. Operational data may be collected as a single dataset, distributed in different locations such as different storage devices, or exist on a system or network, at least partially, simply as electrical signals.

[0056] In this document, relational terms such as "first" and "second" may be used solely to distinguish one entity or action from another, and do not necessarily require or imply any actual relationship or order between such entities or actions. Ordinal numbers such as "first," "second," and "third" simply indicate a single distinct entity among several, unless specifically defined by the wording of the claim, and do not imply any order or sequence. No sequence of text in any of the claims implies that process steps must be performed in a chronological or logical order according to such sequence, unless specifically defined by the wording of the claim. Process steps may be rearranged in any order without departing from the scope of the invention, provided that such rearrangement is not inconsistent with the wording of the claim and is not logically meaningless.

[0057] Furthermore, depending on the context, phrases such as "connect" or "join" used to describe the relationship between different elements do not imply that a direct physical connection must exist between these elements. For example, two elements may be connected to each other physically, electronically, logically, or in any other way through one or more additional elements.

[0058] While the above detailed description of the present invention has presented at least one exemplary embodiment, it should be understood that a vast number of variations exist. It should also be understood that one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of the present invention in any way. Rather, the above detailed description will provide a convenient roadmap for implementing exemplary embodiments of the present invention. It should be understood that various modifications can be made to the function and configuration of the elements described in the exemplary embodiments without departing from the scope of the present invention, as described in the appended claims.

Claims

1. A system for visualizing automated actions using holographic agents, At least one processor, The system comprises at least one memory communicably coupled to the at least one processor, the at least one memory containing instructions, and the instructions, when executed by the at least one processor, Receiving the selection of the holographic format for the holographic agent via the user interface, Receiving a first automatic control notification associated with a first automatic control action at a first time, wherein the first automatic control action is scheduled to be performed by the vehicle's autopilot system at a second time, and the second time is after the first time. To generate a first agent action based at least in part on the first automatic control action, A system that transmits a first command to a hologram generation system to generate the selected holographic form of the holographic agent involved in the first agent action at the first time, wherein the first agent action is a visual representation of the autopilot system performing the first automatic control action prior to the performance of the first automatic control action at the second time.

2. The at least one memory includes an instruction, and when the instruction is executed by the at least one processor, the at least one processor receives the instruction. Receiving a second auto-control notification associated with a second auto-control action in a third time, wherein the second auto-control action is scheduled to be performed by the autopilot system in a fourth time, and the second auto-control action is in response to a first manual control action performed via a control interface after the execution of the first auto-control action, and the fourth time is after the third time. The second agent action is generated based at least in part on the second automatic control action, The system according to claim 1, which transmits a second command to the hologram generation system to generate the selected holographic form of the holographic agent involved in the second agent action in the third time, wherein the second agent action is a visual representation of the autopilot system performing the second automatic control action prior to the performance of the second automatic control action in the fourth time.

3. The at least one memory includes an instruction, and when the instruction is executed by the at least one processor, the at least one processor receives the instruction. Receiving a third auto-control notification associated with a third auto-control action in a fifth time, wherein the third auto-control action is scheduled to be performed by the autopilot system in a sixth time, and the second auto-control action is in response to a second manual control action performed via the control interface before the execution of the first auto-control action in the second time, wherein the second manual control action is operated to override the first auto-control action. The third agent action is generated based at least in part on the third automatic control action, Sending a third command to the hologram generation system to generate the selected holographic form of the holographic agent involved in the third agent action in the fifth time, wherein the third agent action is a visual representation of the autopilot system performing the third automatic control action prior to the performance of the third automatic control action in the sixth time, The system according to claim 1, which transmits a second command to the hologram generation system to generate the selected holographic form of the holographic agent involved in the second agent action in the third time, wherein the second agent action is a visual representation of the autopilot system performing the second automatic control action prior to the performance of the second automatic control action in the fourth time.

Images