AI and vr-based aviation safety investigation and active training integrated system and method

Abstract

This invention discloses an integrated system and method for aviation safety investigation and proactive training based on AI and VR, belonging to the field of aviation safety technology. The system aims to solve the problems of low efficiency in traditional investigation methods, poor flexibility of training simulators, and fragmented technology application. The system adopts a three-layer architecture: a data perception and fusion layer that accesses and synchronizes multi-source heterogeneous data; an AI intelligent analysis hub layer that extracts structured knowledge from the data and generates standardized contextual data packages through event sequence reconstruction and hybrid causal reasoning models; and a VR immersive application layer that transforms the data packages into an immersive virtual environment, supporting three modes: in-depth investigation, proactive training, and risk prediction. This invention achieves a closed loop from intelligent analysis of multi-source data to immersive contextual reproduction, directly transforming in-depth investigation conclusions into training resources, significantly improving investigation efficiency, training accuracy, and proactive risk management capabilities.

Description

Technical Field

[0001] This invention belongs to the field of aviation safety technology, specifically relating to a system that integrates artificial intelligence (AI) and virtual reality (VR) technologies, and in particular an integrated intelligent system for in-depth investigation of aviation safety incidents and proactive personnel training. Background Technology

[0002] The cornerstone of aviation safety lies in continuous learning and improvement from past incidents. However, with the exponential growth in the complexity of aircraft systems and the vast increase in the dimensions of operational data, traditional aviation safety investigation and personnel training methods are facing serious challenges, and their inherent limitations are becoming increasingly apparent.

[0003] 1. Bottlenecks in traditional aviation safety investigation methods

[0004] Currently, aviation safety investigations primarily rely on independent analysis of data recorded by flight data recorders (FDR) and cockpit voice recorders (CVR), as well as interviews with relevant personnel. This approach has reached its limits in handling modern aviation accidents. Investigators are forced to confront massive amounts of heterogeneous, fragmented two-dimensional data, including time-series flight parameters, unstructured audio conversations, possible cockpit video clips, meteorological data, route information, and text reports. They must perform a difficult "mental piecing together" to attempt to reconstruct a multi-dimensional, dynamic, and complex human-machine interaction scenario.

[0005] This process is not only inefficient, but more importantly, it easily misses fleeting yet potentially decisive clues. For example, during the critical phase of approach and landing, the subtle coupling between the pilot's brief shift in gaze and the autopilot's state is difficult to capture and correlate effectively using traditional data visualization analysis. Essentially, existing methods lack the ability to transform discrete "data" into an immersive, interactive "context," resulting in insufficient in-depth causal analysis and difficulty in accurately quantifying the contribution of each factor in the "human-machine-environment" system.

[0006] 2. Limitations of existing flight training simulators

[0007] In terms of personnel training, while mainstream full-motion flight simulators (FFS) can provide high-fidelity flight control training, they have significant limitations in reproducing the complete scenarios of specific historical unsafe events. Particularly for training in non-technical skills (such as crew resource management, communication, decision-making, and load management), constructing highly customized emergency scenarios is costly and lacks flexibility. Simulators typically operate based on pre-set scripts, making it difficult to quickly respond to personalized training needs based on real-world event data, and also unable to reproduce rare but severe extreme scenarios for crew experience and learning at low cost.

[0008] 3. Isolated applications of artificial intelligence and virtual reality technologies

[0009] In recent years, the parallel development of artificial intelligence (AI) and virtual reality (VR) technologies has provided new tools for enhancing data analysis capabilities and scenario reproduction capabilities, respectively. In the aviation field, research has explored applying machine learning algorithms to analyze Quick Access Recorder (QAR) data for trend monitoring or fault warning, while VR technology has been used for general procedures or emergency drills. However, most of these studies and practices focus on the application of single technologies and have not yet formed a complete technical framework that deeply and systematically integrates the intelligent analysis core of AI with the immersive interactive front-end of VR. AI analysis results are often presented in traditional forms such as charts and reports, disconnected from the experiential virtual environment; while the construction of VR training scenarios largely relies on manual modeling and scripting, lacking the ability to automatically generate high-fidelity, interactive scenarios from real, multi-source event data.

[0010] In summary, existing technological systems in the field of aviation safety suffer from core problems such as difficulties in data valuation, a disconnect between post-incident analysis and pre-incident prevention, and fragmented technology application. The industry urgently needs a new paradigm that can bridge in-depth post-incident investigations and efficient pre-incident training, systematically integrating multi-source data, intelligently mining causal mechanisms, and immersively recreating scenarios to fundamentally improve the effectiveness of safety management. Summary of the Invention

[0011] In view of the problems of inefficiency and lack of depth in traditional aviation safety investigation methods described in the background art, high cost and poor flexibility of existing flight training simulators when reproducing specific unsafe events, and fragmented and unsystematic integration of AI and VR technologies, this invention proposes an innovative technical solution.

[0012] (a) The technical problem to be solved by the present invention

[0013] This invention aims to solve the following three core technical problems:

[0014] 1. Difficulty in data valuation: Traditional methods rely on investigators to manually "mentally piece together" multi-source heterogeneous data such as FDR, CVR, QAR, and text reports, making it difficult to efficiently and comprehensively transform discrete two-dimensional data into a three-dimensional "context" that can be immersively interacted with and deeply analyzed.

[0015] 2. Disconnect between post-incident analysis and pre-incident prevention: The conclusions of in-depth investigations are difficult to translate into training scenarios that can be used for proactive training at low cost and with high fidelity, resulting in a separation between the two key safety links of "investigation" and "training".

[0016] 3. Fragmentation of technology applications: In existing technologies, artificial intelligence (AI) is mostly used for isolated data analysis (such as QAR data mining), and virtual reality (VR) is mostly used for general program training. The two have failed to be deeply integrated to form a technology paradigm that integrates "intelligent analysis core" and "immersive interactive front end".

[0017] (II) Technical Solution of the Invention

[0018] To address the aforementioned technical challenges, this invention provides an integrated system for aviation safety investigation and proactive training based on artificial intelligence and virtual reality technologies. Its core design concept is to construct a closed loop of "perception-cognition-interaction." Through a three-tiered architecture, this system transforms multi-source heterogeneous data into structured knowledge and ultimately delivers it as an immersive and explorable intelligent context.

[0019] 1. System Overall Architecture This system adopts a hierarchical and modular integrated architecture, comprising, from top to bottom: a data perception and fusion layer, an AI intelligent analysis hub layer, and a VR immersive application layer. These three layers are connected through standardized data interfaces, forming a complete technical pathway from raw data input to immersive application output. (See the attached instruction manual for a schematic diagram of the "Tianmou" system overall architecture.) Figure 1 )

[0020] 2. Specific composition and function of each level

[0021] (1) Data perception and fusion layer: As the system foundation, it is responsible for accessing, synchronizing and fusion of multi-source heterogeneous data.

[0022] Data access: Unified access to four types of data: time-series parameter data (such as flight status and system parameters of QAR / FDR); unstructured media data (such as CVR audio and in-cabin video); environmental and operational data (such as meteorological, route, and terrain databases, and maintenance records); and text report data (such as preliminary investigation reports and standard operating procedures (SOPs)).

[0023] Core processing: Achieving high-precision spatiotemporal synchronization and preliminary semantic association of data. Specifically, a layered synchronization strategy is adopted: millisecond-level alignment is performed on time-stamped data (such as QAR); for data lacking precise time sources (such as CVR), an "event-driven alignment method" is used, leveraging the common salient features shared with QAR data—"event anchors" (such as landing gear lowering sound and landing gear position signal transitions)—for cross-correlation matching. At the semantic layer, an "aviation safety ontology" is constructed, mapping multi-source data to a unified conceptual model (such as "flight phase," "system status," and "crew actions"), enabling cross-modal information association queries.

[0024] (2) AI intelligent analysis central layer: As the "brain" of the system, it is responsible for extracting knowledge from the fused data and reconstructing the logic of events.

[0025] Event sequence reconstruction engine: Based on time sequence segmentation and pattern recognition algorithms, it automatically cuts continuous data streams into meaningful event units (such as "start descent", "deploy flaps", "receive wind shear warning") and constructs an initial event timeline.

[0026] Causal Reasoning and Contribution Analysis Model: This is the core innovative module of the system. It employs a hybrid reasoning model, combining data-driven and knowledge-driven methods. First, it uses a time-series graph neural network to model event sequences, capturing complex time dependencies and initially identifying potential causal pairs. Then, a verification module based on physical models (such as flight dynamics equations) and rule bases (such as standard operating procedures) is introduced to verify and quantify the data-driven causal hypotheses. For example, it verifies whether a "change in elevator deflection angle" can lead to an observed "change in pitch angle" within a given time period, or determines whether "failure to execute checklist items" is a necessary condition for subsequent "system error configurations." Ultimately, this model achieves a quantitative contribution analysis of the "human-machine-environment" coupling effect.

[0027] Context Database Generator: This generator encapsulates the results of AI analysis—including structured event chains, cause-effect graphs, and key state parameters—into standardized "context data packages." These packages contain complete spatiotemporal information, entity states, and logical relationships, providing a unified and standardized input for scene reproduction in the VR layer.

[0028] (3) VR immersive application layer: As the "interactive interface" of the system, it transforms the AI-generated context data package into a perceptible, operable and explorable virtual environment.

[0029] In-depth investigation mode: Investigators can enter the recreated event scene from a first-person perspective (e.g., "possessed" by the crew) or a God's-eye view. It supports free pausing, fast-forwarding, and jumping, allowing observation from any angle. The system visually overlays key event nodes and causal hypotheses analyzed by AI onto the scene to aid in verification.

[0030] Active training mode: Instructors can quickly generate specific training scenarios based on a historical event database or custom parameters. Trainees perform operations and make decisions in VR. The system integrates eye tracking, biosensors, etc., to record operations and assess cognitive load and decision-making process, generating in-depth review reports that include eye-tracking heatmaps and decision-making timeline comparisons.

[0031] Risk prediction model: Decision-makers can conduct intuitive risk assessments by observing the distribution of multiple flight trajectories and changes in conflict hotspots through digital twins and rapid simulations after different risk factors (new procedures, extreme weather, new obstacles) are injected into the system in a virtual environment (such as from a control tower perspective).

[0032] Interaction design principles: Follow cognitive ergonomics, adopt layered information presentation, intuitive 3D interaction (such as timeline sliders, causal chain pulls, and perspective shifts), and multi-channel feedback (3D spatial audio, force feedback) to reduce cognitive load.

[0033] (III) Beneficial Effects of the Invention

[0034] Compared with the prior art, the technical solution provided by the present invention can bring the following significant beneficial effects:

[0035] 1. A leap in the depth and efficiency of investigation: By automatically reconstructing the causal chain of events and quantifying the contribution through AI, combined with VR immersive verification, the root cause analysis cycle of complex unsafe events can be shortened by more than 50%, and the accuracy of conclusions can be significantly improved.

[0036] 2. Breakthrough in training precision and flexibility: Capable of rapidly generating high-fidelity, low-cost personalized VR training scenarios based on real-world events or custom parameters, achieving "targeted training." It is expected to improve the conversion efficiency of specific skills (such as non-precision approach energy management) by over 30% and significantly reduce the need for high-level full-motion simulators (FFS).

[0037] 3. Systematic integration of technological paradigms: For the first time, the core of AI's intelligent analysis is deeply coupled with the immersive interactive front-end of VR, constructing a complete technological closed loop from multi-source data fusion and intelligent causal analysis to immersive scenario reproduction and interaction, solving the core problem of the separation between data value and technological defense.

[0038] 4. Proactive risk management capability: Through digital twin simulation and immersive visualization, risk assessment is transformed from abstract report review into intuitive "scenario rehearsal", which can identify potential design flaws in the operating program or environment earlier and more accurately, reducing the cost of later revisions.

[0039] 5. Forming a positive closed loop in safety management: This system naturally bridges "post-incident investigation" and "pre-incident training," enabling the profound insights gained from the investigation to be directly and efficiently transformed into training resources to improve personnel capabilities, and promoting the evolution of safety management from a passive "incident response" culture to a proactive "system immunity" culture. Attached Figure Description

[0040] The attached diagram in the instruction manual Figure 1 This is a schematic diagram of the overall architecture of the "Tianmou" system provided in the implementation examples of this invention. As shown in the figure, the system of this invention adopts a three-layer logical architecture, constructing a complete technical path from multi-source heterogeneous data to immersive intelligent scenarios.

[0041] 1. Data Awareness and Fusion Layer (Bottom Layer): As the system's data foundation, it is responsible for accessing and fusing four types of heterogeneous data sources:

[0042] (1) Time series parameter data: such as the time series of flight status and system parameters recorded by QAR / FDR.

[0043] (2) Unstructured media data: such as CVR audio (requiring speech recognition and semantic analysis) and in-cabin video.

[0044] (3) Environmental and operational data: such as meteorological, route, topographic, obstacle databases and aircraft maintenance records.

[0045] (4) Text report data: such as preliminary investigation reports, standard operating procedures (SOPs) and checklists. The core function of this layer is to achieve high-precision spatiotemporal synchronization of data (such as using GPS timestamps for millisecond-level alignment, or using an event-driven alignment method based on "event anchors") and preliminary semantic association (by constructing a "aviation safety ontology" mapping concept relationship), and output a unified "raw data fusion stream".

[0046] 2. AI Intelligent Analysis Central Layer (Middle Layer): As the intelligent core of the system, it receives the fused data stream and includes three core modules:

[0047] (1) Event sequence reconstruction engine: Based on time sequence segmentation and pattern recognition algorithms, the continuous data stream is automatically segmented into meaningful event units (such as "start to fall" and "receive wind shear warning") and an initial event timeline is constructed.

[0048] (2) Causal reasoning and contribution analysis model: This module adopts a hybrid reasoning model, combining time-series graph neural network (data-driven) and verification based on physical model and rule base (knowledge-driven) to infer the causal relationship between events and quantify the contribution of each factor in "human-machine-environment".

[0049] (3) Context database generator: The analysis results (event chain, cause-effect graph, key state parameters) are structured into standardized "context data packages" to provide input for upper-layer applications.

[0050] 3. VR Immersive Application Layer (Top Layer): As the system's interactive front end, it receives "context data packages" and generates a 3D immersive environment, supporting three core application modes:

[0051] (1) In-depth investigation mode: Users (investigators) can enter the recreated scene from a first-person or God's-eye view, freely control the timeline (pause, fast forward, jump), observe from multiple angles, and the system will visualize and overlay the key event nodes and causal chains analyzed by AI.

[0052] (2) Active training mode: Instructors can quickly generate specific training scenarios based on historical events or custom parameters. Trainees operate in VR, and the system integrates eye tracking, biosensors and other technologies to assess cognitive load and decision-making process, generating in-depth analysis reports including eye-tracking heatmaps and decision-making timelines.

[0053] (3) Risk Foresight Mode: Decision-makers can conduct intuitive risk assessments by observing the distribution of multiple flight trajectories and changes in conflict hotspots through digital twin simulation after different risk factors (such as extreme weather and new obstacles) are injected in a virtual environment (such as from a control tower perspective). This layer is designed in accordance with the principles of cognitive ergonomics, providing intuitive 3D interactions such as "timeline sliders", "causal chain pulls", and "viewpoint teleportation", and integrating 3D spatial audio to enhance immersion.

[0054] The three layers are connected through standardized data interfaces, forming a closed loop of "perception-cognition-interaction".

Claims

1. An integrated intelligent system combining AI and VR for in-depth investigation of aviation safety incidents and proactive personnel training, characterized in that: It comprises a data perception and fusion layer, an AI intelligent analysis hub layer, and a VR immersive application layer connected in sequence. The data perception and fusion layer is used to access and synchronize multi-source heterogeneous aviation safety data with high precision in time and space, and output the raw data fusion stream. The AI ​​intelligent analysis hub layer is used to receive the raw data fusion stream and generate standardized contextual data packages containing event chains, cause-effect graphs, and key state parameters through intelligent analysis. The VR immersive application layer is used to receive the standardized contextual data packages, generate and provide an immersive and interactive virtual environment, and support three application modes: in-depth investigation, proactive training, and risk prediction.

2. The system according to claim 1, characterized in that, The multi-source heterogeneous data accessed by the data perception and fusion layer includes: time-series parameter data from QAR / FDR, unstructured media data from CVR, environmental and operational data, and text report data.

3. The system according to claim 1, characterized in that, The AI ​​intelligent analysis central layer includes: an event sequence reconstruction engine, used to cut continuous data streams into event units and construct an initial timeline based on time sequence segmentation and pattern recognition algorithms; a causal reasoning and contribution analysis model, used to combine data-driven methods and domain knowledge models to establish causal relationships between events and quantify contribution; and a context database generator, used to structure the analysis results into the standardized context data package.

4. The system according to claim 1, characterized in that, The VR immersive application layer provides an in-depth investigation mode that allows users to enter the recreated event scene from a first-person or God's-eye view, freely manipulate time and switch perspectives, and visualize the key event nodes and causal chains analyzed by AI.

5. The system according to claim 1, characterized in that, The active training mode provided by the VR immersive application layer can generate personalized VR training scenarios based on a historical event database or set parameters, and integrate eye tracking and / or biosensors to assess the learner's cognitive load and decision-making process.

6. The system according to claim 1, characterized in that, The risk prediction mode provided by the VR immersive application layer can, based on digital twins and rapid simulation, deduce the distribution of multiple flight trajectories and changes in conflict hotspots after the injection of different risk factors, and perform multi-dimensional risk visualization.

Images