Training method of flight training simulation model and flight training simulation method

Abstract

The invention provides a training method of a flight training simulation model and a flight training simulation method, relates to the technical field of artificial intelligence and flight training simulation, and aims to accurately predict a video token sequence of an environment state at a next moment based on the flight training simulation model, effectively restore a real training environment of a pilot and improve the training efficiency of the pilot. The authenticity of flight training simulation is improved. The method comprises the steps of obtaining training data; the training data comprises a video frame sequence of the flight environment; based on the training data, training a video word segmentation device, a potential action model and a dynamic environment prediction model in the flight training simulation model; wherein the video word segmentation device is used for processing an input video frame sequence of a flight environment into a video token sequence; the potential action model is used for determining a potential action token sequence of the pilot based on an input video frame sequence of the flight environment; and the dynamic environment prediction model is used for determining and predicting a next frame of video token sequence based on the video token sequence and the pilot potential action token sequence.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and flight training simulation technology, and in particular to a training method for a flight training simulation model and a flight training simulation method. Background Technology

[0002] With the rapid development of the air transport industry and the increasing complexity of flight missions, pilots' emergency response and adaptability to complex scenarios have become core guarantees of flight safety. Current pilot training mainly relies on physically based flight simulators, but their training scenario coverage is limited, making it difficult to generate diverse, high-fidelity environments. Customized development is required for emergency scenarios such as extreme weather and equipment failures, resulting in insufficient training flexibility and timeliness. Furthermore, traditional systems lack multimodal interaction capabilities and cannot autonomously learn and generate new scenarios from massive amounts of flight-related video data. In short, the shortcomings of existing flight simulation training systems in terms of scenario adaptability, interactive experience, and intelligent feedback seriously affect the effectiveness of flight training. Summary of the Invention

[0003] The purpose of this application is to provide a training method for a flight training simulation model and a flight training simulation method, which aims to accurately predict the video token sequence of the environmental state at the next moment based on the flight training simulation model, effectively restore the real training environment of the pilot, and improve the realism of the flight training simulation.

[0004] To achieve the above objectives, the embodiments of this application provide the following technical solutions: In a first aspect, this application provides a training method for a flight training simulation model. The method includes: acquiring training data; the training data includes a video frame sequence of a flight environment; and training a video tokenizer, a latent action model, and a dynamic environment prediction model in the flight training simulation model based on the training data. The video tokenizer is used to process the input video frame sequence of the flight environment into a video token sequence; the latent action model is used to determine a pilot's potential action token sequence based on the input video frame sequence of the flight environment; and the dynamic environment prediction model is used to determine the predicted next frame video token sequence based on the video token sequence and the pilot's potential action token sequence.

[0005] The flight training simulation model provided in this application comprises a video tokenizer, a latent action model, and a dynamic environment prediction model. The flight training simulation model is trained using training data, enabling it to directly learn the mapping relationship between environmental video sequences, pilot latent actions, and subsequent environmental changes from real flight video data. Specifically, the video tokenizer processes the video frame sequence of the flight environment into a video token sequence, the latent action model generates a sequence of potential pilot action tokens based on the video frame sequence, and the dynamic environment prediction model uses these two token sequences as input to predict the next frame's video token sequence. This process effectively recreates the future environmental changes faced by the pilot in actual training environments, significantly improving the realism of the flight training simulation.

[0006] In some embodiments, the latent action model is trained by: acquiring real pilot operation data; obtaining a sequence of potential pilot action tokens based on the video frame sequence of the flight environment in the training data and the real pilot operation data; the latent action model includes: a low-level action branch and a high-level action branch; wherein, the low-level action branch is used to determine the low-level latent action token sequence from the input video frame sequence of the flight environment; the high-level action branch is used to determine the high-level latent action token sequence from the input video frame sequence of the flight environment; in the training of the low-level action branch of the latent action model, a first loss value is determined; the first loss value is used to characterize the loss relationship between the low-level latent action token sequence and the low-level action data in the real pilot operation data; in the training of the high-level action branch of the latent action model, a second loss value is determined; the second loss value is used to characterize the loss relationship between the high-level latent action token sequence and the high-level action data in the real pilot operation data; by minimizing the first loss value and / or the second loss value, the trained latent action model is obtained.

[0007] In some embodiments, the dynamic environment prediction model is trained by: acquiring a video token sequence and a pilot's potential action token sequence; the video token sequence includes a historical video token sequence and the actual next frame video token sequence; based on the video token sequence and the pilot's potential action token sequence, obtaining a predicted next frame video token sequence through the dynamic environment prediction model; determining a prediction loss value based on the predicted next frame video token sequence and the actual next frame video token sequence; and obtaining the trained dynamic environment prediction model by minimizing the prediction loss value.

[0008] In some embodiments, the flight training simulation model further includes a multimodal encoder; the multimodal encoder is used to encode multimodal special case events in the training of the flight training simulation model into an event token sequence; the dynamic environment prediction model is used to determine the predicted next frame token sequence based on the video token sequence and the pilot's potential action token sequence, including: updating the predicted next frame token sequence through the dynamic environment prediction model based on the video token sequence, the pilot's potential action token sequence and the event token sequence.

[0009] In some embodiments, the multimodal encoder includes at least one of the following: a text encoder for encoding text instructions in a multimodal special event into a text token sequence; a speech encoder for encoding speech instructions in a multimodal special event into a speech token sequence; an image encoder for encoding image instructions in a multimodal special event into an image token sequence; a file encoder for encoding structured files in a multimodal special event into a file token sequence; and a sensor data encoder for encoding sensor data streams in a multimodal special event into a data stream token sequence.

[0010] In some embodiments, the flight training simulation model further includes a historical data memory; the historical data memory is used to provide historical scenario token sequences during the training of the flight training simulation model; the dynamic environment prediction model is used to determine the predicted next frame token sequence based on the video token sequence and the pilot's potential action token sequence, including: retrieving multiple historical scenario token sequences from the historical data memory according to a preset similarity algorithm based on the video token sequence and the pilot's potential action token sequence; and updating the predicted next frame token sequence through the dynamic environment prediction model based on the video token sequence, the pilot's potential action token sequence, and the multiple historical scenario token sequences.

[0011] Secondly, this application also provides a flight training simulation method, which includes: acquiring flight environment data; determining a predicted next frame video token sequence based on the flight environment data using the flight training simulation model; and updating the flight training simulation environment based on the next frame video token sequence; wherein the flight training simulation model is trained based on the method described in the first aspect above.

[0012] Thirdly, this application also provides a training device for a flight training simulation model, the device comprising: an acquisition module and a training module; the acquisition module is used to: acquire training data; the training data includes a video frame sequence of the flight environment; the training module is used to: train a video tokenizer, a latent action model, and a dynamic environment prediction model in the flight training simulation model based on the training data; wherein, the video tokenizer is used to process the input video frame sequence of the flight environment into a video token sequence; the latent action model is used to determine the pilot's potential action token sequence based on the input video frame sequence of the flight environment; the dynamic environment prediction model is used to determine the predicted next frame video token sequence based on the video token sequence and the pilot's potential action token sequence.

[0013] Fourthly, this application also provides an electronic device comprising: a processor and a memory; the memory storing processor-executable instructions; when the processor is configured to execute the instructions, causing the electronic device to implement the methods of the first or second aspect described above.

[0014] Fifthly, this application also provides a computer-readable storage medium comprising: computer software instructions; when the computer software instructions are executed in an electronic device, the electronic device causes the electronic device to implement the methods of the first or second aspect described above.

[0015] Sixthly, this application also provides a computer program product comprising a computer program that, when run on an electronic device, causes the electronic device to perform the method of the first aspect or the second aspect.

[0016] The beneficial effects of the second to sixth aspects mentioned above can be referred to the corresponding descriptions in the first aspect, and will not be repeated here. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This application provides a schematic diagram of the architecture of a training system for a flight training simulation model. Figure 2 A flowchart illustrating a training method for a flight training simulation model provided in this application embodiment; Figure 3 A flowchart illustrating a training method for a potential action model provided in an embodiment of this application; Figure 4 A flowchart illustrating a training method for a dynamic environment prediction model provided in an embodiment of this application; Figure 5 A schematic diagram of the architecture of a flight training simulation model provided in this application embodiment; Figure 6 A flowchart illustrating a flight training simulation method provided in this application embodiment; Figure 7 This is a schematic diagram illustrating the composition of a training device for a flight training simulation model provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0021] In the embodiments of this application, the terms "first," "second," "third," "fourth," "fifth," and "sixth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," "third," "fourth," "fifth," and "sixth" may explicitly or implicitly include one or more of that feature.

[0022] In embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. For "A and / or B," this includes three combinations: A only, B only, and a combination of A and B.

[0023] Based on the aforementioned background technology, the generative interactive environments (Genie) model, as a world model, consists of three core architectures: inferring the latent actions between each pair of frames through a simple and scalable latent action model; converting the original video frames into discrete tokens using a video segmenter; and finally, using a dynamic model to combine the latent actions with past frame tokens to predict the state of the next frame. This world model provides a novel technical path for constructing data-driven flight training simulation environments.

[0024] However, this world model still has several key limitations when adapting to the high-fidelity and high-safety requirements of professional pilot training: First, the model derives potential actions from differences between video frames in a completely unsupervised manner, lacking explicit alignment and semantic association with standard operating actions of real pilots. This makes it difficult for the generated environmental responses to accurately reflect the physical and logical impacts of actual flight operations. Second, when facing sudden emergencies that may occur during flight and manifest in a multimodal manner, the existing architecture lacks an effective encoding and fusion mechanism for multi-source heterogeneous information such as text commands, voice alarms, and sensor data streams, limiting the system's ability to understand and respond to complex input conditions. In addition, the world model is insufficient in long-term time-series modeling. Its inherent memory and state retention mechanisms cannot guarantee the dynamic coherence and physical state consistency of long-term, multi-stage flight mission simulations, failing to meet the high-fidelity simulation requirements of the entire flight from takeoff and cruise to approach and landing. In short, the existing world model cannot obtain a flight training environment that meets the current flight training needs, thus affecting the effectiveness of flight training.

[0025] To address the aforementioned issues, this application provides a training method for a flight training simulation model. This method constructs a flight training simulation model comprising a video tokenizer, a latent action model, and a dynamic environment prediction model. The flight training simulation model is trained using training data, enabling it to directly learn the mapping relationship between environmental video sequences, pilot latent actions, and subsequent environmental changes from real flight video data. Specifically, the video tokenizer processes the video frame sequence of the flight environment into a video token sequence; the latent action model generates a sequence of potential pilot action tokens based on the video frame sequence; and the dynamic environment prediction model uses these two token sequences as input to predict the next frame's video token sequence. This process effectively recreates the future environmental changes faced by the pilot in actual training environments, significantly improving the realism of the flight training simulation.

[0026] The training method for the flight training simulation model provided in this application embodiment can be applied to a model training device, wherein the model training device can be an electronic device such as a server, server cluster, high-performance computing platform or dedicated flight simulation host. This application embodiment does not impose any restrictions on the specific form of the model training device.

[0027] In some embodiments, the model training device belongs to Figure 1 The training system of the flight training simulation model shown. For example... Figure 1 As shown, the flight training simulation model training system may include a flight training data acquisition device 101 and a flight training model training device 102. The flight training data acquisition device 101 is equipped with a multimodal data acquisition and preprocessing unit (for synchronously acquiring and aligning flight environment videos, pilot operation signals, and emergency event commands). The flight training model training device 102 may be equipped with an initial video segmenter, a potential action model, and a dynamic environment prediction model. The flight training model training device 102 can establish a communication connection with the flight training data acquisition device 101 to obtain the preprocessed and aligned multimodal flight training dataset, and then perform phased or end-to-end joint training on the aforementioned models, ultimately obtaining a well-trained flight training simulation model capable of high-fidelity prediction of environmental evolution based on historical states and operational intentions.

[0028] In other embodiments, the flight training model training device 102 integrates a multimodal data acquisition and preprocessing unit, a video segmenter, a potential action model, and a dynamic environment prediction model, completing the entire process from data acquisition and preprocessing to model training and optimization directly on a single device.

[0029] The flight training data acquisition device 101 and the flight training model training device 102 can be integrated on the same device (such as an integrated flight simulator), or the flight training data acquisition device 101 and the flight training model training device 102 can be set in separate devices (such as a distributed training cluster).

[0030] This application also provides a flight training simulation method for dynamically generating and updating high-fidelity virtual training scenarios based on a trained flight training simulation model and according to real-time input flight environment status and pilot operations.

[0031] Understandably, this flight training simulation method can be applied to... Figure 1 The flight training simulation application device 103 shown can also be applied to the flight training model training device 102 described above, and this application does not impose any restrictions on it.

[0032] The flight training simulation application device 103 contains a pre-trained flight training simulation model. The flight training simulation application device can be an electronic device such as a full-motion flight simulator, a fixed-base simulator, or a desktop training system. This application embodiment does not impose any restrictions on the specific form of the flight training simulation application device 103.

[0033] The training method of the flight training simulation model provided in this application will be described below with reference to specific embodiments and accompanying drawings.

[0034] Figure 2 This is a flowchart illustrating a training method for a flight training simulation model provided in an embodiment of this application. Figure 2 As shown, the training method for this flight training simulation model includes S201-S202: S201. Obtain training data.

[0035] The training data includes a sequence of video frames depicting the flight environment. This sequence serves as input for the subsequent flight training simulation model. A video frame sequence is a series of still images arranged chronologically, with each video frame being a single image.

[0036] In one possible implementation, the video frame sequence of the flight environment in the training data can be obtained by receiving multimodal data such as text, photos, sketches, synthetic images or voice from the instructor, having the flight training simulation model parse and synthesize the input multimodal data into a corresponding initial static scene image, and then decomposing and expanding the image into a continuous sequence of flight environment video frames.

[0037] For example, the flight training simulation model includes a scene generation module that may include a mapping relationship between multimodal data and images, which can be converted into an initial static scene image by parsing the multimodal data.

[0038] For example, during the environment initialization phase, instructors can precisely set the initial flight environment for training through multimodal input. Specifically, the system supports specifying structured parameters (such as flight information, routes, weather conditions, aircraft load, and training subjects) through text, forms, or prompts; visually defining airport layouts, weather scenes, and terrain features by uploading photos, images, or sketches; or directly describing flight routes, weather, and runway status through voice commands. After the aforementioned multimodal inputs are converted into a unified semantic representation by the multimodal encoder, the scene generation module synthesizes the corresponding initial static scene image. Then, the video segmenter encodes this image sequence into an initial video token sequence, thereby providing structured initial state input for subsequent video segmenters, latent action models, and dynamic environment prediction models, completing the rapid construction and initialization of a high-fidelity training environment.

[0039] For example, an instructor inputs the text "slippery runway under thunderstorm clouds at dusk" and uploads a photo of the runway. The flight training simulation model generates a static airport visual image that matches the description through the scene generation module, and then expands it into a frame sequence with coherent weather and lighting effects. Another example is when an instructor draws a sketch containing mountains and a runway and verbally says "mountain airport, crosswind 20 knots." The flight training simulation model integrates the sketch layout and voice parameters to synthesize a static image that reflects the mountain terrain and crosswind weather, and then combines the flight parameters to generate a physically consistent approach scene frame sequence.

[0040] S202. Based on training data, train the video segmenter, potential action model, and dynamic environment prediction model in the flight training simulation model.

[0041] This application uses a video tokenizer to process a sequence of video frames of the flight environment into a sequence of video tokens.

[0042] The latent motion model generates a sequence of potential motion tokens for the pilot based on a sequence of video frames.

[0043] The dynamic environment prediction model takes the two token sequences mentioned above as input to predict the next frame's video token sequence, thereby achieving a realistic simulation of future environmental changes and effectively improving the realism of flight training simulation.

[0044] The following sections will provide a detailed introduction to the video segmenter, latent action model, and dynamic environment prediction model in the flight training simulation model, including (1)-(3): (1) The video tokenizer is used to process the input video frame sequence of the flight environment into a video token sequence. The video token sequence is the discretized semantic visual encoding sequence corresponding to the video frame sequence of the flight environment after processing by the video tokenizer.

[0045] In some implementations, the video frame sequence is modeled using spatial-temporal attention, and the input video frame sequence is transformed into a discrete video token sequence using a vector quantized-variational autoencoder (VQ-VAE).

[0046] Among them, the spatial-temporal attention mechanism is a method that uses self-attention to collaboratively model the spatial and temporal dependencies in sequential data. The spatial-temporal attention mechanism can include a spatial-temporal transformer (ST-Transformer) structure.

[0047] For example, the processing flow of the ST-Transformer structure can be described as follows: the input token is first adjusted in dimension by a linear transformation (W, H), and then sequentially passes through a spatial attention layer and a temporal attention layer—spatial attention focuses on the semantic relationships between regions (such as meters, views outside the window) within a single frame, while temporal attention captures the dynamic evolution of the same or different regions across frames; finally, feature enhancement is performed through a feedforward layer, outputting an output token rich in spatiotemporal context, thereby achieving a deep, structured, and temporally sequential representation of the video content. In this application, these tokens are continuous feature vectors rich in spatiotemporal semantics, not the video token sequence of this application.

[0048] VQ-VAE is a codebook-based generative model whose architecture mainly consists of three parts: an encoder, a vector quantization module (containing the codebook), and a decoder. The encoder first maps the input data (such as image or video frames) into continuous latent feature vectors. These latent feature vectors are then fed into the vector quantization module, which uses nearest neighbor search to match the closest codebook vector in a learnable discrete codebook, thus transforming the continuous features into sparse and semantically clear discrete codes. The decoder then retrieves the corresponding vector from the codebook based on the quantized discrete codes and reconstructs data that is visually consistent with the original input.

[0049] In one possible implementation, a video tokenizer processes the input sequence of flight environment video frames into a sequence of video tokens for use by subsequent potential action inference and dynamic environment prediction modules. The video tokenizer's processing flow consists of two stages: In the first stage, the video segmenter uses the encoder in VQ-VAE to extract visual features frame by frame and adopts a spatial-temporal attention mechanism for spatiotemporal modeling. Spatial attention is used to model the semantic associations between different regions within a single frame, such as the dashboard, the view outside the window, and the control device, while temporal attention is used to model the continuous evolution of the state of the same region across frames, and finally outputs a continuous feature vector.

[0050] In the second stage, the video tokenizer receives continuous feature vectors by using a vector quantizer in VQ-VAE and queries a pre-trained visual codebook containing multiple discrete visual concept codes. It maps each continuous feature vector to the discrete visual concept codebook that best matches it, thereby realizing the transformation from continuous features to discrete codes. Finally, the entire video frame sequence is transformed into a video token sequence arranged in temporal and spatial order.

[0051] (2) The latent action model (LAM) is used to determine the pilot's latent action token sequence based on the video frame sequence of the input flight environment.

[0052] Among them, the pilot latent action token sequence refers to the structured action semantic sequence inferred from the flight environment video by the latent action model, represented in the form of discrete tokens, and corresponding to the pilot's standard operating intentions and physical control inputs.

[0053] Standard operating intent refers to the flight mission or objective that a pilot intends to accomplish in a specific flight phase or situation, which has a clear semantic meaning; that is, the high-level potential action token sequence described below.

[0054] Physical control inputs refer to the physical action commands that a pilot actually applies to the flight control system to achieve their operational intentions; these commands are directly measurable or executable, i.e., the low-level potential action token sequences described below.

[0055] For example, this potential motion model can identify and encode complete motion semantics from low-level specific operations to high-level mission intentions. For instance, low-level actions include specific control commands such as pushing / pulling the joystick forward / backward (corresponding to pitch and speed changes), pushing / retracting the throttle forward / backward (corresponding to thrust and noise response), pressing the rudder left / right (corresponding to yaw and coordinated turn), lowering / retracting the flaps (corresponding to changes in lift and drag characteristics), landing gear operation (corresponding to drag and landing status), spoiler / thrust reverser control (corresponding to ground deceleration), and trim adjustment (corresponding to pitch balance); high-level actions correspond to mission-level intentions such as switching autopilot modes, navigation and communication settings, TCAS / APU system operations, and executing emergency procedures.

[0056] In one possible implementation, the latent action model is used to determine the pilot's latent action token sequence based on the input flight environment video frame sequence. This can be achieved by: the model receiving the video token sequence output by a video segmenter and analyzing the state evolution patterns between historical frames through its internal network; during the training phase, the model learns the most compact interpretation of the change from the current state to the next state in an unsupervised manner, or, with partial operation labels, semantically aligning the learned latent action representation with operation data from the aircraft's actual physical control equipment through supervised or weakly supervised signals; finally, during inference, the model outputs a latent action token sequence that represents the pilot's operational intentions and control commands in a discrete action semantic space.

[0057] It should be noted that the pilot's latent action token sequence can include high-level latent action token sequences and low-level latent action token sequences, and the specific training methods related to its layering will be explained in the subsequent detailed description of latent action model training.

[0058] (3) The dynamic environment prediction model is used to determine the next frame video token sequence based on the video token sequence and the pilot's potential action token sequence.

[0059] In one possible implementation, the dynamic environment prediction model is used to determine the next frame's video token sequence based on the video token sequence and the pilot's potential action token sequence. Specifically, the dynamic environment prediction model is built on the MaskGIT framework, the core of which is a decoder based on the ST-Transformer framework. The decoder uses the historical video token sequence and the pilot's potential action token sequence as joint conditional inputs, and ensures through a temporal causal masking mechanism that it can only access current and previous historical information when predicting the token at each moment, thereby gradually generating the complete token sequence of future frames in an autoregressive manner.

[0060] Based on (1)-(3), the video segmenter first encodes the video frame sequence of the flight environment into a discrete video token sequence to structurally represent the flight environment state; the latent action model infers the pilot's potential action token sequence based on the video sequence, establishing a semantic representation of the operational intent or action; the dynamic environment prediction model uses the historical video token sequence and the potential action token sequence as joint inputs to autoregressively predict the video token sequence of future frames, thereby achieving accurate simulation of the environmental state evolution caused by the potential action token sequence. This process realizes end-to-end data-driven modeling from pilot operation to environmental response in the token semantic space.

[0061] The training methods for the video segmenter, latent action model, and dynamic environment prediction model video segmenter in the flight training simulation model of this application will be introduced next.

[0062] In some embodiments, the video tokenizer in the flight training simulation model can be trained as follows: based on a large number of flight environment video frame sequences, unsupervised training is performed using a VQ-VAE architecture. The loss function of the video tokenizer consists of three parts: reconstruction loss, codebook update loss, and commitment loss. The reconstruction loss constrains the reconstructed frames output by the decoder to be as consistent as possible with the original input frames, while optimizing both the encoder and decoder. The codebook update loss makes the embedding vectors in the codebook converge towards the continuous features output by the encoder to update the codebook. The commitment loss constrains the output features of the encoder to stably approximate its selected codebook embedding vectors to optimize the encoder. By jointly optimizing the above losses, the video tokenizer learns the ability to efficiently compress continuous video frames into discrete and semantically clear video token sequences.

[0063] It should be noted that the video word segmenter mentioned above uses VQ. The VAE architecture mainly consists of three parts: an encoder, a vector quantization module (including a learnable codebook), and a decoder. The encoder maps the input video frames to continuous features, the vector quantization module discretizes them using the codebook, and the decoder is responsible for reconstructing the video frames from the discrete codes. These three components work together to achieve efficient vision processing. Discrete representation conversion.

[0064] In one possible implementation, as described in Formula 1, the specific expression for the loss function of the video word segmenter can be:

[0065] in, This refers to the loss function value of the video segmenter. This refers to the reconstruction loss, the purpose of which is to allow the decoder to pass the quantized discrete encoding. Reconstructing the input and output The outputs should be as similar as possible to the encoder and decoder. This refers to codebook loss, which aims to improve the embedding vectors in the codebook. Output to encoder To be close enough to cover the distribution of the encoder output, This indicates stopping the gradient; only here... Towards Move, the encoder does not update, only the codebook is updated. This refers to the commitment loss, the purpose of which is to constrain the encoder output. Avoid arbitrary drifting; ensure it stably approximates the selected codebook vector. , here express No updates, only encoder to fixed Proximity, used only for encoders. Among them, and These are weighted variables.

[0066] In another possible implementation, in the above formula 1 When the distribution is Gaussian, then Equivalent to mean square error As shown in Equation 2 under this distribution, another specific expression for the loss function of the video segmenter can be:

[0067] in, Represents the original input data Output data from model reconstruction The square of the Euclidean norm between them is the mean square error. With Formula 1 The reason for their mathematical equivalence is that both essentially approximate the original input. The other variables in Formula 2 are the same as in Formula 1 above, and will not be elaborated upon here.

[0068] The video segmenter provided in this application adopts a loss function based on VQ-VAE. By co-optimizing the reconstruction loss, codebook loss and commitment loss, it can not only ensure high-quality reconstruction of video frames, but also learn compact and stable discrete representations. At the same time, it uses a stopping gradient mechanism to decouple codebook updates from encoder training, avoid optimization conflicts, and improve the training stability of the model in video generation and editing tasks.

[0069] In some embodiments, such as Figure 3 As shown, the potential motion model in the flight training simulation model is trained in the following way: S301, Obtain real pilot operation data.

[0070] In one possible implementation, continuous measurement signals from the pilot's interaction with physical control devices are collected; these signals constitute the actual pilot's operational data. This actual pilot operational data and the corresponding video frame sequence require strict timestamp alignment, thus using the video frame sequence and operational signal sequence as input to a potential motion model. The physical control devices include, but are not limited to, the control stick, throttle lever, rudder pedal, flap handle, trim wheel, and various switches.

[0071] For example, when a video frame shows the aircraft's pitch angle increasing and altitude rising, the corresponding continuous signal of "pull back the control stick by X degrees" serves as a low-level action label, teaching the model to map this visual change to specific pitch control. When the video shows the landing gear indicator light changing from red to green, the corresponding discrete signal of "lower the landing gear handle" serves as a high-level action label, teaching the model to map this state change to system commands. These labels enable the model to learn a fine-grained mapping from visual observation to standardized operations.

[0072] S302. Based on the video frame sequence of the flight environment in the training data and the real pilot operation data, the potential action token sequence of the pilot is obtained through the potential action model.

[0073] In one possible implementation, the latent action model obtains the pilot's latent action token sequence through supervised learning. Specifically, the latent action model takes a time-aligned sequence of flight environment video frames as input and uses the corresponding real pilot operation data, after encoding and discretization, to form a real operation token sequence as the supervision target. Through training and optimization, the latent action model gradually learns to infer and generate a pilot latent action token sequence that is semantically aligned with the real operation logic from visual observation.

[0074] In some embodiments, the potential action model includes: low-level action branches and high-level action branches.

[0075] The low-level action branch is used to determine the low-level potential action token sequence from the input video frame sequence of the flight environment. The high-level action branch is used to determine the high-level potential action token sequence from the input video frame sequence of the flight environment.

[0076] For example, the high-level potential action token sequence can be a sequence representing high-level semantics, such as "perform a landing approach," "make an emergency climb," or "maintain cruise level flight," which characterizes the strategic and phased objectives of the operation. The low-level potential action token sequence can be a sequence of specific control quantities, such as "pull back the stick by X degrees," "press the left lever by Y degrees," or "reduce the throttle to Z%," which characterizes the specific execution steps required to achieve the high-level intent.

[0077] In one possible implementation, a sequence of video frames from the flight environment is input into a latent action model. The model classifies the video frame sequence, assigning video frame sequences identified as low-level related to a low-level action branch for processing, and video frame sequences identified as high-level related to a high-level action branch for processing, thereby training the respective action branches based on the respective video frame sequences.

[0078] S303. In the training of the low-level action branches of the potential action model, determine the first loss value.

[0079] The first loss value is used to characterize the loss relationship between the low-level potential action token sequence and the low-level action data in the real pilot operation data.

[0080] In some implementations, the low-level action branch of the latent action model in the flight training simulation model is trained as follows: Based on training data with low-level operation labels, supervised training is performed using a VQ-VAE-based architecture. The loss function of the low-level branch consists of three parts: vector quantization loss, commitment loss, and alignment loss. The vector quantization loss constrains the matching error between the encoder output and the low-level action codebook entries to optimize the quantization effect. The commitment loss constrains the encoder output to stably correspond to the selected codebook entries to prevent encoder fluctuations. The alignment loss constrains the low-level operation signal reconstructed by the decoder from the quantized latent actions to be as consistent as possible with the real pilot's low-level operation data to achieve physical interpretability of the action representation. By jointly optimizing the first loss value determined by the above losses, the low-level action branch learns to generate a sequence of low-level latent action tokens that are numerically and semantically aligned with the fine-grained operations of the real pilot.

[0081] In one possible implementation, the first loss value can be obtained through the loss function of the lower-level branches, as shown in Equation 3. The specific expression of this lower-level loss function is:

[0082] in, This represents the first loss value; The vector quantization loss represents the low-level action and is used to measure the matching error between the encoder output feature and its nearest neighbor entry in the low-level action codebook. This represents the commitment loss of low-level actions, used to constrain the encoder output to stably correspond to the selected codebook entries, preventing encoder fluctuations. This represents the low-level action alignment loss, where The data represents the actual low-level pilot operations at time t. The loss term is the operation signal obtained by decoding and restoring the low-level potential action output by the model at time t. This loss term is used to force the potential action to numerically approximate the real operation. and Used to adjust the weights of commitment loss and alignment loss in the total loss.

[0083] S304. In the training of high-level action branches of the latent action model, determine the second loss value.

[0084] The second loss value is used to characterize the loss relationship between the high-level potential action token sequence and the high-level action data in the real pilot operation data.

[0085] In one possible implementation, the second loss value can be obtained through the loss function of the higher-level branch, as shown in Equation 4. The specific expression of the loss function of the higher-level branch is as follows:

[0086] Among them, vector quantization loss Used to optimize the matching degree between the high-level action encoder and the corresponding high-level semantic codebook; commitment loss Used to stabilize the correspondence between the output of the high-level encoder and the entries in the high-level codebook; sequence similarity loss. Used to measure the high-level latent action sequence of the model output. Corresponding high-level action label sequence The semantic differences between them; among them, Represents cross-entropy loss, as well as These are the adjustable weights corresponding to the commitment loss and sequence similarity loss, respectively. This loss function aims to enable the high-level action branches to learn a high-level potential action token sequence that has both a compact discrete structure and accurately reflects the pilot's mission phase or operational intention.

[0087] S305. By minimizing the first loss value and / or the second loss value, a well-trained potential action model is obtained.

[0088] In some implementations, during the training of the latent action model, the first loss value and / or the second loss value are continuously minimized, and the encoder, codebook, and decoder parameters of the low-level and high-level branches in the latent action model are iteratively updated. When the loss value converges to a preset threshold or the training reaches a specified number of rounds, the model learns a stable mapping relationship from the flight environment video frame sequence to the hierarchical latent action token sequence, that is, a well-trained latent action model is obtained.

[0089] In one possible implementation, during the training of the latent action model, the overall objective function, composed of a first loss value and a second loss value, is continuously optimized. For example, given a video of an aircraft beginning to climb and its corresponding labels (lower layer: specific stick and rudder inputs; higher layer: "climb" intention), after the model obtains hierarchical action representations through forward inference, it calculates the numerical and semantic differences between these representations and the actual labels as the first and second loss values, respectively, which constitute the overall objective function. All branch parameters are then jointly updated through backpropagation. When the overall objective function converges to a predetermined threshold, the latent action model has learned to stably map precise and interpretable hierarchical action token sequences such as [pull back stick_X, throttle_Y] and [climb] from visual input, thus completing the training.

[0090] In other implementations, the latent action model obtains the pilot's latent action token sequence through unsupervised learning. The training of the latent action model does not rely on any real pilot operation data as supervision labels. It is based on the state mapping relationship between the current flight environment video frame sequence and the future frame sequence, and infers the core control factors that best explain the state evolution through self-supervised objectives. The discrete representation of the factor is used as the pilot's latent action token sequence to complete the training of the latent action model.

[0091] In other implementations, the latent action model can be trained using a combination of unsupervised, supervised, and weakly supervised methods. By performing mixed training or phased training on data from different sources and with varying levels of annotation, the latent action token sequences output by the model can possess both the generalization ability to infer causal relationships from visual dynamics and the alignment with real-world operational logic, thereby obtaining a latent action model that meets the training requirements.

[0092] In steps S301-S305, the potential action model is trained hierarchically, enabling the model to simultaneously grasp both the high-level mission semantics of pilot operations and the low-level physical control details. This hierarchical action representation not only enhances the interpretability of the model's behavior but also provides semantically clear input to the downstream dynamic environment prediction model, thereby driving the generated environment evolution to better fit the real flight environment and effectively improving the realism of flight training simulation.

[0093] In some embodiments, such as Figure 4 As shown, the dynamic environment prediction model in the flight training simulation model is trained in the following way: S401, Obtain the video token sequence and the pilot's potential action token sequence.

[0094] The video token sequence includes the historical video token sequence and the actual next frame video token sequence.

[0095] S402. Based on the video token sequence and the pilot's potential action token sequence, the next frame video token sequence is predicted through a dynamic environment prediction model.

[0096] In one possible implementation, the historical video token sequence and the pilot's potential action token sequence are used as inputs to the dynamic environment prediction model, and the actual next frame video token sequence is used as the label for training the dynamic environment prediction model.

[0097] For example, the dynamic environment prediction model is built on the MaskGIT framework, which takes the historical video token sequence and the pilot's potential action token sequence as joint conditional inputs, and predicts the next frame video token sequence in an autoregressive manner under the constraint of temporal causality masking through the ST-Transformer decoder.

[0098] S403. Determine the prediction loss value based on the predicted next frame video token sequence and the actual next frame video token sequence.

[0099] In one possible implementation, based on the predicted next-frame video token sequence output by the dynamic environment prediction model and the actual next-frame video token sequence, a prediction loss value is calculated using a preset prediction loss function to optimize the model parameters, thereby training the dynamic environment prediction model. As shown in Equation 5, the specific expression of this prediction loss function is:

[0100] in, This represents the model's total prediction loss across all spatial locations; For the model at position The predicted video token distribution; The actual video token for the corresponding location; This represents the spatial location index in the video token sequence. Formula 5 is essentially a representation of the loss cross-entropy, i.e. Equivalent to CE ( , ), indicating that in a given historical video token sequence With potential action sequences Under the condition of [condition], for the next frame's real token sequence With predicted sequence The difference between them is measured. By minimizing this loss, the dynamic environment prediction model can progressively improve the accuracy and stability of its multi-step visual prediction based on historical information and action input.

[0101] S404. By minimizing the prediction loss value, a well-trained dynamic environment prediction model is obtained.

[0102] In one possible implementation, when the predicted loss value converges to a preset threshold or the training reaches a preset number of training rounds, the dynamic environment prediction model has learned the accurate mapping rules from the historical video token sequence and the pilot's potential action token sequence to the future frame video token sequence, that is, the trained dynamic environment prediction model is obtained.

[0103] In steps S401-S404, the dynamic environment prediction model is trained to accurately generate future environmental evolution based on historical states and pilot operational intentions. This training enables the simulation system to reflect the dynamic response under any operational command in real time and realistically, providing pilots with a high-fidelity, interactive, and physically consistent virtual training environment.

[0104] In summary, during the training process of the flight training simulation model in this application, a comprehensive total loss value is obtained by summing the loss values ​​generated by the video segmenter, the latent action model, and the dynamic environment prediction model at their respective training stages. This total loss value is monitored and compared with a preset total training threshold for optimization. When the total loss value converges to below this threshold, it is determined that each sub-model has achieved the expected training objective, thus identifying a well-trained flight training simulation model. This flight training simulation model exhibits coordinated overall performance and can stably perform high-fidelity environment simulation and response. The specific expression for this total loss value is shown in Formula 6, and can be:

[0105] in, This is the total loss value; It can be obtained from formula 1 or formula 2; It can be obtained from Formula 3; It can be obtained from Formula 4; It can be obtained from Formula 5.

[0106] It should be noted that adjustments to any part of the flight training simulation model during training may affect the setting of its corresponding threshold; in addition, the total loss value can be composed of at least two sub-losses, and the specific combination method can be dynamically determined according to actual training needs and effects, which is not limited in this application.

[0107] In some embodiments, the flight training simulation model further includes a multimodal encoder; the multimodal encoder is used to encode multimodal special case events in the training of the flight training simulation model into a sequence of event tokens.

[0108] Among them, multimodal special events include at least one of the following: text commands, voice commands, images, structured documents, and sensor data streams.

[0109] Examples of text commands include: an instructor inputting commands such as "Left engine failure" or "Maintain altitude 3000 meters" in text format. An example of a voice command is given by the instructor via microphone: "Initiate engine fire alarm emergency." An example of an image command is uploaded: a screenshot containing a specific weather radar image is uploaded to indicate an area of ​​thunderstorms ahead. An example of a structured file is imported: a navigation database file conforming to standards such as ARINC 424, or a JSON / XML mission script describing the flight plan is imported. An example of sensor data streams is received in real time: simulated or real aircraft sensor data, such as timing signals of sudden drops in fuel flow, excessive engine vibration, and abnormal drops in cabin pressure.

[0110] In one possible implementation, the multimodal encoder is a Transformer-based encoder. The Transformer is a deep learning architecture based on a self-attention mechanism. Its core function is to capture global dependencies by calculating the association weights between all elements in the input sequence, without relying on the sequential processing order of recurrent neural networks. It mainly consists of stacked encoders and decoders. The self-attention layer allows the model to dynamically focus on key information at different locations, while the feedforward neural network performs non-linear transformations on the features, thereby achieving efficient context encoding and representation learning for sequential data such as text, speech, and images.

[0111] In some embodiments, the multimodal encoder includes at least one of the following (1) to (5): (1) A text encoder is used to encode text instructions in multimodal special case events into a text token sequence.

[0112] In one possible implementation, the text encoder is based on a pre-trained language model with a Transformer architecture, capable of converting input natural language instructions into semantically rich text token sequences. This text token sequence serves as a high-level semantic conditional input, enabling the flight training simulation model to accurately identify operational intentions and environmental setting requirements related to the flight mission, thereby guiding the dynamic environment prediction model to generate simulation scene evolution that conforms to the semantics of the instruction.

[0113] (2) A speech encoder is used to encode speech instructions in multimodal special events into a speech token sequence.

[0114] In one possible implementation, a self-supervised learning pre-trained speech encoder (such as Wav2Vec2.0) extracts robust acoustic and linguistic features from the raw speech signal, mapping the pilot's or instructor's verbal instructions (such as "prepare to land") into a semantically rich speech token sequence. This speech token sequence can be further aligned to the text semantic space through a lightweight adaptation layer, ultimately transforming into corresponding high-level flight action intentions or specific event condition tokens, which serve as multimodal conditional inputs to guide the dynamic environment prediction model in generating scene evolutions that conform to the speech instructions.

[0115] (3) Image encoder, used to encode image instructions in multimodal special case events into image token sequences.

[0116] In one possible implementation, the image encoder employs vision. The language pre-trained model can simultaneously process image instructions with explanatory text. Through alignment learning, this model fuses the visual content of an image with its associated textual description into a unified semantic representation. It then outputs a sequence of image tokens that contains both visual details and semantic text, serving as event condition input to precisely guide the dynamic environment prediction model in generating training scenes that conform to the image and textual descriptions.

[0117] (4) File encoder, used to encode structured files in multimodal special case events into a sequence of file tokens.

[0118] In one possible implementation, the file encoder is designed for structured or semi-structured documents such as weather bulletins, training configuration files, and flight notices. By combining rule parsing and sequence models, it extracts key information (such as weather conditions, training subject parameters, and airspace restrictions) and maps it into a semantically rich sequence of file tokens. This sequence serves as a structured condition input and can be directly used to dynamically adjust training environment parameters or generate corresponding action tokens, thereby achieving precise and automated simulation initialization and condition control of the scenario described by the file content.

[0119] (5) Sensor data encoder, used to encode sensor data streams in multimodal special events into a data stream token sequence.

[0120] In one possible implementation, the sensor data encoder uses models such as recurrent neural networks or temporal Transformers to model continuous time series data such as wind speed, engine speed, and airborne alarms in real time, encoding the multi-channel sensor data stream into a data stream token sequence with temporal context. The real-time system state and abnormal modes represented by this sequence can be used as key condition inputs to directly affect the low-level action generation of the potential action model, and constrain the physical consistency simulation of the dynamic environment prediction model for scenarios such as system failure and performance degradation.

[0121] In one possible implementation, the multimodal encoder may further include a video encoder. This encoder employs a self-supervised pre-trained visual backbone network such as DINOv2 to extract highly generalizable universal visual features frame-by-frame. Then, it utilizes a temporal model to fuse and model the inter-frame features, thereby obtaining a video-level representation rich in spatiotemporal semantics. Finally, this representation is encoded into a sequence of video tokens, which serves as event conditional inputs describing continuous dynamic scenes, thereby enhancing the dynamic environment prediction model's ability to perceive and generate complex, long-range visual events.

[0122] It should be noted that the video encoder can be configured as a dedicated encoder for video modes in a multimodal encoder, or it can be reused as the encoder structure used by the video word segmenter in this application. This application does not limit its specific implementation form.

[0123] In one embodiment, the dynamic environment prediction model is used to determine the predicted next frame token sequence based on the video token sequence and the pilot's potential action token sequence, including: updating the predicted next frame token sequence through the dynamic environment prediction model based on the video token sequence, the pilot's potential action token sequence, and the event token sequence.

[0124] In one possible implementation, the event token sequence serves as a crucial conditional input, directly influencing the generation process of the dynamic environment prediction model. This sequence is concatenated or fused with cross-attention by historical video token sequences and latent action token sequences, providing the model with explicit contextual semantics and external constraints (such as situation descriptions and environmental mutation instructions). This guides the model to maintain a high degree of consistency between its visual content, physical state evolution, and the scene requirements defined by the event when generating the next frame's token sequence autoregressively, thereby achieving accurate simulation of environmental dynamics triggered by specific multimodal events.

[0125] In another embodiment, the event token sequence generated by the multimodal encoder directly influences the reasoning process of the latent action model. This sequence, as a high-level semantic condition, is input into the latent action model along with visual observations, guiding and modulating the model's internal attention mechanism. This causes the model to prioritize the specific context, constraints, or intentions described by the event (such as "engine failure" or "entering a thunderstorm area") when inferring pilot actions, thereby outputting a latent action token sequence that is semantically and temporally strongly correlated with the event. This ensures that the generated operational logic conforms to the current sudden or pre-set special circumstances.

[0126] For example, a specific implementation plan is given based on the above-mentioned multimodal special situation event injection. External instructors can trigger various special situation events through input such as text, voice, or files. After these events are converted into event token sequences by a multimodal encoder, they can be deeply integrated into the simulation loop through two paths: On the one hand, the event tokens, as strong conditions, are input into the dynamic environment prediction model together with the current flight state and the pilot's potential action tokens, directly driving the virtual environment to generate visual, system, and alarm feedback that matches the event in real time; on the other hand, the environmental state changes caused by the event will be fed back to the potential action model in real time, constraining and guiding it to subsequently infer the pilot's operational intentions that are consistent with the special situation.

[0127] Among these, various special events may include: single engine failure, dual engine failure, insufficient fuel, engine fire, landing gear failure, wet runway, low visibility / dense fog, wind shear, bird strike, severe turbulence, electrical failure, navigation equipment failure, cabin medical / conflict incidents, weather radar severe convection warning, crew incapacitation, abnormal airframe structure, airspace restrictions, airspeed indicator failure, hydraulic / lubricating oil failure, and terrain approach warning.

[0128] For example, when an instructor triggers a single-engine failure event via text command "left engine failure" or a simulated sensor, the system renders in real-time the exhaust smoke (or flame) and noise abrupt changes of the failed engine in the external environment, and simulates a slight yaw caused by thrust asymmetry. The aircraft system's response includes a sharp drop in engine N1 / N2 indications, a sudden increase in EGT, triggering fuel flow and fuel pressure warnings, and the electronic centralized aircraft monitor (ECAM) displaying "ENG 1 FAIL" or "ENG 1 FIRE" messages. The autothrottle / autopilot may automatically disengage or switch to fault mode. The cockpit simultaneously activates the main warning lights, illuminated signs, and fire alarm (if accompanied by a fire). During training, the system records the pilot's fault identification delay (in seconds), the accuracy of executing key procedures (fuel shut-off, backup pump activation, thrust conversion), and yaw control accuracy measured by rudder symmetry, thereby completing a quantitative assessment of the pilot's emergency response capabilities.

[0129] For example, in the event of a landing gear malfunction, when the malfunction is triggered by a sensor error or instructor intervention, the external rendering shows the landing gear remaining in the retracted position. Upon landing, it displays an animation indicating that the gear cannot be lowered and simulates a tire scuffing incident. The landing gear indicator lights malfunction, triggering an "unsafe" alarm, and the ECAM prompts for a backup manual deployment procedure. The cockpit simultaneously activates the main warning / alarm and manual deployment check prompts. Training metrics focus on the accuracy of pilots in executing emergency deployment procedures, as well as the operational effectiveness of alternate landing selection and taxiing deceleration management.

[0130] For example, when a severe convective weather warning is issued by the weather radar, a thunderstorm event is triggered based on the meteorological input or images, and visual effects of thunderstorm clouds, lightning, and heavy precipitation areas are generated in the external rendering. The cockpit system simultaneously illuminates the weather radar warning on the navigation display (ND) and recommends an evasive route, while also indicating decreased visibility and increased turbulence level. In this scenario, the training indicators focus on the pilot's decision-making speed in reasonably changing the route and avoiding dangerous areas.

[0131] This application allows instructors to trigger emergency scenarios such as single-engine failure, heavy fog, and thunderstorms in real time, thereby automatically updating the environment and enabling pilots to conduct highly immersive emergency response training.

[0132] In some embodiments, the flight training simulation model further includes a historical data memory; the historical data memory is used to provide a sequence of historical scenario tokens during the training of the flight training simulation model.

[0133] The historical scenario token sequence is a discrete semantic record of verified and structured past flight training segments stored in a historical data memory. This historical scenario token sequence can be at least one of the following: a historical video token sequence, a historical potential action token sequence, a historical event token sequence, or a historical state evolution token sequence.

[0134] The historical video token sequence is a visual semantic encoding of several consecutive frames of the flight environment (such as external view and cockpit instruments) recorded during a past training session. The historical latent action token sequence is a semantic encoding of pilot operations (including high-level intentions and low-level executions) inferred from the latent action model, synchronously corresponding to the above video sequences. The historical event token sequence is a semantic encoding of multimodal incident events (such as "engine failure" and "wind shear") triggered in this segment. The historical state evolution token sequence is an abstract representation of key flight state parameters (such as altitude, speed, and attitude) further derived from the video and action sequences.

[0135] Through a long-term memory mechanism, it breaks through the limitations of traditional world models in short-term prediction, supporting training on complete routes and long-range flights; the multi-scale video segmenter ensures that details such as cockpit panel warning lights and instruments are not lost, while maintaining the high efficiency of external environment generation.

[0136] In one embodiment, the dynamic environment prediction model is used to determine the next frame token sequence based on the video token sequence and the pilot's potential action token sequence, including: retrieving multiple historical scenario token sequences from a historical data memory according to a preset similarity algorithm based on the video token sequence and the pilot's potential action token sequence; and updating the predicted next frame token sequence through the dynamic environment prediction model based on the video token sequence, the pilot's potential action token sequence, and the multiple historical scenario token sequences.

[0137] The preset similarity algorithm can be cosine similarity matching or sequence matching based on attention mechanism, etc.

[0138] For example, cosine similarity matching encodes the current video token sequence and the potential action token sequence into fixed-length feature vectors, calculates their cosine similarity with the corresponding vectors of each scene in the historical data memory, and selects the several historical scene token sequences with the highest similarity. Sequence matching based on an attention mechanism dynamically calculates the relevance weights of the current token sequence and each historical scene token sequence at each position through a cross-attention layer, and selects the several historical scene token sequences with the best semantic matching based on the comprehensive attention scores.

[0139] In one possible implementation, multiple retrieved historical scenario token sequences are used as additional conditional context, and are either concatenated with the current video token sequence and the pilot's potential action token sequence, or fused through a cross-attention mechanism. This fusion process enables the dynamic environment prediction model to generate the next frame token autoregressively, based not only on the current state and operational intent, but also on state evolution patterns and operations in similar historical scenarios. Environmental response patterns.

[0140] For example, in long-range missions, when the dynamic environment prediction model is limited by short-term context and has difficulty generating continuous fuel consumption or gradual weather trends, the historical data memory can retrieve stored historical scenarios such as "long-range low-temperature icing" to drive the model to gradually render processes such as wing icing and pitot tube blockage and trigger corresponding alarms. In the evaluation, it can then compare whether the pilot has performed the correct de-icing operation and generate targeted improvement suggestions, thereby achieving coherent simulation and refined evaluation of long-term, gradual special scenarios.

[0141] In summary, the schematic diagram of the architecture of a flight training simulation model provided in this application is as follows: Figure 5 The video frame sequence is used as input, and the video segmenter converts it into a video token sequence for the flight environment. The latent action model includes low-level action branches and high-level action branches, where the high-level action branches output high-level action codebooks, and the low-level action branches output low-level action codebooks. Finally, based on the video token sequence, high-level action codebook, and low-level action codebook, the dynamic environment prediction model outputs a predicted frame.

[0142] This includes a controller that influences the dynamic environment prediction model based on a historical data memory. The inputs include text encoders for text commands, speech encoders for voice commands, image encoders for image inputs, video encoders for video inputs, structured file encoders for file inputs, and sensor data encoders for sensor data inputs. The data encoded by these encoders affects higher-level action branches and the dynamic environment prediction model, while the data processed by the sensor data encoder affects lower-level action branches and the dynamic environment prediction model.

[0143] This application also provides a flight training simulation method, applied to... Figure 1 The flight training simulation application device shown has the flight training simulation model provided in the above embodiments deployed on it. Figure 6 As shown, the flight training simulation method includes: S601, Acquire flight environment data.

[0144] The flight environment data includes video frame sequences of the flight environment and pilot operation data.

[0145] S602. Based on flight environment data, determine the sequence of video tokens for the next frame through a flight training simulation model.

[0146] The flight training simulation model is obtained based on the training method of any of the foregoing embodiments of this application.

[0147] In one possible implementation, the current flight environment video frame sequence is first encoded into a historical video token sequence using a video segmenter in a flight training simulation model. Simultaneously, a latent action model infers a pilot latent action token sequence based on the video frame sequence. Subsequently, a dynamic environment prediction model uses the historical video token sequence and the pilot latent action token sequence as joint input to autoregressively generate a predicted next frame video token sequence. The pilot latent action token sequence includes both high-level and low-level latent action token sequences.

[0148] This application can map pilot operations to potential actions, so that every pilot maneuver can receive real-time feedback in the generated simulation environment; the potential actions are consistent with the pilot's real operating logic, which can avoid "phantom actions" or unreasonable automated reactions and improve the realism of the simulation.

[0149] In one possible implementation, when a multimodal special event is injected, the multimodal encoder encodes the received text, voice, image, file, or sensor data stream into an event token sequence; the dynamic environment prediction model uses the historical video token sequence, the pilot's potential action token sequence, and the event token sequence as joint conditional inputs to generate a predicted next frame video token sequence that matches the event semantics, thereby achieving real-time and accurate simulation of sudden or pre-set special situation scenarios.

[0150] S603. Update the flight training simulation environment based on the next frame video token sequence.

[0151] In one possible implementation, based on the predicted next frame video token sequence, the decoder corresponding to the video segmenter in the flight training simulation model restores it to pixel-level video frames, and drives the graphics rendering engine and the cockpit instrument system to synchronously update the visual scene, instrument readings and alarm status, thereby completing the real-time and coherent iteration of the flight training simulation environment and providing pilots with a high-fidelity and dynamically responsive virtual training environment.

[0152] In some implementations, different levels of quantitative thresholds are set to systematically evaluate pilots' basic operational precision, procedural execution standardization, emergency response timeliness, and high-level behavioral consistency. After the evaluation is completed, a structured report is automatically generated, integrating scenario summaries, multi-dimensional replays, sub-item scores, and personalized improvement suggestions, and can dynamically adjust the evaluation weights according to training objectives.

[0153] For example, in terms of basic control, the root mean square value of heading hold deviation is required to be no more than X degrees, the root mean square value of altitude hold deviation is required to be no more than Y feet, and the speed deviation is required to be no more than Z knots; during approach and landing, the touchdown point error must be within N meters, the number of glide slope instability incidents must be controlled, and the longitudinal velocity of touchdown must be monitored to limit the occurrence of a hard landing; in terms of emergency response, the recognition delay from the triggering of an event to the first cockpit response is required to be no more than T1 seconds, the percentage of correct steps executed on the checklist must be no less than p%, and the sequence of actions must be accurate; management capabilities include the rationality of diversion decisions in fuel shortage scenarios, and whether the electro-hydraulic switching complies with the manual; cognitive and communication performance is assessed through the timeliness of communication with air traffic control (ATC), flight attendants, and instructors, and the completeness of information (such as the accuracy of critical data broadcasts); behavioral consistency is tested by the similarity between potential actions and actual pilot operations (measured using mean square error or cosine similarity) to verify whether the model effectively captures real control logic; long-term stability is assessed by using historical data memory technology to evaluate trajectory consistency across 30-minute segments and the ability to continuously track weather changes.

[0154] For example, the generated training report will include at least one of the following: a summary page, clearly presenting a summary of the flight path, weather, and event scenarios for this training, as well as an overview of the total score and key points lost; a time-series replay, providing parallel video playback that supports arbitrary jumping and synchronized playback with the dashboard, supplemented by intuitive annotations using red, yellow, and green event markers; itemized scoring, providing independent scores and targeted suggestions for capability dimensions such as basic operation, emergency response, management, and communication collaboration; behavioral suggestions, generating standard operation templates based on common errors (e.g., "In low-temperature icing scenarios, the Pitot tube heating should be checked first to avoid premature acceleration") and providing repeatable micro-tasks (e.g., "Specialized practice on thrust management within 0-60 seconds after single-engine failure takeoff"); and a teacher annotation layer, allowing instructors to add handwritten or voice comments, and to save this complete case as historical experience to a historical data memory for subsequent teaching and analysis.

[0155] For example, rule-based judgments are performed based on ECAM or Quick Reference Handbook (QRH) standard procedures to check the correctness and sequence of steps. A machine learning model is used to calculate the similarity between the pilot's potential actions and the reference operation sequence, generating a data-driven evaluation score. Finally, a weighted sum formula is used to calculate the composite total score, with the weights of each component dynamically adjusted according to the training objective (such as emergency drills, basic operations, or management) to precisely focus on the evaluation priorities.

[0156] In summary, this application provides specific embodiments of some flight training simulation methods, as follows: Exemplarily, a normal takeoff process. The instructor inputs "Clear weather, runway 09L, calm wind" via text. An initial airport visual scene is generated based on the instruction. The pilot pushes the throttle stick forward to the Takeoff / Go-Around (TOGA) position, monitoring the increase in engine parameters. The latent motion model receives environmental information and outputs the low-level motion "maximum thrust, pitch +2°" and the high-level intention "takeoff". The dynamic prediction model combines the environmental frame and the motion to predict the next frame of the aircraft's acceleration roll. The visual scene is updated, showing the aircraft accelerating on the runway and increasing airspeed. When the airspeed reaches the nose wheel lift speed, the pilot pulls back on the control stick. The model outputs the new motion "maximum thrust, pitch +10°". The image of the aircraft taking off and landing gear retracting is predicted and displayed, accompanied by the "positive rate of climb" voice prompt from the ground proximity warning system (GPWS). Throughout the process, all operation sequences and environmental evolution data are recorded in the background. After the training, the AI ​​coach module compared the standard operating procedure (SOP) with the pilot's operating sequence and pointed out in the evaluation report that "the timing of lifting the front wheel was accurate and the takeoff attitude was stable," and gave a quantitative score.

[0157] For example, handling a single-engine failure situation. During the stable climb phase, the instructor inputs a "right engine failure" command via the event injection module. This command is encoded as a special event token. The next instant, the dynamic prediction model combines this event token with the current flight status to predict a sudden drop in the right engine low-pressure rotor speed (N1) and a right yaw. The visual scene is updated; the right engine fire warning light illuminates, and the aircraft attitude shows a significant right yaw. After recognizing the situation, the pilot immediately executes the memorized items according to the QRH: applying left rudder to counteract the yaw, while simultaneously reducing the right engine throttle to idle. The latent action model maps these actions to the higher-level intention "execute the single-engine failure checklist" and corresponding lower-level control commands. Integrating the environment, actions, and event token, the model predicts the aircraft gradually returning to equilibrium. In the visual scene, the aircraft yaw is controlled, and the primary flight display (PFD) and navigation display (ND) show flight parameters stabilizing. The SOP conformance module in the background verifies in real time whether the operation steps conform to the process sequence specified by QRH, while the safety constraint module simultaneously monitors for any prohibited actions that violate the energy isolation principle (such as accidentally reducing normal engine throttle). Finally, the evaluation module generates a report: "The emergency was identified in a timely manner, and the handling sequence conforms to QRH. There were no violations of safety regulations. Improvement: The throttle reduction to idle position could be advanced by approximately 2 seconds." For example, an instrument approach procedure under complex weather conditions. During the cruise phase, the instructor injects an emergency by issuing a voice command, "Thunderstorm ahead." The voice recognition module converts the command into a text token. The weather radar layer on the ND is updated, displaying a red echo area. The pilot decides to detour according to the SOP and requests a heading change by selecting a new frequency via the audio panel. After receiving simulated ATC voice clearance, the pilot inputs a new waypoint via the multifunction control and display unit (MCDU). The latent action model maps this series of actions to the higher-level intention, "Perform detour and heading change." The dynamic prediction model combines the weather event token with the pilot's actions to predict a continuous visual scenario of the aircraft smoothly turning to avoid the thunderstorm area. During the process, the engine indicating and crew alerting system (ECAM) displays normal engine parameters, and the heading indication on the PFD is updated synchronously. After the training, the evaluation report states that "weather information was fully utilized and the detour decision was timely," and scores the communication procedure.

[0158] For example, consider the identification and handling of a fault. During flight, a series of warning sounds suddenly occur in the cockpit, the EICAS main warning light illuminates, and "Hydraulic B Pressure Low" is displayed. The pilot first presses the main warning light to confirm and attempt to reset it. The model processes this action, but predicts that the warning message will be triggered again in the next frame of the visual view. The pilot then follows the checklist according to the QRH, checks the status of relevant pumps and switches, and reports the emergency via the audio panel. This series of actions is interpreted by the potential action model as a high-level intention to "handle the hydraulic fault." The dynamic prediction model integrates the fault event token with the check actions to predict subtle changes in the aircraft's handling characteristics. The pilot perceives the changes through the PFD and control stick and implements compensation. Finally, the evaluation module verifies the checklist steps and assesses the pilot's "situational awareness and status monitoring capability" by analyzing the delay of the pilot's first control surface input after the fault, generating a report containing reaction time data.

[0159] For example, emergency communication and alarm management are based on the audio panel. When the simulation training enters the approach phase, the pilot tunes the communication frequency to the destination tower channel via the audio panel. This operation serves as a control signal input system, and the model processing module accordingly activates the corresponding virtual ATC voice channel. Immediately, the simulated ATC command "Air China 101, cleared for runway 36L" enters the training environment, and the pilot must execute the landing procedure accordingly. Simultaneously, if a ground proximity warning is triggered due to operational deviation, a continuous "Terrain, PullUp" alarm will sound in the cockpit. The pilot can attempt to abort the alarm by pressing the warning reset button (inputting a reset signal). The model will determine whether to immediately silence the alarm or maintain the alarm state based on the current aircraft status (e.g., whether a successful recovery has been achieved). This process simulates the interaction of a real cockpit acoustic environment, requiring the pilot to effectively allocate attention among multiple audio messages and promptly handle critical commands and warnings, achieving high fidelity in the construction of the comprehensive training environment.

[0160] For example, a voice-command-based dynamic route update is implemented. During the cruise phase, the pilot directly issues the command via microphone: "Request a direct route to Beijing Capital International Airport." This voice stream is converted into a text token in real time by an automatic speech recognition system and injected as an event token into the system's event injection module. The latent action model interprets this command as a high-level intent: "Execute route change request." The dynamic environment prediction model then combines the current flight status with this intent to predict and generate a continuous visual context and flight parameters for the aircraft to change to the new route. The system response is manifested in: the planned route on the ND is immediately updated, showing a new segment pointing to the flight path; the heading indicator on the PFD changes synchronously, with the trend vector pointing to the new waypoint; simultaneously, a brief route update notification may sound in the cockpit. This process simulates the voice interaction between the pilot and ATC and demonstrates how the flight training simulation model can transform unstructured natural language commands into precise control conditions that drive the evolution of the simulated environment in real time, showcasing the system's capabilities in advanced human-machine interaction and contextual understanding.

[0161] It should be noted that the above examples illustrate some application scenarios. Other training scenarios, special situation variations, or interaction modes that can be conceived by those skilled in the art based on the core technical concepts of this application are all within the protection scope of this application. This application does not limit the specific number of embodiments, scenario complexity, or operational details.

[0162] This application also provides a training device for a flight training simulation model, which may include one or more functional modules for implementing the methods described in the above method embodiments.

[0163] In an exemplary embodiment, Figure 7 This is a schematic diagram of the composition of a training device for a flight training simulation model provided in an embodiment of this application. The device includes: an acquisition module 701 and a training module 702. The acquisition module 701 is used to: acquire training data; the training data includes a video frame sequence of the flight environment; the training module 702 is used to: train a video tokenizer, a latent action model, and a dynamic environment prediction model in the flight training simulation model based on the training data; wherein, the video tokenizer is used to process the input video frame sequence of the flight environment into a video token sequence; the latent action model is used to determine the pilot's potential action token sequence based on the input video frame sequence of the flight environment; and the dynamic environment prediction model is used to determine the predicted next frame video token sequence based on the video token sequence and the pilot's potential action token sequence.

[0164] This application also provides an electronic device, which may be a flight training model training device or a flight training model application device in the above-mentioned flight training simulation model training system. Figure 8This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 8 As shown, the electronic device includes a processor 801 and a memory 802; the memory 802 stores instructions executable by the processor 801; when the processor 801 is configured to execute the instructions, the electronic device implements the method described in the foregoing method embodiments.

[0165] This application also provides a computer-readable storage medium storing computer program instructions thereon; when the computer program instructions are executed by a computer, the computer causes the computer to implement the methods described in the foregoing embodiments. The computer may be an electronic device, a network device, or a manager. The computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0166] This application also provides a computer program product that, when run on a computer, causes the computer to execute the relevant method steps described in the above method embodiments.

[0167] The electronic devices, computer-readable storage media, or computer program products provided in this application are all used to perform the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0170] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0172] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0173] In the description of the embodiments of this application, specific features, structures, materials or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0174] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a flight training simulation model, characterized in that, The method includes: Acquire training data; the training data includes video frame sequences of the flight environment; Based on the training data, the video segmenter, potential action model, and dynamic environment prediction model in the flight training simulation model are trained. The video tokenizer is used to process the input video frame sequence of the flight environment into a video token sequence; The latent action model is used to determine the pilot's latent action token sequence based on the input video frame sequence of the flight environment; The dynamic environment prediction model is used to determine the next frame video token sequence based on the video token sequence and the pilot's potential action token sequence.

2. The method according to claim 1, characterized in that, The latent action model is trained in the following way: Obtain real pilot operation data; Based on the video frame sequence of the flight environment in the training data and the real pilot operation data, a pilot potential action token sequence is obtained through the potential action model; the potential action model includes: a low-level action branch and a high-level action branch; wherein, the low-level action branch is used to determine the low-level potential action token sequence from the input video frame sequence of the flight environment; the high-level action branch is used to determine the high-level potential action token sequence from the input video frame sequence of the flight environment. In the training of the low-level action branch of the latent action model, a first loss value is determined; the first loss value is used to characterize the loss relationship between the low-level latent action token sequence and the low-level action data in the real pilot operation data. In the training of the high-level action branch of the latent action model, a second loss value is determined; the second loss value is used to characterize the loss relationship between the high-level latent action token sequence and the high-level action data in the real pilot operation data. By minimizing the first loss value and / or the second loss value, a well-trained potential action model is obtained.

3. The method according to claim 1, characterized in that, The dynamic environment prediction model is trained in the following way: Obtain the video token sequence and the pilot's potential action token sequence; the video token sequence includes the historical video token sequence and the actual next frame video token sequence. Based on the video token sequence and the pilot's potential action token sequence, the predicted next frame video token sequence is obtained through the dynamic environment prediction model; Based on the predicted next frame video token sequence and the actual next frame video token sequence, determine the prediction loss value; By minimizing the prediction loss value, a trained dynamic environment prediction model is obtained.

4. The method according to claim 1 or 3, characterized in that, The flight training simulation model also includes a multimodal encoder; the multimodal encoder is used to encode multimodal special case events in the training of the flight training simulation model into a sequence of event tokens; The dynamic environment prediction model is used to determine the next frame token sequence based on the video token sequence and the pilot's potential action token sequence, including: Based on the video token sequence, the pilot's potential action token sequence, and the event token sequence, the dynamic environment prediction model updates and predicts the next frame token sequence.

5. The method according to claim 4, characterized in that, The multimodal encoder includes at least one of the following: A text encoder is used to encode text instructions in the multimodal special case event into a text token sequence; A voice encoder is used to encode voice commands in the multimodal special case events into a voice token sequence; An image encoder is used to encode image instructions in the multimodal special case event into an image token sequence; A file encoder is used to encode structured files in the multimodal special case event into a sequence of file tokens; A sensor data encoder is used to encode the sensor data stream in the multimodal special event into a data stream token sequence.

6. The method according to claim 1 or 3, characterized in that, The flight training simulation model also includes a historical data memory; the historical data memory is used to provide a sequence of historical scenario tokens during the training of the flight training simulation model. The dynamic environment prediction model is used to determine the next frame token sequence based on the video token sequence and the pilot's potential action token sequence, including: Based on the video token sequence and the pilot's potential action token sequence, multiple historical scenario token sequences are retrieved from the historical data memory according to a preset similarity algorithm; Based on the video token sequence, the pilot's potential action token sequence, and the multiple historical scenario token sequences, the dynamic environment prediction model updates and predicts the next frame token sequence.

7. A flight training simulation method, characterized in that, The method includes: Acquire flight environment data; Based on the flight environment data, the next frame video token sequence is determined by the flight training simulation model; wherein, the flight training simulation model is trained based on the method described in any one of claims 1-6; The flight training simulation environment is updated based on the next frame video token sequence.

8. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device implements the training method as described in any one of claims 1-6 or the method as described in claim 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes: computer software instructions; When computer software instructions are executed in an electronic device, the electronic device causes the electronic device to implement the training method as described in any one of claims 1-6 or the method as described in claim 7.

10. A computer program product, characterized in that, The computer program product includes computer programs; When the computer program is run on the device, the device implements the training method as described in any one of claims 1-6 or the method as described in claim 7.

Images