Flight simulator multi-modal data-based feature fusion model construction method

Abstract

The invention belongs to the technical field of pilot training evaluation, and discloses a flight simulator multi-modal data-based feature fusion model construction method, which specifically comprises the following steps of 1, selecting a pilot training video, obtaining flight parameter data, and establishing a flight simulator multi-modal data set; 2, data preprocessing; step 3, providing a key point sequence by the ViTPose; step 4, embedding key point sequence features; 5, extracting flight parameter time sequence and pilot operation behavior frequency domain features; step 6, carrying out cross-modal feature fusion; and 7, performing comparative learning. According to the method, signals of different modes in the use process of the flight simulator are fully captured, and operation behavior characteristics in a pilot training video and data time sequence characteristics generated by the flight simulator are fused, so that more accurate and efficient pilot driving training behavior representation and recognition are realized; and meanwhile, the whole learning process is processed in a layered manner, and the information of different signals can be more deeply and effectively combined through the interaction effect of the mode in the main features of different stages of each layer of model, so that more advanced and abstract feature representation can be obtained in the subsequent hierarchical learning.

Description

Technical Field

[0001] The present invention belongs to the technical field of pilot training and evaluation, and specifically relates to a method for constructing a feature fusion model based on multimodal data of a flight simulator. Background Art

[0002] With the development of aviation technology, the demand for flight safety is increasing. Flight simulators can simulate aircraft control, flight environment and various possible flight conditions, including normal flight, emergency situations and complex weather conditions, for pilots or flight trainees to train or test. As a core tool for pilot training, flight technology research and aviation system testing, they not only improve pilot skills and safety, but also provide valuable support for aircraft design, technology development and flight improvement.

[0003] Current flight evaluation methods are mainly divided into direct observation of trainees' operations by instructors, flight data generated by flight simulators, and analysis of pilots' operating actions to evaluate the effectiveness of flight training. The first method is easily affected by subjective factors, ignoring the operational tasks and problems during pilot training, and the evaluation results often depend on the instructor's personal judgment, which is subjective and arbitrary. The second method relies on sensor accuracy, is susceptible to single-point failures, and has difficulty capturing environmental or human factors. The third method is easily affected by flight missions. Each flight mission has different operating methods, making it difficult to make a comprehensive evaluation of pilot training. Therefore, these evaluation methods may have a significant impact on the results of flight training evaluation, which may not only affect flight safety, but also reduce the efficiency of training. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for constructing a feature fusion model based on multimodal data of a flight simulator to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for constructing a feature fusion model based on multimodal data of a flight simulator, comprising the following specific steps:

[0006] Step 1: Select pilot training videos, obtain flight parameter data, and build a multimodal dataset for the flight simulator

[0007] The data of the flight simulator pilot training process is collected, including multiple and different pilot training videos and flight simulator flight parameter data. The flight training data involves two modal signals: the flight parameter time series signal (t) and the pilot driving behavior video frame signal (f). The data set D is defined, including N time series data {x1, x2, x3, ..., x n}, for each sequence data All include their associated time series features t i and video frame features f i , where x i ∈{t i ,f i}、l m is the sequence length, d m is the representation vector dimension of signal m;

[0008] Step 2: Data preprocessing

[0009] The CSV file of flight parameter data is input into the data selection network at a sampling frequency of 6 times per second. A random number retention method is used to randomly retain data within one second, and the columns containing the required flight parameters are retained. The retained flight parameters are then re-counted. The flight parameter data includes flight altitude, flight simulator attitude data, flight simulator motion data, and pilot operation data. For pilot training videos, the slow-fast network and key frame extraction technology are used to convert them into image sequences.

[0010] Step 3: ViTPose provides key point sequence

[0011] Based on pilot training videos, the ViTPose key point detection algorithm is used to extract the pilot's action key point sequence;

[0012] Step 4: Key point sequence feature embedding

[0013] Design a CA module that generates the attention map of the current layer by introducing the attention map of the previous layer as prior knowledge and uses a 1x1 convolution kernel to obtain more local attention maps;

[0014] Step 5: Extracting frequency domain features of flight parameter time series and pilot operation behavior

[0015] Two Transformer layers are introduced to unify the features of each signal. A basic Transformer network is used to extract the time series features of flight parameters, and a Swin Transformer network is used to extract the frequency domain features of pilot operation behavior.

[0016] Step 6: Cross-modal feature fusion

[0017] Perform cross-modal feature fusion of flight parameter time series features and pilot operation behavior frequency domain features, including CAP, Attention architecture, and global-local interactive learning mode to improve parameter efficiency;

[0018] Step 7: Contrastive Learning

[0019] The CPC unsupervised learning algorithm is used to capture the potential features of the data by maximizing the interactive information of the data and improve the quality of feature expression.

[0020] As a preferred technical solution of the present invention, the flight parameter data described in step one includes flight altitude, flight simulator attitude data, flight simulator motion data, and pilot operation action data.

[0021] As a preferred technical solution of the present invention, the specific method of providing the key point sequence of ViTPose in step 4 is as follows: given an instance image X∈R of a pilot H×W×3 As input, the image is first embedded into tokens through the patch embedding layer, i.e. Where d is the downsampling rate of the patch embedding layer, C is the channel dimension, and the embedded token is then processed by the Transformer layer. Each layer consists of a multi-head self-attention (MHSA) layer and a feed-forward network (FFN), that is:

[0022] F i+1’ =F i +MHSA(LN(F i )), F i+1 =F i+1’ +FFN(LN(F i+1’ ))

[0023] Where i represents the output of the i-th Transformer layer, the initial feature F0 = PatchEmbed(X) represents the feature after the block embedding layer, and the output feature of the backbone network is expressed as

[0024] As a preferred technical solution of the present invention, the CA module described in step 5 is defined as:

[0025]

[0026] Where M i is the CA module of the i-th CA block, which is expected to focus on the muscle movement area, Attn i is the attention graph of the (i-1)th layer, represents the features extracted by the first two convolutional layers of the i-th layer as the input of the CA module, and Describes the characteristics of maximum pooling and average pooling respectively, P M represents the maximum set operation on the attention map of the (i-1)th layer to match the size of the attention map of the current layer, and N represents the product of the elements introduced as prior knowledge of the attention map of the previous layer, F i ∈R 2C×2H×2Wrepresents the input of the i-th CA block. By associating the attention mechanism between adjacent layers, the CA block can gradually and robustly focus on areas with subtle motions, rather than focusing on different areas of the face in different layers;

[0027] Four CA blocks are used to form the main branch of MFNet to learn subtle muscle movement pattern features. Its size is 512×14×14. The CA module composed of a CA module and two convolutional layers can be expressed as:

[0028]

[0029] Where CA represents the proposed continuous attention block, stands for element-wise multiplication, which means Each channel of will be multiplied by the spatial attention map to pay attention to the region of interest, so the difference between the vertex frame and the starting frame is taken as the input of the main branch to learn motion pattern features.

[0030] As a preferred technical solution of the present invention, when introducing two Transformer layers to unify the features of each signal as described in step 5, a low-dimensional token is randomly initialized for each signal. And use Transformer to embed basic modality information:

[0031]

[0032] Where, is the feature learned for each signal, θ m Represents the parameter information of the current Transformer learning, Concat(·) represents the splicing operation, T represents the length, and d represents the dimension.

[0033] As a preferred technical solution of the present invention, the method for extracting the frequency domain features of the pilot's operating behavior using the Swin Transformer network in step 5 is as follows: first, the input RGB image is segmented into non-overlapping patches through the patch segmentation module, each patch is regarded as a "token", and its feature is set as the concatenation of the original pixel RGB value. A linear embedding layer is applied to this original value feature to project it to an arbitrary dimension (denoted as C); a Swin Transformer block is applied to the patch token to maintain the number of tokens. and together with the linear embedding is referred to as “stage1”;

[0034] In order to produce a hierarchical representation, as the network goes deeper, the number of tokens is reduced by patch merging layers. The first patch merging layer connects the features of each group of 2×2 adjacent patches and applies a linear layer on the features of the 4C-dimensional connection, thereby reducing the number of tokens by a factor of 2×2=4 (equivalent to a 2x downsampling of the resolution), and the output dimension is set to 2C. After that, the Swin Transformer block is applied for feature transformation, and the resolution is kept at The first block of block merging and feature transformation is denoted as “stage 2”;

[0035] The process is repeated twice, and the output resolutions are and These stages jointly produce a hierarchical representation, where W-MSA and SW-MSA represent window-based multi-head self-attention using regular and shifted window partitioning configurations, respectively;

[0036] Self-attention in non-overlapping windows,The windows divide the image uniformly in a non-overlapping manner. Assume that each window contains M×M blocks. The computational complexity of the global MSA module and the window-based MSA module (W-MSA) is on the image of h×ω blocks:

[0037] Ω(MSA)=4hωC 2 +2(hω) 2 C

[0038] Ω(W-MSA)=4hωC 2 +2M 2 hωC

[0039] Where the former is quadratic with the number of patches h, while the latter is linear when M is fixed. Global self-attention computation is usually unaffordable for large h, while window-based self-attention is scalable.

[0040] As a preferred technical solution of the present invention, the shifted window partitioning method, when used, maintains efficient computation of non-overlapping windows while introducing cross-window connections. The method alternates between two partitioning configurations in consecutive Swin Transformer blocks:

[0041]

[0042] Where, and Z l Represent the output features of the (S)W-MSA module and the MLP module of block 1, respectively.

[0043] As a preferred technical solution of the present invention, the CAP described in step 6 uses symmetric Cross-Attention to explore the intrinsic correlation between the two input feature sequences. In order to allow further information integration, self-attention is used to simulate the temporal dependency in each feature sequence. CAP uses two sequences H t and H f As input, and output their mutual promotion information H t→f and H f→t , specifically, The calculation of is as follows:

[0044] H′ t→f =MCA(LN(H t ),LN(H f ))+H t

[0045] H″ t→f =MSA(LN(H′ t→f )))+H′ t→f

[0046] H t→f =FFN(LN(H′ t→f )))+H″ t→f

[0047] Where LN represents layer normalization, FFN is the feedforward neural network in Transformer;

[0048] The Attention architecture described in step 6 includes a self-attention mechanism, which is used to calculate the attention weights between elements in a sequence in order to capture the dependencies between elements. The multi-head self-attention mechanism uses the same calculation method for multiple heads and gives different parameters to express features from multiple subspaces, which can capture richer feature information. Taking the time domain signal as an example, the input of the multi-head self-attention mechanism is defined as Among them, d t Represents the encoding dimension of the time series signal. The whole process can be expressed as:

[0049] Q i =H t W q ∈R n×d ,K i =H t W k ∈R n×d ,V i =H t W v ∈R n×d

[0050]

[0051] MSA(H t )=concat(head1,head2…,head n )W o

[0052] Where Q i , K i 、V i They represent the results of the linear transformation of the input vector by the i-th head, W q 、W k 、W v They are the weight parameters of Query, Key, and Value mapping, mapping the input to the d-dimensional output. concat represents the concatenation operation. o is the weight matrix of the final linear transformation,

[0053] It also includes a cross-attention mechanism, which is used to handle the dependency between two different sequences. By learning the directional pairwise attention between the source modality and the target modality, the source modality information is used to strengthen the target modality H. t , i.e. the query comes from the target modality, while the keys and values ​​come from the source modality H v :

[0054] Q j =H t W q ,K j =H f W k ,V j =H f W f

[0055] Provide interaction from the modal in this way:

[0056]

[0057] MSA(H t ,H f )=concat(head1,head2…,head n )W o

[0058] Where H t 、H f Represents different signal features and is the calculation result of the multi-head cross attention mechanism;

[0059] The global-local interactive learning model described in step 6 is specifically as follows: setting global fusion context information Where i represents the number of global-local interaction layers. The entire interaction process is as follows:

[0060]

[0061] In this way, one-to-many global-local interactions can be captured in two (memory protection units). By stacking multiple layers, the global fusion context and local signal features can promote each other and gradually refine themselves. This operation requires M MPUs in each layer. Since the global fusion context length is small, the overall time complexity is reduced to (In fact, there are M< <T m ), which degenerates to O(MT 2 ), therefore, the default global-local fusion strategy in MPU not only has linear spatial complexity, but also enjoys linear computation on the involved signals,

[0062] At the same time, each global-local fusion will pass through the pooling layer to aggregate the enhanced information of different modalities to promote subsequent fusion. Therefore, a fully connected layer based on nonlinear activation is used to implement this operation, and two enhanced global fusion contexts are defined, namely The new global fusion context is obtained as follows:

[0063]

[0064] Where,

[0065] As a preferred technical solution of the present invention, the CPC unsupervised learning algorithm described in step seven includes contrastive learning loss, which learns useful representations of data by maximizing the similarity between positive samples and minimizing the similarity between negative samples. Specifically, given an anchor sample, the goal is to make it more similar to the positive sample and less similar to the negative sample. This can be achieved by calculating the similarity score between samples and applying a variant contrastive loss function. In this task, the representations in each batch are divided into multiple subsets according to the flight behavior parameters. For each subset, the representations in the subset serve as positive samples, while the representations in the other subset serve as negative samples:

[0066]

[0067] Where N is the number of samples in the batch, x i is the anchor sample, is a positive sample, is the jth sample of the negative sample, and f(x,y) is the mapping function representing the learning model, which projects samples x and y into the latent space.

[0068] The beneficial effects of the present invention are as follows:

[0069] The present invention fully captures signals of different modes during the use of a flight simulator, and integrates the operational behavior characteristics in pilot training videos with the data timing characteristics generated by the flight simulator, thereby achieving more accurate and efficient characterization and identification of pilot driving training behaviors, and further distinguishing between good and bad pilot training. At the same time, the entire learning process is processed in layers, and the characteristics of the main different stages of the model at each layer are mainly analyzed. Through the interaction of this mode, the information of different signals can be combined more deeply and effectively, thereby obtaining more advanced and abstract feature representations in subsequent layer learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a schematic diagram of the model framework of the present invention;

[0071] Figure 2 This is a flow chart of the feature fusion model of the present invention;

[0072] Figure 3 This is the flight parameter explanation table of the present invention;

[0073] Figure 4 Schematic diagram of data selection network processing of the present invention;

[0074] Figure 5 This is a schematic diagram of the video frame to image frame conversion operation of the present invention;

[0075] Figure 6 This is the ViTPose architecture diagram of the present invention;

[0076] Figure 7 This is a structural diagram of the CA module of the present invention;

[0077] Figure 8 This is the structural diagram of the Swin Transformer of the present invention:

[0078] Figure 9 Schematic diagram of two continuous Swin Transforme of the present invention:

[0079] Figure 10 Schematic diagram of the efficient batch processing calculation method for self-attention in shift window partitioning of the present invention;

[0080] Figure 11 This is the architecture diagram of the CAP of the present invention;

[0081] Figure 12 Schematic diagram of the self-attention mechanism and cross-attention mechanism of the present invention;

[0082] Figure 13 This is the effect diagram of the multi-modal flight parameter fusion model of the present invention. DETAILED DESCRIPTION

[0083] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0084] like Figures 1 to 13 As shown, an embodiment of the present invention provides a method for constructing a feature fusion model based on multimodal data of a flight simulator, and the specific steps are as follows:

[0085] Step 1: Select pilot training videos, obtain flight parameter data, and build a multimodal dataset for the flight simulator

[0086] The data of the flight simulator pilot training process is collected, including multiple and different pilot training videos and flight simulator flight parameter data. The flight training data involves two modal signals: the flight parameter time series signal (t) and the pilot driving behavior video frame signal (f). The data set D is defined, including N time series data {x1, x2, x3, ..., x n}, for each sequence data All include their associated time series features t i and video frame features f i , where x i ∈{t i ,f i}、l m is the sequence length, d m is the representation vector dimension of signal m;

[0087] Step 2: Data preprocessing

[0088] The CSV file of flight parameter data is input into the data selection network at a sampling frequency of 6 times per second. A random number retention method is used to randomly retain data within one second, and the columns containing the required flight parameters are retained. The retained flight parameters are then re-counted. Pilot training videos are input into the video-photo network, and the slow-fast network and key frame extraction technology are used to convert the pilot training videos into image sequences.

[0089] Step 3: ViTPose provides key point sequence

[0090] Based on pilot training videos, the ViTPose key point detection algorithm is used to extract the pilot's action key point sequence;

[0091] Step 4: Key point sequence feature embedding

[0092] Design a CA module that generates the attention map of the current layer by introducing the attention map of the previous layer as prior knowledge and uses a 1x1 convolution kernel to obtain more local attention maps;

[0093] Step 5: Extracting frequency domain features of flight parameter time series and pilot operation behavior

[0094] Two Transformer layers are introduced to unify the features of each signal. A basic Transformer network is used to extract the time series features of flight parameters, and a Swin Transformer network is used to extract the frequency domain features of pilot operation behavior.

[0095] Step 6: Cross-modal feature fusion

[0096] Perform cross-modal feature fusion of flight parameter time series features and pilot operation behavior frequency domain features, including CAP, Attention architecture, and global-local interactive learning mode to improve parameter efficiency;

[0097] Step 7: Contrastive Learning

[0098] The CPC unsupervised learning algorithm is used to capture the potential features of the data by maximizing the interactive information of the data and improve the quality of feature expression.

[0099] By fully capturing the signals of different modes during the use of the flight simulator and fusing the information between them, a more accurate and efficient characterization and identification of pilot training behavior can be achieved, thereby distinguishing the quality of pilot training.

[0100] The flight parameter data in step one include flight altitude, flight simulator attitude data, flight simulator motion data, and pilot operation action data.

[0101] Flight altitude refers to the height of the aircraft above the ground; flight simulator attitude data includes roll Euler angle, heading Euler angle and pitch Euler angle; flight simulator motion data includes calibrated airspeed, aircraft ground speed, aircraft pitch rate, aircraft yaw rate and aircraft roll rate; pilot operation action data includes calibrated airspeed, speed brake lever position and rudder pedal position, gear lever position, such as Figure 3 The flight simulator can record the flight data of the tasks in the simulated flight mission. After the mission is completed, the recorded flight parameters are exported in CSV format through the program. Each data record is arranged in the form of row data.

[0102] The specific method of providing key point sequence by ViTPose in step 3 is: given an instance image X∈R of a pilot H×W×3 As input, the image is first embedded into tokens through the patch embedding layer, i.e. Where d is the downsampling rate of the patch embedding layer, C is the channel dimension, and the embedded token is then processed by the Transformer layer. Each layer consists of a multi-head self-attention (MHSA) layer and a feed-forward network (FFN), that is:

[0103] F i+1 '=F i +MHSA(LN(F i )), F i+1 =F i+1 '+FFN(LN(F i+1 '))

[0104] Where i represents the output of the i-th Transformer layer, the initial feature F0 = PatchEmbed(X) represents the feature after the block embedding layer, and the output feature of the backbone network is expressed as

[0105] like Figure 6 As shown, for each Transformer layer, the spatial and channel dimensions are constant.

[0106] The CA module in step 4 is defined as:

[0107]

[0108] Where M i is the CA module of the i-th CA block, which is expected to focus on the muscle movement area, Attn i is the attention graph of the (i-1)th layer, represents the features extracted by the first two convolutional layers of the i-th layer as the input of the CA module, and Describes the characteristics of maximum pooling and average pooling respectively, P M represents the maximum set operation on the attention map of the (i-1)th layer to match the size of the attention map of the current layer, and N represents the product of the elements introduced as prior knowledge of the attention map of the previous layer, F i ∈R 2C×2H×2W represents the input of the i-th CA block. By associating the attention mechanism between adjacent layers, the CA block can gradually and robustly focus on areas with subtle motions, rather than focusing on different areas of the face in different layers;

[0109] Four CA blocks are used to form the main branch of MFNet to learn subtle muscle movement pattern features. Its size is 512×14×14. The CA module composed of a CA module and two convolutional layers can be expressed as:

[0110]

[0111] Where CA represents the proposed continuous attention block, stands for element-wise multiplication, which means Each channel of will be multiplied by the spatial attention map to pay attention to the region of interest, so the difference between the vertex frame and the starting frame is taken as the input of the main branch to learn motion pattern features.

[0112] The spatial attention map is calculated using the maximum pooling output and the average pooling output. Through this operation, on the one hand, it can include the prior information between different layers, that is, the knowledge of the supervisory signal; on the other hand, it can include the common knowledge in the entire task, thereby providing more robust features.

[0113] Among them, when introducing two Transformer layers in step 5 to unify the features of each signal, a low-dimensional token is randomly initialized for each signal. And use Transformer to embed basic modality information:

[0114]

[0115] Where, is the feature learned for each signal, θ m Represents the parameter information of the current Transformer learning, Concat(·) represents the splicing operation, T represents the length, and d represents the dimension.

[0116] Due to the influence of the self-attention mechanism, T tokens can be used to condense and integrate different signal features. This operation allows the model to compress information from each signal and share only necessary information, maintaining complementary fusion performance while reducing computational complexity. In addition, it is worth noting that transferring the time domain information of basic flight parameters and the frequency domain information of pilot operation behavior to initialized low-dimensional tokens helps reduce redundant information irrelevant to the original information, thereby achieving higher efficiency with fewer parameters.

[0117] Among them, the method of extracting the frequency domain features of pilot operation behavior using the Swin Transformer network in step 5 is as follows: first, the input RGB image is segmented into non-overlapping patches through the patch segmentation module. Each patch is regarded as a "token" and its feature is set as the concatenation of the original pixel RGB value. A linear embedding layer is applied to this original value feature to project it to an arbitrary dimension (denoted as C); a Swin Transformer block is applied to the patch token to maintain the number of tokens. and together with the linear embedding is referred to as “stage1”;

[0118] In order to produce a hierarchical representation, as the network goes deeper, the number of tokens is reduced by patch merging layers. The first patch merging layer connects the features of each group of 2×2 adjacent patches and applies a linear layer on the features of the 4C-dimensional connection, thereby reducing the number of tokens by a factor of 2×2=4 (equivalent to a 2x downsampling of the resolution), and the output dimension is set to 2C. After that, the Swin Transformer block is applied for feature transformation, and the resolution is kept at The first block of block merging and feature transformation is denoted as “stage 2”;

[0119] The process is repeated twice, and the output resolutions are and These stages jointly produce a hierarchical representation, where W-MSA and SW-MSA represent window-based multi-head self-attention using regular and shifted window partitioning configurations, respectively;

[0120] Self-attention in non-overlapping windows,The windows divide the image uniformly in a non-overlapping manner. Assume that each window contains M×M blocks. The computational complexity of the global MSA module and the window-based MSA module (W-MSA) is on the image of h×ω blocks:

[0121] Ω(MSA)=4hωC 2 +2(hω) 2 C

[0122] Ω(W-MSA)=4hωC 2 +2M 2 hωC

[0123] Where the former is quadratic with the number of patches h, while the latter is linear when M is fixed. Global self-attention computation is usually unaffordable for large h, while window-based self-attention is scalable.

[0124] This layered Transformer has the flexibility to model at various scales and has linear computational complexity relative to the image size. In addition, its representation is calculated using a shifted window. The shifted window scheme brings higher efficiency by limiting the calculation of the self-attention mechanism to non-overlapping local windows while also allowing cross-window connections; Figure 10 As shown in Figure 2, the Swin Transformer module is constructed by replacing the standard multi-head self-attention module in the Transformer module with a shift window-based module. The Swin Transformer block includes a shift window-based MSA module, a Layer Norm (LN) layer is applied before each MSA module and each MLP, and a residual connection is applied after each module.

[0125] Among them, the shifted window partitioning method is used to maintain efficient computation of non-overlapping windows while introducing cross-window connections. This method alternates between two partitioning configurations in consecutive Swin Transformer blocks:

[0126]

[0127] Where, and Z l Represent the output features of the (S)W-MSA module and the MLP module of block 1, respectively.

[0128] like Figure 10 As shown in the figure, the problem with shifting window division is that it will lead to more windows, so SwinTransformer adopts a more efficient batch calculation method by cyclically shifting in the upper left direction. After this shift, the batch window may be composed of several non-adjacent sub-windows in the feature map, so a masked mechanism is used to limit the self-attention calculation to each sub-window. By cyclic shifting, the number of batch windows remains the same as the number of regular window divisions, thus maintaining efficient calculation.

[0129] Among them, CAP in step 6 uses symmetric Cross-Attention to explore the intrinsic correlation between the two input feature sequences. In order to allow further information integration, self-attention is used to simulate the temporal dependency in each feature sequence. CAP is based on two sequences H t and H f As input, and output their mutual promotion information H t→f and H f→t , specifically, The calculation of is as follows:

[0130] H′ t→f =MCA(LN(H t ),LN(H f ))+H t

[0131] H″ t→f =MSA(LN(H′ t→f )))+H′ t→f

[0132] H t→f =FFN(LN(H″) t→f )))+H″ t→f

[0133] Where LN represents layer normalization, FFN is the feedforward neural network in Transformer;

[0134] The Attention architecture in step 6 includes a self-attention mechanism, which is used to calculate the attention weights between elements in a sequence in order to capture the dependencies between elements. The multi-head self-attention mechanism uses the same calculation method on multiple heads and gives different parameters to express features from multiple subspaces, which can capture richer feature information. Taking the time domain signal as an example, the input of the multi-head self-attention mechanism is defined as Among them, d t Represents the encoding dimension of the time series signal. The whole process can be expressed as:

[0135] Q i =H t W q ∈R n×d ,K i =H t W k ∈R n×d ,V i =H t W v ∈R n×d

[0136]

[0137] MSA(H t )=concat(head1,head2…,head n )W o

[0138] Where Q i , K i 、V i They represent the results of the linear transformation of the input vector by the i-th head, W q 、W k 、W v They are the weight parameters of Query, Key, and Value mapping, mapping the input to the d-dimensional output. concat represents the concatenation operation. o is the weight matrix of the final linear transformation,

[0139] It also includes a cross-attention mechanism, which is used to handle the dependency between two different sequences. By learning the directional pairwise attention between the source modality and the target modality, the source modality information is used to strengthen the target modality H. t , i.e. the query comes from the target modality, while the keys and values ​​come from the source modality H v :

[0140] Q j =H t W q ,K j =H f Wk ,V j =H f W f

[0141] Provide interaction from the modal in this way:

[0142]

[0143] MSA(H t ,H f )=concat(head1,head2…,head n )W o

[0144] Where H t 、H f Represents different signal features and is the calculation result of the multi-head cross attention mechanism;

[0145] The global-local interactive learning model in step 6 is as follows: setting global fusion context information Where i represents the number of global-local interaction layers. The entire interaction process is as follows:

[0146]

[0147] In this way, one-to-many global-local interactions can be captured in two (memory protection units). By stacking multiple layers, the global fusion context and local signal features can promote each other and gradually refine themselves. This operation requires M MPUs in each layer. Since the global fusion context length is small, the overall time complexity is reduced to (In fact, there is M<<T m ), which degenerates to O(MT 2 ), therefore, the default global-local fusion strategy in MPU not only has linear spatial complexity, but also enjoys linear computation on the involved signals,

[0148] At the same time, each global-local fusion will pass through the pooling layer to aggregate the enhanced information of different modalities to promote subsequent fusion. Therefore, a fully connected layer based on nonlinear activation is used to implement this operation, and two enhanced global fusion contexts are defined, namely The new global fusion context is obtained as follows:

[0149]

[0150] Where,

[0151] like Figure 11As shown, CAP enables the exchange of beneficial information between the two sequences, so that they can promote each other. Similarly, we can get Considering MCA(H t ,H f ) has a computational time complexity of O(T t T f ), while MSA(H t ) has a complexity of O(T t 2 ), the total time complexity of a CAP is O(T t T f +T t 2 +T t T f +T f 2 )=O((T t +T f )2), where T t Respectively represent the length of the time domain signal, T f Represents the sequence length of the frequency domain signal; Figure 12 The left picture is a schematic diagram of the self-attention mechanism. Figure 12 The right picture is a schematic diagram of the cross attention mechanism.

[0152] Among them, the CPC unsupervised learning algorithm in step seven includes contrastive learning loss, which learns useful representations of data by maximizing the similarity between positive samples and minimizing the similarity between negative samples. Specifically, given an anchor sample, the goal is to make it more similar to the positive sample and less similar to the negative sample. This can be achieved by calculating the similarity score between samples and applying a variant of the contrastive loss function. In this task, the representations in each batch are divided into multiple subsets according to the flight behavior parameters. For each subset, the representation in the subset acts as a positive sample, and the representation in the other subset acts as a negative sample:

[0153]

[0154] Where N is the number of samples in the batch, x i is the anchor sample, is a positive sample, is the jth sample of the negative sample, and f(x,y) is the mapping function representing the learning model, which projects samples x and y into the latent space.

[0155] In supervised contrastive learning, the model learns representations by comparing a pair of samples (an anchor sample and a positive and negative sample).

[0156] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0157] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a feature fusion model based on multimodal data of a flight simulator, characterized in that: The specific steps are as follows: Step 1: Select pilot training videos, obtain flight parameter data, and build a multimodal dataset for the flight simulator The data of the flight simulator pilot training process is collected, including multiple and different pilot training videos and flight simulator flight parameter data. The flight training data involves two modal signals: the flight parameter time series signal (t) and the pilot driving behavior video frame signal (f). The multimodal dataset D is defined, including N time series data {x1, x2, x3, ..., x n }, for each sequence data All include their associated time series features t i and video frame features f i , where x i ∈{t i ,f i }、l m is the sequence length, d m is the representation vector dimension of signal m; Step 2: Data preprocessing The CSV file of flight parameter data is input into the data selection network at a sampling frequency of 6 times per second. A random number retention method is used to randomly retain data within one second, and the columns containing the required flight parameters are retained. The retained flight parameters are then re-counted. The flight parameter data includes flight altitude, flight simulator attitude data, flight simulator motion data, and pilot operation data. For pilot training videos, the slow-fast network and key frame extraction technology are used to convert them into image sequences. Step 3: ViTPose provides key point sequence Based on pilot training videos, the ViTPose key point detection algorithm is used to extract the pilot's action key point sequence; Step 4: Key point sequence feature embedding Design a CA module that generates the attention map of the current layer by introducing the attention map of the previous layer as prior knowledge and uses a 1x1 convolution kernel to obtain more local attention maps; Step 5: Extracting frequency domain features of flight parameter time series and pilot operation behavior Two Transformer layers are introduced to unify the features of each signal. A basic Transformer network is used to extract the time series features of flight parameters, and a Swin Transformer network is used to extract the frequency domain features of pilot operation behavior. Step 6: Cross-modal feature fusion Perform cross-modal feature fusion of flight parameter time series features and pilot operation behavior frequency domain features, including CAP, Attention architecture, and global-local interactive learning mode to improve parameter efficiency; Step 7: Contrastive Learning The CPC unsupervised learning algorithm is used to capture the potential features of the data by maximizing the interactive information of the data and improve the quality of feature expression.

2. The method for constructing a feature fusion model based on multimodal data of a flight simulator according to claim 1, wherein: The CA module described in step 4 is defined as: Where M i is the CA module of the i-th CA block, which is expected to focus on the muscle movement area, Attn i is the attention graph of the (i-1)th layer, represents the features extracted by the first two convolutional layers of the i-th layer as the input of the CA module, and Describes the characteristics of maximum pooling and average pooling respectively, P M represents the maximum set operation on the attention map of the (i-1)th layer to match the size of the attention map of the current layer, and N represents the product of the elements introduced as prior knowledge of the attention map of the previous layer, F i ∈R 2C×2H×2W represents the input of the i-th CA block. By associating the attention mechanism between adjacent layers, the CA block can gradually and robustly focus on areas with subtle motions, rather than focusing on different areas of the face in different layers; Four CA blocks are used to form the main branch of MFNet to learn subtle muscle movement pattern features. Its size is 512×14×14. The CA module composed of a CA module and two convolutional layers can be expressed as: Where CA represents the proposed continuous attention block, stands for element-wise multiplication, which means Each channel of will be multiplied by the spatial attention map to pay attention to the region of interest, so the difference between the vertex frame and the starting frame is taken as the input of the main branch to learn motion pattern features.

3. The method for constructing a feature fusion model based on multimodal data of a flight simulator according to claim 1, wherein: When introducing two Transformer layers to unify the features of each signal as described in step 5, a low-dimensional token is randomly initialized for each signal. And use Transformer to embed basic modality information: Where, is the feature learned for each signal, θ m Represents the parameter information of the current Transformer learning, Concat(·) represents the splicing operation, T represents the length, and d represents the dimension.

4. The method for constructing a feature fusion model based on multimodal data of a flight simulator according to claim 1, wherein: The CAP described in step 6 uses symmetric Cross-Attention to explore the intrinsic correlation between two input feature sequences. In order to allow further information integration, self-attention is used to model the temporal dependency in each feature sequence. CAP is based on two sequences H t and H f As input, and output their mutual promotion information H t→f and H f→t , specifically, The calculation of is as follows: H′ t→f =MCA(LN(H t ),LN(H f ))+H t H″ t→f =MSA(LN(H′ t→f )))+H′ t→f H t→f =FFN(LN(H″ t→f )))+H″ t→f Where LN represents layer normalization, FFN is the feedforward neural network in Transformer; The Attention architecture described in step 6 includes a self-attention mechanism, which is used to calculate the attention weights between elements in a sequence in order to capture the dependencies between elements. The multi-head self-attention mechanism uses the same calculation method for multiple heads and gives different parameters to express features from multiple subspaces, which can capture richer feature information. Taking the time domain signal as an example, the input of the multi-head self-attention mechanism is defined as Among them, d t Represents the encoding dimension of the time series signal. The whole process can be expressed as: Q i =H t W q ∈R n×d ,K i =H t W k ∈R n×d ,V i =H t W v ∈R n×d MSA(H t )=concat(head1,head2…,head n )W o Where Q i , K i 、V i They represent the results of the linear transformation of the input vector by the i-th head, W q 、W k 、W v They are the weight parameters of Query, Key, and Value mapping, mapping the input to the d-dimensional output. concat represents the concatenation operation. o is the weight matrix of the final linear transformation, It also includes a cross-attention mechanism, which is used to handle the dependency between two different sequences. By learning the directional pairwise attention between the source modality and the target modality, the source modality information is used to strengthen the target modality H. t , i.e. the query comes from the target modality, while the keys and values ​​come from the source modality H v : Q j =H t W q ,K j =H f W k ,V j =H f W f Provide interaction from the modal in this way: MSA(H t ,H f )=concat(head1,head2…,head n )W o Where H t 、H f Represents different signal features and is the calculation result of the multi-head cross attention mechanism.

5. The method for constructing a feature fusion model based on multimodal data of a flight simulator according to claim 1, wherein: The global-local interactive learning model described in step 6 is specifically as follows: setting global fusion context information Where i represents the number of global-local interaction layers. The entire interaction process is as follows: In this way, one-to-many global-local interactions can be captured in two (memory protection units). By stacking multiple layers, the global fusion context and local signal features can promote each other and gradually refine themselves. This operation requires M MPUs in each layer. Since the global fusion context length is small, the overall time complexity is reduced to (In fact, there are M< <T m ), which degenerates to O(MT 2 ), therefore, the default global-local fusion strategy in MPU not only has linear spatial complexity, but also enjoys linear computation on the involved signals, At the same time, each global-local fusion will pass through the pooling layer to aggregate the enhanced information of different modalities to promote subsequent fusion. Therefore, a fully connected layer based on nonlinear activation is used to implement this operation, and two enhanced global fusion contexts are defined, namely The new global fusion context is obtained as follows: Where,

Images