A hand motion trajectory prediction method for flight simulation environment

Abstract

The present invention provides a hand motion trajectory prediction method for a flight simulation environment, which is used in the field of flight virtual reality simulation. The method of the present invention comprises: establishing a hand motion trajectory prediction model, comprising a feature engineering module, a feature encoding module, a global information interaction module, and a feature decoding module; the feature engineering module extracts and vectorizes flight simulation environment information, the feature encoding module performs internal feature integration on various local vector sequences, the global information interaction module determines the vector representation of each node fused with environmental information having an influence relationship, and the feature decoding module decodes the vector representation of the hand motion trajectory historical information fused with environmental information, and predicts and outputs a hand motion trajectory vector sequence. The method of the present invention achieves early prediction of the hand motion trajectory, solves the delay problem during fast hand tracking in a flight simulation environment, fully considers the relationship with flight simulation environment information, and achieves accurate prediction.

Description

Technical Field

[0001] The present invention relates to the technical fields of flight simulation, virtual reality and computers, and in particular to a hand motion trajectory prediction method for a flight simulation environment. Background Art

[0002] Pilot training and air combat simulation can be carried out in a flight simulation environment without being restricted by time, location and other conditions, thus saving costs. Especially after the introduction of virtual reality technology, the flight simulation environment with a high sense of immersion and realism has ushered in rapid development.

[0003] In a flight simulation environment, whether the pilot's hand movements can be accurately restored is very critical and directly affects the simulation effect. Accurately restoring the pilot's hand movements requires not only the correct fitting of the hand shape, but also the rapid tracking of the hand's spatial position. The challenges faced by fast hand tracking are mainly the delay problems caused by processes such as data transmission, tracking algorithms, and screen rendering. The delay problem will cause the screen rendering results to be out of sync with the actual movements. One solution is to predict the hand's motion trajectory and render the future hand motion trajectory in advance at the current moment to offset the impact of delay. The existing technology has achieved good results in the task of motion trajectory prediction in fields such as transportation based on deep learning. However, the composition, emphasis, and relationship of many information in the flight simulation environment are different from those in fields such as transportation. At present, there is no mature and available solution that can establish the relationship between the hand motion trajectory and various complex information in the flight simulation environment. Summary of the Invention

[0004] To address the above problems, the present invention proposes a hand motion trajectory prediction method for a flight simulation environment, constructs a deep learning framework suitable for the flight simulation environment, and uses historical hand motion trajectory information and flight simulation environment information to predict future hand motion trajectory information to solve the delay problem during fast hand tracking in the flight simulation environment.

[0005] The present invention provides a method for predicting hand motion trajectories in a flight simulation environment, comprising the following steps:

[0006] Step 1: Design a hand motion trajectory prediction model, which includes a feature engineering module, a feature encoding module, a global information interaction module, and a feature decoding module;

[0007] The feature engineering module extracts and vectorizes flight simulation environment information; the flight simulation environment information includes hand motion trajectory history information, cockpit information, aircraft information, airport information, route information, and weather information; the geometric information of the three-dimensional cockpit model and flight control information are extracted from the cockpit information, the static parameter information and dynamic parameter information of the aircraft are extracted from the aircraft information, and the airport runway information and tower instruction information are extracted from the airport information; the feature engineering module extracts corresponding local vector sequences from various flight simulation environment information;

[0008] The feature encoding module performs internal feature integration on the various local vector sequences output by the feature engineering module, and outputs aggregated features of each type of flight simulation environment information;

[0009] The global information interaction module corresponds each type of flight simulation environment information to a node, and the corresponding aggregated feature is a node vector. A graph attention network is constructed for each node to obtain the influence relationship between the node vectors, determine the vector representation of each node that integrates the neighborhood features, and represent the global relationship between the nodes as a graph structure and output it;

[0010] The feature decoding module adopts an attention mechanism network structure to obtain a vector representation of the hand motion trajectory history information node that integrates neighborhood features from the global graph, decodes the vector representation, and predicts and outputs the hand motion trajectory vector;

[0011] Step 2: Collect historical flight simulation environment information to generate training data sets and test data sets, train the established hand motion trajectory prediction model, and then use the trained hand motion trajectory prediction model to perform real-time prediction of the hand motion trajectory in the flight simulation environment.

[0012] The advantages and positive effects of the present invention are:

[0013] (1) The method of the present invention establishes a hand motion trajectory prediction model for a flight simulation environment. The hand motion trajectory and related environmental information are acquired through a feature engineering module. A local vectorized sequence is extracted for each type of environmental information. Each type of environmental information is internally integrated through a feature encoding module. A global information interaction module is used to establish a global graph of the mutual influence between nodes, thereby obtaining a vector representation of the hand motion trajectory that integrates various environmental information. Finally, a feature decoding module is used to decode and output the future hand motion trajectory. The present invention fully utilizes and considers the various complex information in the flight simulation environment, establishes the relationship between the hand motion trajectory and the flight simulation environmental information, and thus accurately predicts the future hand motion trajectory.

[0014] (2) The method of the present invention can be used to predict the hand movement trajectory in advance, solving the delay problem when performing fast hand tracking in a flight simulation environment. The method of the present invention can not only be used for early rendering to eliminate delays, but also can be used to predict whether the pilot's operating behavior is correct, thereby achieving training purposes, etc., and has a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a diagram showing the implementation framework of the hand motion trajectory prediction method of the present invention;

[0016] Figure 2 It is a graph structure constructed by the global information interaction module of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0018] The present invention's hand motion trajectory prediction method for flight simulation environment uses a deep learning framework to design a hand motion trajectory prediction model. Figure 1 As shown, the hand motion trajectory prediction model designed by the present invention includes a feature engineering module, a feature encoding module, a global information interaction module, and a feature decoding module. The implementation of each module in the flight simulation environment is described in detail below.

[0019] (1) The feature engineering module extracts and vectorizes the flight simulation environment information. In the embodiment of the present invention, the hand motion trajectory history information, cockpit information, aircraft information, airport information, route information, and weather information are vectorized.

[0020] Specifically, the hand motion trajectory history information is used as an example to describe the vectorized expression of the information. The vectorized expression formats of different types of information are similar, so other flight simulation environment information only describes the composition of the information without repeating its vectorized expression.

[0021] The hand motion trajectory history information is represented by abstracting the hand information collected by the sensor into a point in three-dimensional space, recording it at fixed time intervals over a period of time to form a broken line. The vector expression V formed by any two points in the broken line in time sequence is i as follows:

[0022]

[0023] in, and are the three-dimensional coordinates of the starting and ending points of the vector; and is the timestamp of the vector’s starting and ending points; s is the identifier for the left and right hands. The symbol i represents the vector number, such as Vi Represents the information of vector i.

[0024] For each hand, 40 frames of continuous data are collected, the first 20 frames are used as the historical information of the hand motion trajectory, and the last 20 frames are used as the predicted information of the hand motion trajectory.

[0025] Cockpit information includes the geometric information of the 3D cockpit model and flight control information.

[0026] The geometric information of the three-dimensional cockpit model is to regard the model contour boundary as a line and the control mechanism as a composition of multiple basic geometric bodies. The connection between the boundary line and the center axis of the geometric body is sampled at a fixed distance to form a broken line, and it is noted whether it belongs to the contour boundary or the control mechanism.

[0027] The flight control information records the hardware input information such as the gear position, pressing status, deflection value, etc. of different control mechanisms over a period of time, as well as timestamps.

[0028] Aircraft information includes static and dynamic parameter information of the aircraft.

[0029] The static parameter information of the aircraft includes the model, size, engine thrust, ceiling, maximum take-off weight, maximum range with full load, etc.

[0030] The dynamic parameter information of the aircraft is the real-time flight status information and timestamp of the aircraft's attitude angle, azimuth angle, torque, three-dimensional spatial coordinates, etc. over a period of time.

[0031] Airport information includes airport runway information and tower instruction information.

[0032] Airport runway information is represented by a polyline formed by sampling runway lines and runway markings at fixed distances, as well as attribute annotations of runway lines and markings.

[0033] The tower command information is the command information issued by the tower within a period of time, such as wait, prepare to taxi, allow takeoff, restricted landing, etc., as well as the timestamp of the command issuance.

[0034] The route information is represented by sampling the predetermined three-dimensional flight trajectory at a fixed distance to form a broken line, and the route width and height range are indicated.

[0035] The meteorological information includes real-time wind speed, wind direction, light intensity, light direction, humidity, temperature, air pressure, etc. in the flight simulation environment as well as timestamp.

[0036] (2) The feature encoding module performs internal feature integration on the various local vector sequence information established by the feature engineering module.

[0037] As described above, the feature engineering module of the embodiment of the present invention extracts 9 different local vector sequences from the flight simulation environment information, specifically: hand motion trajectory history information, three-dimensional cockpit model geometric information, flight control information, aircraft static parameter information, aircraft dynamic parameter information, airport runway information, tower instruction information, route information and weather information. Among them, the vector sequences are classified according to whether the information has time continuity or spatial position regularity. For example, the static parameter information of the aircraft is information that does not have time continuity or spatial position regularity. Internal feature integration is performed on each local vector sequence according to different types.

[0038] Specifically, the embodiment of the present invention integrates features of information that does not have temporal continuity or spatial position regularity through an MLP (Multi-layer Perceptron), and the design is as follows:

[0039] The input of the fully connected layer is the vector sequence generated by the feature engineering module, which is then subjected to LayerNorm (layer normalization) and ReLU activation function, and the output is a 200-dimensional vector sequence of the same length.

[0040] Perform feature aggregation on the output vector sequence in the following way:

[0041]

[0042] Where x is the input feature vector sequence, x i is the i-th vector in the sequence, n is the sequence length, and SumPooling represents sum pooling.

[0043] Specifically, in the embodiment of the present invention, for information with temporal continuity or spatial position regularity, the feature encoding module realizes feature integration through a network structure based on the self-attention mechanism, and the design is as follows:

[0044] Input part: designed as a combination of embedding and encoding operations.

[0045] The local vector sequence generated by the feature engineering module is used as the input of the embedding layer, and the input vector of length n is mapped to a d-dimensional embedding vector A through a d*n embedding matrix E. In this example of the present invention, the embedding matrix E is obtained by data training, and the value of d is set to 300.

[0046] For hand motion trajectory history information and aircraft dynamic parameter information with time-continuous characteristics, the position encoding method in the embodiment of the present invention is to perform it in the order of the time in the timestamp. The specific position encoding vector PE has the same dimension d as the embedding vector A, and its expression is as follows:

[0047]

[0048] The even columns of the matrix PE are a sin function, the odd columns are a cos function, p represents the current position, j identifies whether it is an odd or even column, and i is the current dimension of the input local vector.

[0049] For spatially regular geometric information such as the 3D cockpit model and airport runway information, position encoding is performed based on the topological relationships of this information in 3D space. Specifically, position encoding transforms the input local vector sequence into a multidimensional array, which is then fed into a CNN (convolutional neural network) layer. The resulting d-dimensional vector representation is then mapped through multiple layers of multi-layered multi-processor layers (MLPs).

[0050] In the input part, the local vectors are mapped by the embedding matrix E to obtain the vector A, and the vector PE is obtained by position encoding. A and PE are added and combined to obtain the output vector B of the input part.

[0051] Encoding part: In this embodiment, the encoding part consists of 6 encoders with the same structure. Each encoder contains a multi-head self-attention mechanism sublayer and a feedforward network sublayer. Each sublayer is also connected to a residual connection and a normalization (LayerNorm) operation layer.

[0052] The d-dimensional vector output by the input part is compared with the query vector matrix W obtained through data training Q , key vector matrix W K , value vector matrix W V Multiply them separately, and the results are query vector Q, key vector K, and value vector V. The expression for determining the weight distribution of V using the similarity between Q and K is as follows:

[0053]

[0054] Among them, d k represents the scale factor, and the superscript T represents the transpose.

[0055] Multiple groups of attention (attention mechanism) are mapped to dimension d after weighting, splicing, and linear transformation operations to form the final attention. Then, the fully connected layer (FC) and ReLU activation function are applied to the vector at each position for nonlinear transformation.

[0056] Furthermore, to prevent the gradient from vanishing and speed up the convergence, the residual connection between the input and output and the LayerNorm operation are performed after each sub-layer.

[0057] Feature aggregation is performed at the output of the last encoder, and the aggregation method also uses SumPooling.

[0058] (3) The global information interaction module establishes the influence relationship between each local information based on the aggregated features of each flight simulation environment information output by the feature encoding module.

[0059] Specifically, the global relationship between each local information can be represented as a graph structure, and each local information is represented as a node in the graph, such as Figure 2 As shown, in the embodiment of the present invention, there are 9 types of flight simulation environment information, that is, 9 nodes. After the feature engineering module and the feature encoding module, the aggregated features corresponding to the 9 nodes are obtained.

[0060] Similarly, the global information interaction module builds the relationship between nodes through the self-attention mechanism. In the embodiment of the present invention, a graph attention network is constructed for each node in the graph, and the design is as follows:

[0061] For the node vector h i and h j , the activation function LeakyReLU is used to calculate the correlation e ij as follows:

[0062] e ij =LeakyReLU(a[h i ||h j ])

[0063] Among them, a represents the weight parameter, h i ||h j Represents vector concatenation operation. Node vector h i and h j is the aggregated feature of the i-th and j-th flight simulation environment information output by the feature encoding module. According to the above correlation calculation method, the correlation of different node vectors is calculated.

[0064] like Figure 2 As shown, in the embodiment of the present invention, it is not necessary to calculate the correlation between all two different node vectors, and the correlation calculation between irrelevant nodes is eliminated in advance. For example, for the node of the geometric information of the three-dimensional cockpit model, it is only necessary to calculate the correlation between it and the three nodes of the hand motion trajectory history information, the flight control information, and the static parameter information of the aircraft; and for the hand motion trajectory history information, it is necessary to calculate the correlation between it and the other 8 nodes.

[0065] For the i-th node, first determine the nodes that have an influencing relationship with the node, then calculate the correlation between the node and other nodes according to the above method. After the calculation is completed, use the softmax function to normalize all the correlations of the node, and then perform weighted summation to output the vector representation of the current node feature that integrates the neighborhood features.

[0066] Suppose that all calculated correlations of the i-th node are normalized and the correlation e ij Normalized to α ij , then the output feature of the i-th node that integrates the neighborhood features is h i '=σ(∑α ij h j ), σ is the activation function.

[0067] Preferably, the present invention can also stack multiple layers of graph attention networks to enhance the ability to fit complex features. Specifically, D groups of independent processes are used to fuse the neighborhood information of each node, and then the results are spliced. The process expression is as follows:

[0068]

[0069] In the formula, || represents the splicing operation, is the normalized weight coefficient of node i and node j calculated by the kth group of attention mechanism.

[0070] (4) The feature decoding module recovers the predicted information from a certain node in the source information format. In the embodiment of the present invention, the future trajectory information is predicted from the hand motion trajectory information node, which is implemented by using the network structure of the attention mechanism.

[0071] Specifically, the feature decoding module consists of an MLP and six decoders with the same structure. In addition to the multi-head self-attention mechanism sublayer, feedforward network sublayer, residual connection and normalization operation layer, each decoder also has an attention mechanism layer that associates the global graph with the decoder.

[0072] First, the hand motion trajectory history information node vector h is obtained through MLP i ′, the length n of the vector sequence for restoring the historical information of the hand motion trajectory.

[0073] In the multi-head self-attention mechanism sublayer, an n*n matrix M is constructed, where n is the length of the input sequence. The i-th row and j-th column element of the matrix M is expressed as:

[0074]

[0075] Among them, -inf is the preset lower limit value.

[0076] Calculate QK using the same method as in the encoder using the decoder’s training data T , multiplied by the matrix M, and continue to calculate the output of the multi-head Attention using the same method as in the encoder.

[0077] The decoder has a similar structure to the encoder, but with an additional attention layer linking the global graph to each decoder. The attention layer uses the vectors K and V for attention calculation, derived from the hand trajectory information node outputs in the global graph, and the vector Q is calculated by the previous decoder.

[0078] Finally, the output of the last decoder is feature mapped through an FC layer, and the Softmax function predicts and outputs the hand motion trajectory vector one by one in chronological order.

[0079] After constructing the hand motion trajectory prediction model, historical flight simulation environment information is collected to generate training and test sets. The hand motion trajectory prediction model is then trained using the trained hand motion trajectory prediction model to perform real-time predictions of hand motion trajectories within the flight simulation environment. When using the model for real-time predictions, the collected flight simulation environment information for the current period is used to extract local vector sequences through the feature engineering module. For local vector sequences that do not change over time, the established data can be directly used. The current period's historical hand motion trajectory information is subjected to local vector sequence extraction, feature aggregation extraction, and global influence relationship extraction. The feature decoding module then predicts future hand motion trajectories from the hand motion trajectory information that has been aggregated with other environmental information, and outputs the predicted results in a vector form that can be interpreted by the feature engineering module.

[0080] Except for the technical features described in the specification, all other technical features are known to those skilled in the art. The present invention omits descriptions of well-known components and well-known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.

Claims

1. A hand motion trajectory prediction method for a flight simulation environment, characterized in that: The steps include: Step 1: Design a hand motion trajectory prediction model, which includes a feature engineering module, a feature encoding module, a global information interaction module, and a feature decoding module; The feature engineering module extracts and vectorizes flight simulation environment information; the flight simulation environment information includes hand motion trajectory history information, cockpit information, aircraft information, airport information, route information, and weather information; extracts three-dimensional cockpit model geometry information and flight control information from the cockpit information, extracts aircraft static parameter information and dynamic parameter information from the aircraft information, and extracts airport runway information and tower command information from the airport information; The feature engineering module extracts corresponding local vector sequences from various flight simulation environment information; The feature encoding module performs internal feature integration on the various local vector sequences output by the feature engineering module, and outputs aggregated features of each type of flight simulation environment information; The global information interaction module corresponds each type of flight simulation environment information to a node, and the corresponding aggregated feature is a node vector. A graph attention network is constructed for each node to obtain the influence relationship between the node vectors, determine the vector representation of each node that integrates the neighborhood features, and represent the global relationship between the nodes as a graph structure and output it; The feature decoding module adopts an attention mechanism network structure to obtain a vector representation of the hand motion trajectory history information node that integrates neighborhood features from the global graph, decodes the vector representation, and predicts and outputs the hand motion trajectory vector; Step 2: Collect historical flight simulation environment information to generate training data sets and test data sets, train the established hand motion trajectory prediction model, and then use the trained hand motion trajectory prediction model to perform real-time prediction of the hand motion trajectory in the flight simulation environment.

2. The method according to claim 1, characterized in that In step 1, the feature engineering module extracts the following information from the flight simulation environment information: From the historical information of hand motion trajectory, the hand position collected by the sensor is abstracted into a point in three-dimensional space, and the hand motion trajectory is represented by a broken line formed by recording the hand position at fixed time intervals; Extracting geometric information and flight control information from the cockpit information: The geometric information of the 3D cockpit model is represented by a polyline formed by sampling the boundary line and the central axis of the geometric body at fixed intervals, treating the model's outline as a line and the control mechanisms as a combination of geometric bodies. Each distance segment is identified as belonging to the outline boundary or the control mechanism. The flight control information includes hardware input information and timestamps for different control mechanisms within the extraction time period. Extracting static and dynamic parameter information of the aircraft from the aircraft information; wherein the static parameter information of the aircraft includes aircraft model, size information, engine thrust, ceiling, maximum takeoff weight, and maximum range with full load; the dynamic parameter information of the aircraft includes the real-time flight status information and timestamp of the aircraft during the extraction time period; the real-time flight status information includes the attitude angle, azimuth angle, torque, and spatial coordinates of the aircraft; Extract airport runway information and tower command information from the airport information; the runway information is a polyline representation formed by sampling runway lines and runway markings at fixed intervals, with the attributes of the runway lines and markings at each distance on the polyline marked; the tower command information includes the command information issued by the tower during the extraction time period and the timestamp of the command issuance; Extract the predetermined three-dimensional flight trajectory from the route information, represent it as a broken line formed by sampling at a fixed distance, and mark the route width and altitude range; Extract real-time weather data and timestamps of the flight simulation environment from meteorological information; meteorological data includes wind speed, wind direction, light intensity, light direction, humidity, temperature and air pressure.

3. The method according to claim 2, characterized in that In the step 1, the feature engineering module vectorizes the information extracted from the hand motion trajectory information, specifically: extracting the information of the vector formed by the hand at any two points in time sequence, and the information V of the extracted vector i i as follows: in, and are the three-dimensional coordinates of the starting point and end point of vector i, and are the timestamps of the starting and ending points of vector i, and s is the identifier for distinguishing the left and right hands.

4. The method according to claim 1 or 2, characterized in that In the step 1, for a local vector sequence of flight simulation environment information that does not have temporal continuity or spatial position regularity, the feature encoding module performs internal feature integration through a multi-layer perceptron (MLP), including: inputting the local vector sequence into a fully connected layer of the MLP, then performing layer normalization and a ReLU activation function, outputting a vector sequence with the same length, and then performing feature aggregation on the output vector sequence to output aggregated features.

5. The method according to claim 1 or 2, characterized in that In step 1, for a local vector sequence with temporal continuity or spatial position regularity, the feature encoding module uses a network structure based on a self-attention mechanism to perform internal feature integration, including processing of the input part and the encoding part; The input part performs an embedding operation and an encoding operation on the local vectors output by the feature engineering module, and then outputs them to the encoding part after summation and combination. Among them, the embedding operation refers to inputting the local vector sequence output by the feature engineering module into an embedding layer, and mapping the input vector of length n into a d-dimensional embedding vector through the embedding matrix. The encoding operation refers to: for the local vector sequence with time continuity, position encoding is performed in chronological order, and the local vector is encoded using sine and cosine functions of different frequencies to obtain a d-dimensional position encoding vector; for the local vector sequence with spatial position regularity, position encoding is performed according to the spatial topological relationship, and the local vector is transformed into a multidimensional array and input into a convolutional neural network layer, and then mapped by the multi-layer perceptron MLP to obtain a d-dimensional position encoding vector; d and n are both positive integers; The encoding part consists of six encoders with the same structure connected in sequence. Each encoder contains a multi-head self-attention mechanism sublayer and a feedforward network sublayer. Each sublayer is connected to a residual connection and a normalization operation layer. The d-dimensional vector sequence output by the input part is input into the first encoder, and the output of the last encoder is subjected to feature aggregation to output the aggregated features of the corresponding type of flight simulation environment information.

6. The method according to claim 1 or 2, characterized in that In the step 1, the global information interaction module first determines the nodes that have an influence relationship with the i-th node, and then calculates the node vector correlation, and the calculation method is as follows: Assume that there is an influence relationship between the i-th node and the j-th node, and the i-th node vector is h i , the jth node vector is h j , after splicing the two node vectors, use the activation function LeakyReLU to calculate h i and h j The correlation degree e ij ; After calculating the correlation between the i-th node vector and all node vectors with an influence relationship, all correlations of the i-th node vector are normalized using the softmax function, and then weighted summed up to output the vector representation of the i-th node that integrates the neighborhood features.

7. The method according to claim 1 or 2, characterized in that In the step 1, the global information interaction module stacks a multi-layer graph attention network, and assumes that the i-th node uses D multi-layer graph attention networks to independently fuse neighborhood features, and finally splices the D results to obtain the final vector representation of the i-th node that fuses the neighborhood features.

8. The method according to claim 6, characterized in that In step 1, the feature decoding module consists of a multi-layer perceptron (MLP) and six decoders with the same structure. Each decoder includes a multi-head self-attention mechanism sublayer, a feedforward network sublayer, a residual connection and normalization operation layer, and an attention mechanism layer that associates the global graph with the decoder. First, the length n of the vector sequence of the hand motion trajectory history information is recovered from the node vector of the hand motion trajectory history information through MLP; Secondly, the multi-head self-attention mechanism sub-layer constructs an n*n matrix M, where the element in the i-th row and j-th column is represented as: -inf is the preset lower limit; When calculating the weight distribution of the value vector V in the multi-head self-attention mechanism sub-layer of the decoder, the calculated QK T Multiply it by the matrix M and calculate the output of the multi-head self-attention mechanism; An attention mechanism layer associated with the global graph is set in each decoder. The key vector K and value vector V used by the attention mechanism layer are calculated based on the hand motion trajectory history information node vector output by the global graph, and the query vector Q used is calculated by the previous decoder. Finally, the output of the last decoder is feature mapped through a fully connected layer, and the Softmax function is used to output the predicted hand motion trajectory vectors one by one in time order.

9. The method according to claim 1, characterized in that In the step 2, the flight simulation environment information of the current period is collected and input into the trained hand motion trajectory prediction model. The vector representation of the hand motion trajectory information of the current period that is integrated with other environmental information is obtained through the feature engineering module, feature encoding module and global information interaction module. The feature decoding module is used to decode and output the predicted future hand motion trajectory vector sequence.

Images