An airport scene airborne situation enhancement method based on multi-modal information fusion

Abstract

The application relates to the technical field of airborne situation awareness related to large airport surface operation, in particular to an airport surface airborne situation enhancement method based on multi-modal information fusion, which comprises the following steps: obtaining air traffic control instruction texts and aircraft taxi track images, respectively performing feature extraction on the air traffic control instruction texts and the taxi track images to obtain text features and visual features; performing multi-modal feature fusion on the text features and the visual features based on an adaptive Transformer network to obtain multi-modal features; respectively performing intention recognition and key parameter analysis on the multi-modal features to obtain aircraft behavior intention and aircraft key parameters; and taking the aircraft behavior intention and the aircraft key parameters as airborne situation auxiliary information to provide assistance for subsequent flight decision; and the application can improve the understanding ability of air traffic control instructions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of airborne situation awareness of large airport surface operation, and particularly relates to an airport surface airborne situation enhancement method based on multi-modal information fusion. BACKGROUND

[0002] With the rapid development of the aviation transportation industry, the importance of the air traffic control (ATC) system is increasingly prominent. This system mainly relies on communication between controllers and flight crews to ensure the safe and efficient operation of flights. However, as the number of flights increases and the frequency of communication increases, the workload of controllers is also increasing, which may pose a safety hazard. In the ATC system, it is crucial to accurately extract and understand information from instructions. This not only relates to the normal operation of flights, but also is a key factor in ensuring passenger safety. However, due to the high specialization of ATC instructions and the diversity of expression methods, it is extremely challenging to establish a comprehensive semantic analysis model to extract the key information intent and specific parameters (ISP) of ATC instructions. For China's aviation transportation industry, the above problems are particularly prominent. With the rapid growth of the domestic aviation market, the flight density is increasing, and the requirements for the ATC system are also increasing. Non-standard natural language expressions are still included in the communication process of ATC instructions, so the features are not easy to represent. And in practical applications, the number of samples available for each intent is often small, which makes even the most advanced models prone to overfitting. Especially in high-density environments, strengthening the understanding ability of ATC instructions has become a common problem in the industry. SUMMARY

[0003] In view of the above problems, the present application provides an airport surface airborne situation enhancement method based on multi-modal information fusion, which solves the technical problem of how to improve the understanding ability of air traffic control instructions in the prior art.

[0004] The present application provides an airport surface airborne situation enhancement method based on multi-modal information fusion, comprising the steps of: S1, obtaining air traffic control instruction text and aircraft taxi trajectory images, and extracting features from the air traffic control instruction text and the taxi trajectory images to obtain text features and visual features;

[0005] S2, based on the adaptive Transformer network, multi-modal feature fusion is performed on the text features and visual features to obtain multi-modal features;

[0006] S3, intent recognition and key parameter analysis are performed on the multi-modal features to obtain aircraft behavior intent and aircraft key parameters;

[0007] S4, the aircraft behavior intent and aircraft key parameters are used as airborne situation auxiliary information to provide assistance for subsequent flight decision-making.

[0008] Preferably, step S1 specifically includes:

[0009] Step S1-1: Obtain the instruction text transcribed from the air-to-ground communication or the instruction text sent by digital control as the air traffic control instruction text, and use the BERT model to process the air traffic control instruction text to generate text features.

[0010] Step S1-2: Match the airport surface network map with the expected aircraft taxiing trajectory to generate a taxiing trajectory image. Use the ResNet model to process the taxiing trajectory image to generate visual features.

[0011] Preferably, in step S1-1, the step of processing the air traffic control instruction text using the BERT model to generate text features specifically includes:

[0012] The given instruction text W is used to divide W into a word sequence W = {w1, w2, ..., w} according to a word segmentation method. n}, w n Let n be the nth word, and n represent the total number of words in W. After W is input into the BERT model, it is transformed to obtain the text features S, where S = {t1, t2, ..., tn}. n}, t n For w n The transformed features.

[0013] Preferably, step S1-2 specifically includes:

[0014] Step S1-2-1: Convert the airport surface road topology into a network structure to form the airport surface network map;

[0015] Step S1-2-2: By traversing the aircraft taxi trajectory points and the candidate road segments in the airport surface network map, Euclidean distance matching is performed to finally obtain the taxi trajectory image;

[0016] Step S1-2-3: Use the ResNet model to extract the visual features of the trajectory image, and use the output of the penultimate hidden layer of the ResNet model as the visual features.

[0017] Preferably, step S2 specifically includes:

[0018] Step S2-1: Combine text features and visual features to obtain combined features; determine the input of the adaptive Transformer network based on the combined features;

[0019] Step S2-2, respectively adding adaptive weights in and between the text features and visual features, calculating the multi-modal attention of the adaptive Transformer network output, obtaining the multi-modal features.

[0020] Preferably, in step S2-1, the expression of the input of the adaptive Transformer network is:

[0021]

[0022] Wherein, Q F , K F and V F are query, key and value respectively, W Q , W K and W V are query, key and value weights respectively, Q S , Q T are text component and visual component of query respectively, K S , K T are text component and visual component of key respectively, V S , V T are text component and visual component of value respectively.

[0023] Preferably, in step S2-2, the calculation expression of the multi-modal attention Attention(Q F , K F , V F ) output by the adaptive Transformer network is:

[0024]

[0025] Wherein, wherein, λ1, λ2, λ3, λ4 are adaptive weight parameters of the data attention scores within and between the modalities;

[0026] The expression of the multi-modal features is:

[0027]

[0028] Wherein, is the multi-modal feature, is the text component of the multi-modal feature, is the visual component of the multi-modal feature, is the inter-modal attention of the text, is the inter-modal attention of the visual information, is the inter-modal attention of the text to the visual, is the inter-modal attention of the visual to the text.

[0029] fl This represents the l-th representation of the multimodal features.

[0030] Preferably, step S3 specifically includes:

[0031] Step S3-1: Perform fully connected layer calculation on the first representation of the multimodal features to obtain the aircraft's behavioral intent;

[0032] Step S3-2: Perform fully connected layer calculations on the multimodal features other than the first feature to obtain key aircraft parameters.

[0033] Preferably, step S3-1 specifically includes:

[0034] Based on the first specific representation f0 of the multimodal features, the predicted intent probability value is transformed into an intent probability value through a softmax function, which serves as the aircraft's behavioral intent. The expression is as follows:

[0035] y i =softmax(W i f0+b i )

[0036] Among them, y i This represents the probability value of intent, sofmtax(·) is the activation function, and W i For intent recognition weights, b i For intent recognition bias;

[0037] Step S3-2 specifically includes:

[0038] Based on the characterization of the multimodal features f1,…,f l The key parameters of the aircraft are identified using the softmax function, and the expression is as follows:

[0039]

[0040] in, f represents the l-th key parameter. l W represents the l-th representation of the multimodal features. i For the key parameter weights, b i is the key parameter bias, and L is the total number of representations of multimodal features.

[0041] Preferably, the objective function of the adaptive Transformer network is:

[0042]

[0043] Among them, y s p(y) represents all key parameters. i ,ys x) represents the conditional probability of y i ,y s the simultaneous occurrence of the conditional probability, p(y i x) represents the conditional probability of y i the simultaneous occurrence of the conditional probability, the conditional probability of y the simultaneous occurrence of the conditional probability;

[0044] The training target of the adaptive Transformer network is to maximize the objective function, and the model is fine-tuned by minimizing the cross-entropy loss, and the training of the adaptive Transformer network is completed by iterative updating.

[0045] Compared with the prior art, the present application has at least the following beneficial effects:

[0046] (1) The present application provides a Transformer bidirectional encoder based on BERT model, which can generate text word features containing word semantics and language context. This feature not only captures the meaning of the word itself, but also considers its position in the sentence and its relationship with other words. At the same time, the pre-trained Resnet network is used to capture the fine-grained features of the image. These two features are combined as input into the adaptive Transformer network, and the relationship between elements within the sequence is considered during processing. It can effectively capture the semantics of the text and the fine-grained features of the image, and realize the effective fusion of multi-modal information.

[0047] (2) The present application constructs an adaptive Transformer network by introducing adaptive weight parameters into the attention score within and between modalities, dynamically calculates the attention score within and between modalities, enhances the similarity representation between features, and can improve the integration effect of multi-modal features.

[0048] (3) The present application can improve the accuracy of semantic understanding and fine-grained analysis ability by taking intent recognition and key parameter extraction as two related subtasks and utilizing their strong correlation. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings are for the purpose of illustrating specific embodiments and are not to be considered limiting of the application.

[0050] Figure 1 The multi-modal feature extraction schematic diagram of the air traffic control instruction text and the aircraft trajectory image provided by the present application.

[0051] Figure 2 The adaptive transformer network structure diagram provided by the present application.

[0052] Figure 3 A flow chart of a multi-modal information fusion airport scene airborne situation enhancement method is provided.

[0053] Figure 4 A flow chart of a multi-modal information fusion airport scene airborne situation enhancement method is provided. DETAILED DESCRIPTION

[0054] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. In addition, the present application can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0055] The purpose of the present application is to adopt the solution idea of "feature extraction, intent recognition, and perception enhancement" around the airborne situation enhancement problem, to provide a multi-modal information fusion airport scene airborne situation enhancement method, which can explore and capture the internal relationship between fine-grained visual features and text features, filter out noise or redundant information in multi-modal data, and enhance modeling capability. Secondly, the multi-head self-attention mechanism is integrated to increase the information weight that contributes to the result. At the same time, the understanding of ATC instructions is effectively realized through the joint model of intent recognition and key parameter filling. The method provided by the present application can be applied to various running scenarios, and the intent such as turning intent, reporting intent or take-off intent can be analyzed through trajectory and instruction, and parameter information such as flight height and flight air pressure can be obtained, which provides a reference for subsequent accurate guidance of trajectory.

[0056] In order to illustrate the effectiveness of the method provided by the present application, the above technical solutions of the present application will be described in detail below through a specific embodiment. As shown in Figure 3 、 Figure 4 A multi-modal information fusion airport scene airborne situation enhancement method is disclosed, and the specific implementation steps are as follows:

[0057] Step S1, obtaining air traffic control instruction text and aircraft taxiway trajectory images, respectively extracting features from the air traffic control instruction text and the taxiway trajectory images to obtain text features and visual features.

[0058] As shown in Figure 1 , the airborne end obtains more accurate situation awareness information by integrating various modal data. Before fusing multi-modal information, the present application collects and extracts features from multi-modal information such as air traffic control instruction text and taxiway trajectory images.

[0059] (1) For air traffic control instruction text, this invention first acquires the instruction text transcribed from ground-to-air communication or the instruction text sent by digital control, and uses it as input. The BERT model is then used to generate features containing the semantics of words in the instruction and the linguistic context. Each word is represented by a fixed-dimensional vector using token embedding. For words with similar meanings, the vector representations output by the BERT model will be more similar, making it easier to distinguish the meanings of words.

[0060] In addition to word embeddings, the BERT model includes two other components: 1) Fragment embeddings: vector representations used to distinguish different sentences. The BERT model can process two sentences simultaneously, usually referred to as sentence A and sentence B. To distinguish these two sentences, each sentence is assigned a specific fragment embedding, denoted as a 768-dimensional vector. This helps the model understand the relationships and context between sentences; 2) Position embeddings: used to represent the position of words in a sentence. In the BERT model, the position of each word is assigned a corresponding 768-dimensional vector. This helps the model understand the order and positional relationships of words in a sentence.

[0061] When a command text W is passed in, W is modeled into a word sequence W = {w1, w2, ..., w} using the word segmentation method. n}, w n For the nth word, where n represents the number of words in the text, the sequence is input into the BERT network and transformed, with its features represented as S = {t1, t2, ..., tn}. n}, where t n Corresponding to w n The transformed features. The text features S are obtained by the pre-trained model BERT, and the expression is:

[0062] S={t1,t2,…,t n}=BERT(W) (1)

[0063] (2) For taxiing trajectory images, the present invention generates trajectory images by matching the map scene with the expected aircraft taxiing trajectory, which intuitively shows the movement path of the aircraft in the airport scene; and extracts features from the generated trajectory images to obtain visual features.

[0064] To this end, the position distribution of landmark objects on the airport surface is needed to help build a semantic map containing landmark positions and trajectory information; through road network modeling, road segment connection, trajectory densification, and map matching, high-precision maps are matched with expected taxiing trajectories. In this way, not only the taxiing of a single aircraft on the airport surface is shown, but also the movement direction of the aircraft at the taxiway intersection; as the aircraft moves on the airport surface, the input information is constantly updated according to the position and orientation of the aircraft in the map. The taxiing state of the aircraft is ensured to be tracked in real time, and the latest information is provided for the flight crew and the surface control personnel.

[0065] In a specific implementation, first, semantic conversion and matching of the surface trajectory are performed, in the following specific manner:

[0066] First, an airport surface network map is built. The airport surface network is represented by a directed graph G(V, E), where V is a set of vertices representing taxiway intersections or turning points, and E is a set of directed edges representing road segments of the airport ground network. Road network construction can be divided into three main steps: 1) data collection: using satellite images and high-precision map data of the airport surface, detailed information such as the position, shape, and length of the apron and taxiway is collected; 2) road topology generation: based on the map shape, the road segments are segmented at the connection points to generate the road topology structure. The road topology represents the connectivity between intersections, nodes, and road segments; 3) network formation: the road topology structure is converted into a network structure, where the intersections and nodes become the nodes of the graph, and the road segments become the edges of the graph, forming the road network.

[0067] Second, road network map matching of the surface trajectory is performed. The map matching module combines geometric matching algorithms and topological relationship algorithms. Before map matching, the trajectory points need to be preprocessed. Specifically, airport contour coordinates are used to remove all trajectory points outside the airport range. The purpose of this is to avoid interference of the trajectory points within the airport range on the subsequent map matching process. For the preprocessed trajectory point sequence (P1, P2, …, P m ), a point-to-line matching method is used to match the initial trajectory point P1 with the starting road segment. All road segments in the road network are used as candidate road segments, and the Euclidean distance between the first trajectory point and all road segments is calculated to obtain the projection distance. After obtaining the projection distance, based on the maximum error distance d MAX All possible candidate segments are filtered and sorted. The purpose of this step is to select the most likely matching road segment from all candidate road segments, thereby improving the accuracy of map matching. Based on the above process, a candidate list L1 can be obtained, and the road segments in this list are all possible matching road segments for the trajectory point sequence. The formula of the Euclidean distance D is as follows:

[0068]

[0069] where i is the index of the trajectory point, m is the total number of trajectory points, x i , y i are the horizontal and vertical coordinates of the i-th trajectory point, respectively.

[0070] By traversing the candidate segments, the topological information of the current segment is considered to obtain a candidate list L2. Under the distance error constraint, the projection distance between the next trajectory point P2 and all candidate segments is calculated. This process is repeated, and according to the topological information of the road network, the point-to-line matching method is used to continuously match the road segments until the last trajectory point is reached. Finally, complete map matching of the trajectory is completed, and the trajectory image is obtained.

[0071] After completing the trajectory matching, a ResNet model is used to extract the visual features of the trajectory image, which are the outputs of the penultimate hidden layer of the ResNet. Given a trajectory image V, the extracted visual features are T, and the expression is:

[0072] T={v1,v2,…,v s}=ResNet(V) (3)

[0073] where v s represents the s-th component of the visual features T.

[0074] Step S2, based on the adaptive Transformer network, the text features and visual features are fused to obtain multi-modal features.

[0075] In a complex scene environment, the diversity of information and the characteristics of rapid changes in space and time make the anti-interference of some information obtained by a single domain not strong, which cannot effectively meet the requirements of environmental perception. For this problem, the present application provides a method for multi-modal feature fusion based on an adaptive Transformer network, which combines the expected trajectory and real-time instructions to perform feature fusion through intra-modal and inter-modal structures, and can obtain the connection between multiple modalities to provide time and spatial dimension information for the original task. The obtained multi-modal features can be used to identify the intended action of the aircraft, and a decision reference is realized. The specific steps include:

[0076] (1) Construct an adaptive Transformer network, and for the input text features and visual features, the adaptive Transformer network is used to fuse the features of the two modalities. Specifically, first, the text features and visual features are combined:

[0077]

[0078] where F is the combined features.

[0079] The main feature of the Transformer encoder is that it uses a self-attention mechanism that allows the model to process all elements of the input sequence in parallel, rather than one element at a time, which makes the Transformer more efficient and better performing. The present invention uses a multi-head self-attention mechanism to determine the degree of association between the text and visual modalities. The combined features of the input are taken as a Query, in this case, the vector representations of Query, Key and Value all come from the same input combined feature vector. The attention mechanism triple (Q F ,K F ,V F ) are all from the mapping of the same input feature, and the input of the adaptive Transformer network is calculated as follows:

[0080]

[0081] Where Q F , K F and V F are the query, key and value, W Q , W K and W V are the weights of the query, key and value, Q S , Q T are the text component and visual component of the query, K S , K T are the text component and visual component of the key, V S , V T are the text component and visual component of the value.

[0082] Although the existing Transformer network can process the fused multi-modal features, it still has limitations in some aspects. In particular, when dealing with the relationship between text features and visual features, it may not be able to fully express the attention similarity between the two. In order to better measure the attention similarity between text and vision, the present invention provides an adaptive Transformer network constructed by adding adaptive weight parameters in the attention scores within and between modalities, which represents the similarity between features by dynamically calculating the attention scores between multi-modal features. The multi-modal attention can be defined as:

[0083]

[0084] Wherein, λ1, λ2, λ3, λ4 are adaptive weight parameters of the data attention scores within and between modalities, λ1+λ2=1, λ3+λ4=1. After the adaptive Transformer network, the fused multi-modal features are obtained as:

[0085]

[0086] in, For multimodal features, For text components of multimodal features, For the visual components of multimodal features, It is inter-modal attention of text. It is intermodal attention of visual information. It is the intermodal attention of text to vision. It is visual intermodal attention to text.

[0087] Multimodal features can be represented as:

[0088]

[0089] Among them, f l This represents the l-th representation of the multimodal features.

[0090] During aircraft operation, the environment and the information received are constantly changing. Therefore, it is necessary to fuse effective air traffic control instructions and trajectory data to obtain a more comprehensive fused feature representation of the aircraft. This representation not only includes the content of air traffic control instructions but also considers the dynamic changes in trajectory data. Then, using the fused features, the aircraft autonomously perceives its environment. Based on this environmental perception, the aircraft obtains real-time guidance for its next move. This helps the aircraft predict and plan its subsequent actions, ensuring the safety and efficiency of its operations.

[0091] like Figure 2 As shown, in the Transformer network, the input and output of a module are directly added together to form the final output; this connection method is called residual connection. Unlike other normalization techniques (e.g., batch normalization), layer normalization normalizes all features for each sample, rather than normalizing the same features across a batch, by using layer normalization. Next, the output feature vector is processed through a feedforward neural network, which can handle complex nonlinear relationships. Residual connections and layer normalization are then performed again to maintain the same length as the original vector after transformation. This helps prevent the vanishing gradient problem and improves the training stability of the model, resulting in multimodal features from the adaptive Transformer network output.

[0092] Step S3: Perform intent recognition and key parameter analysis on the multimodal features respectively to obtain the aircraft's behavioral intent and key parameters;

[0093] (1) Intention recognition. Under the requirement of precision guidance based on the track, in order to realize the real-time and complete output of the guidance information, the multi-modal fused features are matched with the actual control intention, and the space-time parameter information is obtained, which provides information for the subsequent precise guidance of the track. By inputting the fused multi-modal features into the full connection layer and the softmax function, the accurate intention recognition result is obtained.

[0094] In some embodiments, the aircraft behavior intention can include push-out permission intention, turning intention, taxiing intention, take-off intention, etc. The aircraft is prompted in real time at the position in each state, which can assist the flight crew to operate the correct action.

[0095] In this step, the multi-modal features are fed into a full connection layer for classification, and the output is the probability of each intention category. According to the first special representation f0 of the multi-modal features, the predicted intention probability value is converted into a non-negative probability distribution through the softmax function, which corresponds to the executed intention action, and the expression is:

[0096] y i = softmax(W i f0+b i ) (9)

[0097] where y i represents the output intention probability value, softmax(·) is an activation function, W i is an intention recognition weight, and b i is an intention recognition bias.

[0098] In some embodiments, the behavior intention (such as turning intention, taxiing intention, etc.) obtained by intention recognition can be output on the onboard IPAD for display to the flight crew as an auxiliary module to better understand the action instructions and perceive the scene state.

[0099] (2) Key parameter analysis. The current key parameters can be analyzed from the language and visual information of the instruction text combined with the track image. The instruction text modality contains space-time running key parameters such as push-out start time, taxiing path and runway information, and the expected track modality can provide spatial direction and path guidance. After fusing the two information modalities, the action is decided through the key parameters.

[0100] In this step, the remaining representations f1,…,f l are input into the softmax layer to identify the key parameters, and the expression is:

[0101]

[0102] where, represents the lth key parameter, fl the lth representation of the multi-modal feature, W i is a key parameter weight, b i is a key parameter bias, and L is the total number of representations of the multi-modal feature.

[0103] Finally, the objective function of the entire adaptive Transformer network is determined as:

[0104]

[0105] where y s represents all key parameters, p(y i , y s |x) represents the conditional probability of y i , y s occurring simultaneously, p(y i |x) represents the conditional probability of y i occurring given the multi-modal feature x, represents the conditional probability of y occurring given the multi-modal feature x.

[0106] The training objective of the adaptive Transformer network is to maximize the objective function, fine-tune the model by minimizing the cross-entropy loss, complete the training of the adaptive Transformer network through iterative updates, and finally output the key parameter information most likely carried by the air traffic control instruction.

[0107] Step S4, the aircraft behavior intention and aircraft key parameters are taken as airborne situation auxiliary information to provide assistance for subsequent flight decision.

[0108] By combining real-time conditions on the expected taxiway, the current intention of the aircraft and the aircraft key parameters can be obtained. The intention of the scene operation can include: push-out permission intention, turning intention, taxiing intention, take-off intention, etc. The aircraft key parameters can include information such as the moving direction, speed and expected taxiway of the aircraft. Taking the aircraft behavior intention and aircraft key parameters as airborne situation auxiliary information can be provided to the flight crew as an auxiliary module to better understand the action instructions, perceive the scene state, and can control the key nodes of the scene operation in time and space, so as to more accurately plan and adjust the operation of the aircraft to adapt to the changing airport environment.

[0109] The detailed description of the application describes and illustrates with reference to certain specific embodiments. However, the description and illustrations are intended to be merely illustrative and not restrictive of the application. While the application has been described and illustrated with reference to specific embodiments, it will be recognized that variations and modifications can be made by persons skilled in the art depending upon the overall teachings of the present application. In particular, those skilled in the art will recognize that elements of the present application can readily be combined to provide further embodiments of the present application. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. It is therefore to be understood that the scope of the application should be determined with reference to the claims.

[0110] The detailed description of the application describes and illustrates with reference to certain specific embodiments. However, the description and illustrations are intended to be merely illustrative and not restrictive of the application. While the application has been described and illustrated with reference to specific embodiments, it will be recognized that variations and modifications can be made by persons skilled in the art depending upon the overall teachings of the present application. In particular, those skilled in the art will recognize that elements of the present application can readily be combined to provide further embodiments of the present application. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. It is therefore to be understood that the scope of the application should be determined with reference to the claims.

Claims

1. An airport surface multi-modal information fusion airborne situation awareness enhancement method, characterized in that, The method comprises the following steps: Step S1, obtaining an air traffic control instruction text and an aircraft taxiing track image, and performing feature extraction on the air traffic control instruction text and the taxiing track image respectively to obtain text features and visual features; Step S2, performing multi-modal feature fusion on the text features and the visual features based on an adaptive Transformer network to obtain multi-modal features; Step S3, performing intention recognition and key parameter analysis on the multi-modal features to obtain aircraft behavior intention and aircraft key parameters; Step S4, taking the aircraft behavior intention and the aircraft key parameters as airborne situation auxiliary information to provide assistance for subsequent flight decision; An expression of an input of the adaptive Transformer network is: wherein, , and are respectively a query, a key and a value, , and are respectively a weight of a query, a key and a value, , are respectively a textual component and a visual component of a query, , are respectively a textual component and a visual component of a key, , are respectively a textual component and a visual component of a value; Multimodal attention output by the adaptive transformer network The computational expression is: wherein, wherein, is an adaptive weight parameter for the intra- and inter-modal data attention scores; An expression of the multi-modal features is: in, For multimodal features, For text components of multimodal features, For the visual components of multimodal features, It is inter-modal attention of text. It is intermodal attention of visual information. It is the intermodal attention of text to vision. It is visual intermodal attention to text; The first characteristic representing a multimodal feature One representation; An objective function of the adaptive Transformer network is: wherein, denotes all key parameters, denotes the given multimodal feature under the condition that, the conditional probability that, denotes the given multimodal feature under the condition that the conditional probability that, denotes the given multimodal feature under the condition that the conditional probability that; A training target of the adaptive Transformer network is to maximize the objective function, to fine-tune the model by minimizing the cross-entropy loss, and to complete the training of the adaptive Transformer network through iterative updating.

2. The method for multi-modal information fusion based airport surface airborne situation awareness enhancement according to claim 1, characterized in that, Step S1 specifically comprises: Step S1-1, obtaining an instruction text transcribed from a land-air communication or an instruction text sent by a digitalized control as an air traffic control instruction text, and processing the air traffic control instruction text by using a BERT model to generate text features; Step S1-2, matching an airport scene network map with an expected aircraft taxiing track to generate a taxiing track image, and processing the taxiing track image by using a ResNet model to generate visual features.

3. The method for multi-modal information fusion based airport surface airborne situation awareness enhancement according to claim 2, characterized in that, In step S1-1, the processing of the air traffic control instruction text by using the BERT model to generate the text features specifically comprises: Incoming instruction text content , according to the word segmentation method is divided into a word sequence , is the first word, represents the total number of words, after inputting the BERT model, conversion is carried out, and text features , , are converted features.

4. The method for multi-modal information fusion based airport surface airborne situation awareness enhancement according to claim 3, characterized in that, Step S1-2 specifically comprises: Step S1-2-1, converting an airport scene road topology structure into a network structure to form the airport scene network map; Step S1-2-2, performing Euclidean distance matching by traversing aircraft taxiing track points and candidate road segments in the airport scene network map to finally obtain the taxiing track image; Step S1-2-3, using a ResNet model to extract visual features of the track image, and taking an output of a penultimate hidden layer of the ResNet model as the visual features.

5. The method for multi-modal information fusion based airport surface airborne situation awareness enhancement according to claim 4, characterized in that, Step S2 specifically comprises: Step S2-1, combining the text features and the visual features to obtain combined features, and determining an input of the adaptive Transformer network based on the combined features; Step S2-2, adding adaptive weights in and between attention scores of the text features and the visual features respectively, calculating multi-modal attention of an output of the adaptive Transformer network, and obtaining the multi-modal features.

6. The multi-modal information fused airport surface on-board situational awareness enhancement method of claim 5, wherein, Step S3 specifically comprises: Step S3-1, performing full connection layer calculation on a first representation of the multi-modal features to obtain aircraft behavior intention; Step S3-2, performing full connection layer calculation on other representations of the multi-modal features except the first representation to obtain aircraft key parameters.

7. The multi-modal information fused airport surface on-board situational awareness enhancement method of claim 6, wherein, Step S3-1 specifically comprises: According to the first special representation of the multi-modal feature , the predicted intention probability value is converted into an intention probability value through a softmax function as the aircraft behavior intention, and the expression is: wherein, represents an intent probability value, is an activation function, is an intent recognition weight, is an intent recognition bias; Step S3-2 specifically includes: According to the representation of the multi-modal feature , the aircraft key parameter is identified through a softmax function, and the expression is: wherein, denotes the key parameter, denotes the representation of the multi-modal feature, is the key parameter weight, is the key parameter bias, is the total number of representations of the multi-modal feature.

Images