Airport scene airborne situation enhancement method based on multi-modal information fusion

Abstract

Relates to the technical field of airborne situation awareness of large-scale airport scene operation, in particular to an airport scene airborne situation enhancement method based on multi-modal information fusion, which comprises the following steps: acquiring an air traffic control instruction text and a sliding track image of an aircraft, and respectively performing feature extraction on the air traffic control instruction text and the sliding track image to obtain a feature extraction result; obtaining text features and visual features; performing multi-modal feature fusion on the text features and the visual features based on an adaptive Transform network to obtain multi-modal features; performing intention recognition and key parameter analysis on the multi-modal features to obtain aircraft behavior intention and aircraft key parameters; taking the behavior intention of the aircraft and the key parameters of the aircraft as airborne situation auxiliary information, and providing assistance for subsequent flight decision making; according to the invention, the understanding capability of the air traffic control instruction can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of airborne situation awareness in large airport scene operations, and in particular to an airport scene airborne situation enhancement method using multi-modal information fusion. Background Art

[0002] With the booming development of the aviation industry, the importance of the air traffic control (ATC) system has become increasingly prominent. This system mainly relies on communication between controllers and flight crews to ensure the safety and efficient operation of flights. However, with the increase in the number of flights and the increase in the frequency of communication, the work pressure of controllers is also increasing, which may bring safety hazards. In the ATC system, it is crucial to accurately extract and understand information from instructions. This is not only related to the normal operation of flights, but also a key factor in ensuring the safety of passengers. However, due to the high specialization and diversity of expressions of ATC instructions, it has become extremely challenging to establish a comprehensive semantic analysis model to extract the key information intent and specific parameters (ISP) of ATC instructions. For my country's aviation transportation industry, the above problems are particularly prominent. With the rapid growth of the domestic aviation market and the increasing flight density, the requirements for the ATC system are also getting higher and higher. The ATC instruction control and guidance communication process still contains non-standard natural language expressions, so the features are not easy to represent. In addition, in practical applications, the number of samples available for each intent is often small, which makes even the most advanced models prone to overfitting. Especially in high-density environments, strengthening the ability to understand ATC instructions has become a common problem in the industry. Summary of the invention

[0003] In view of the above problems, the present invention provides an airport scene airborne situation enhancement method based on multi-modal information fusion, which solves the technical problem of how to improve the ability to understand air traffic control instructions in the prior art.

[0004] The present invention provides an airport scene airborne situation enhancement method with multi-modal information fusion, comprising step S1, obtaining an air traffic control instruction text and an aircraft taxiing track image, performing feature extraction on the air traffic control instruction text and the taxiing track image respectively, and obtaining text features and visual features;

[0005] Step S2: performing multimodal feature fusion on the text features and visual features based on an adaptive Transformer network to obtain multimodal features;

[0006] Step S3, performing intention recognition and key parameter analysis on the multimodal features respectively to obtain aircraft behavior intention and aircraft key parameters;

[0007] Step S4: Using the aircraft behavior intention and aircraft key parameters as airborne situation auxiliary information to provide assistance for subsequent flight decisions.

[0008] Preferably, step S1 specifically includes:

[0009] Step S1-1, obtaining the instruction text transcribed from the air-ground conversation or the instruction text sent by the digital control as the air traffic control instruction text, and using the BERT model to process the air traffic control instruction text to generate text features;

[0010] Step S1-2: Match the airport surface network map with the expected aircraft taxiing trajectory to generate a taxiing trajectory image, and use the ResNet model to process the taxiing trajectory image to generate visual features.

[0011] Preferably, in step S1-1, the process of processing the air traffic control instruction text using the BERT model to generate text features specifically includes:

[0012] Input the command text content W, and divide W into word sequences W = {w1,w2,...,w n},w n is the nth word, n represents the total number of words in W, W is input into the BERT model and converted to obtain text features S, S = {t1, t2, ..., t n}, t n w n Features after conversion.

[0013] Preferably, step S1-2 specifically includes:

[0014] Step S1-2-1, converting the airport surface road topology structure into a network structure to form the airport surface network map;

[0015] Step S1-2-2, performing Euclidean distance matching by traversing the aircraft taxiing trajectory points and the candidate road sections in the airport surface network map, and finally obtaining the taxiing trajectory image;

[0016] Step S1-2-3: Use the ResNet model to extract the visual features of the trajectory image, and use the output of the penultimate hidden layer of the ResNet model as the visual features.

[0017] Preferably, step S2 specifically includes:

[0018] Step S2-1, combining text features and visual features to obtain combined features; and determining the input of the adaptive Transformer network based on the combined features;

[0019] Step S2-2, adding adaptive weights to the attention scores within and between the text features and visual features, respectively, calculating the multimodal attention output by the adaptive Transformer network, and obtaining the multimodal features.

[0020] Preferably, in step S2-1, the input expression of the adaptive Transformer network is:

[0021]

[0022] Among them, Q F , K F and V F are query, key and value respectively, W Q , W K and W V are the query, key and value weights, Q S , Q T are the textual and visual components of the query, respectively, and K S , K T are the text component and visual component of the key, V S , V T The textual and visual components of the value, respectively.

[0023] Preferably, in step S2-2, the multimodal attention Attention (Q F ,K F ,V F ) is calculated as:

[0024]

[0025] Among them, λ1,λ2,λ3,λ4 are the adaptive weight parameters of the intra-modal and inter-modal data attention scores;

[0026] The expression of the multimodal feature is:

[0027]

[0028] in, is a multimodal feature, is the text component of the multimodal feature, is the visual component of the multimodal feature, is the inter-modal attention of the text, is the inter-modal attention of visual information, is the inter-modal attention of text to vision, is the inter-modal attention of vision to text;

[0029] fl Represents the lth representation of the multimodal feature.

[0030] Preferably, step S3 specifically includes:

[0031] Step S3-1, performing a fully connected layer calculation on the first representation of the multimodal feature to obtain the aircraft behavior intention;

[0032] Step S3-2: Perform full connection layer calculation on the representations of the multimodal features except the first representation to obtain key parameters of the aircraft.

[0033] Preferably, step S3-1 specifically includes:

[0034] According to the first special representation f0 of the multimodal feature, the predicted intention probability value is converted into an intention probability value through the softmax function as the aircraft behavior intention, which is expressed as:

[0035] y i =softmax(W i f0+b i )

[0036] Among them, y i represents the intention probability value, sofmtax(·) is the activation function, and W i is the intention recognition weight, b i bias for intent identification;

[0037] Step S3-2 specifically includes:

[0038] According to the representation of the multimodal features f1,…,f l , the key parameters of the aircraft are identified through the softmax function, and the expression is:

[0039]

[0040] in, Indicates the lth key parameter, f l represents the lth representation of the multimodal feature, W i is the key parameter weight, b i is the key parameter bias, and L is the total number of representations of multimodal features.

[0041] Preferably, the objective function of the adaptive Transformer network is:

[0042]

[0043] Among them, y s Represents all key parameters, p(y i ,ys |x) means that given the multimodal feature x, y i ,y s The conditional probability of simultaneous occurrence, p(y i |x) represents y given the multimodal feature x i The conditional probability of occurrence, Represents given multimodal feature x Conditional probability of occurrence;

[0044] The training goal of the adaptive Transformer network is to maximize the objective function, fine-tune the model by minimizing the cross entropy loss, and complete the training of the adaptive Transformer network through iterative updating.

[0045] Compared with the prior art, the present invention has at least the following beneficial effects:

[0046] (1) The present invention provides a Transformer bidirectional encoder based on the BERT model, which can generate text word features that contain word semantics and language context. This feature not only captures the meaning of the word itself, but also considers its position in the sentence and its relationship with other words. At the same time, a pre-trained Resnet network is used to capture the fine-grained features of the image. These two features are combined as input into an adaptive Transformer network, and the relationship between each element in the sequence is taken into account during the processing. It can effectively capture the semantics of the text and the fine-grained features of the image, and realize the effective fusion of multimodal information.

[0047] (2) The present invention constructs an adaptive Transformer network by introducing adaptive weight parameters into the intra-modal and inter-modal attention scores respectively, dynamically calculates the attention scores within the modality and between the modalities, enhances the similarity representation between features, and can improve the integration effect of multimodal features.

[0048] (3) The present invention regards intent recognition and key parameter extraction as two related subtasks and utilizes the strong correlation between them to improve the accuracy of semantic understanding and fine-grained analysis capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0050] Figure 1 A schematic diagram of multimodal feature extraction of air traffic control instruction text and aircraft trajectory images provided by the present invention.

[0051] Figure 2 This is a diagram of the adaptive transformer network structure provided by the present invention.

[0052] Figure 3 A flowchart of the method for enhancing the airborne situation at an airport using multi-modal information fusion provided by the present invention.

[0053] Figure 4 This is an overall flow chart of the airport scene airborne situation enhancement method based on multi-modal information fusion provided by the present invention. DETAILED DESCRIPTION

[0054] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0055] The purpose of the present invention is to focus on the problem of airborne situation enhancement, adopt the solution idea of ​​"feature extraction, intent recognition, and perception enhancement", and provide an airport scene airborne situation enhancement method with multimodal information fusion, which can explore and capture the intrinsic relationship between fine-grained visual features and text features, and filter out noise or redundant information in multimodal data to enhance modeling capabilities. Secondly, the multi-head self-attention mechanism is integrated to increase the weight of information that contributes to the results. At the same time, the understanding of ATC instructions is effectively realized through the joint model of intent recognition and key parameter filling. The method provided by the present invention can be applied to a variety of operating scenarios, and intentions such as turning intentions, reporting intentions or take-off intentions can be analyzed through trajectories and instructions, and parameter information such as flight altitude and flight pressure can be obtained to provide a reference for the precise guidance of subsequent trajectories.

[0056] In order to illustrate the effectiveness of the method proposed by the present invention, the above technical solution of the present invention is described in detail below through a specific embodiment. Figure 3 , Figure 4 As shown, a method for enhancing the airborne situation at an airport scene by fusion of multi-modal information is disclosed, and the specific implementation steps are as follows:

[0057] Step S1, obtaining air traffic control instruction text and the taxiing trajectory image of the aircraft, performing feature extraction on the air traffic control instruction text and the taxiing trajectory image respectively, and obtaining text features and visual features.

[0058] like Figure 1 As shown, the airborne end obtains more accurate situation awareness information by integrating multiple modal data. Before fusing the multi-modal information, the present invention collects and extracts features from multi-modal information such as air traffic control instruction text and taxiing track image.

[0059] (1) For air traffic control instruction text, the present invention first obtains the instruction text transcribed from the air-ground conversation or the instruction text data sent by the digital control, and uses it as input, and uses the BERT model to generate features containing the semantics of the words in the instruction and the language context. Through the token embedding method, each word is represented by a vector of a fixed dimension. For words with similar meanings, the vector representations output by the BERT model will be more similar, making it easier to distinguish the meanings of the words.

[0060] In addition to word embeddings, the BERT model also contains two other components: 1) Fragment embedding: a vector representation used to distinguish different sentences. The BERT model is able to process two sentences at the same time, usually referred to as sentence A and sentence B. To distinguish the two sentences, each sentence is given a specific fragment embedding, recorded as a 768-dimensional vector. This helps the model understand the relationship and context between sentences; 2) Position embedding: used to represent the position of a word in a sentence. In the BERT model, each word position is assigned a corresponding 768-dimensional vector. This helps the model understand the order and position relationship of words in a sentence.

[0061] When an instruction text content W is input, W is modeled into a word sequence W = {w1,w2,...,w n},w n is the nth word, n represents the number of words in the text, the sequence is input into the BERT network for conversion, and the feature representation is S = {t1, t2, ..., t n}, where t n Corresponding to w n The converted features. The text feature S is converted by the pre-trained model BERT, and the expression is:

[0062] S={t1,t2,…,t n}=BERT(W) (1)

[0063] (2) For taxiing track images, the present invention generates a track image by matching the map scene with the expected aircraft taxiing track, intuitively displaying the movement path of the aircraft on the airport scene; and extracts features from the generated track image to obtain visual features.

[0064] To this end, it is necessary to know the location distribution of landmark objects on the airport surface, which helps to build a semantic map containing landmark location and trajectory information; through technical means such as road network modeling, road segment connection, trajectory densification and map matching, the high-precision map is matched with the expected taxiing trajectory. In this way, not only the taxiing situation of a single aircraft on the airport surface is displayed, but also its movement direction at the intersection of the taxiway can be displayed; as the aircraft actually moves on the airport surface, the input information will be continuously updated according to the position and orientation of the aircraft in the map. Ensure real-time tracking of the aircraft's taxiing status and provide the flight crew and surface control personnel with the latest information.

[0065] In the specific implementation, the semantic conversion and matching of scene trajectories are first performed, and the specific method is as follows:

[0066] First, construct an airport surface network map. The airport surface network is represented by a directed graph G(V,E), where V is a set of vertices representing taxiway intersections or turning points, and E is a set of directed edges representing airport ground network road segments. Road network construction can be divided into three main steps: 1) Data collection: Using satellite images and high-precision map data of the airport surface, collect detailed information such as the location, shape, and length of the apron and taxiways; 2) Road topology generation: Based on the map shape, the road segments are divided at the connection points to generate a road topology structure. Road topology represents the connectivity between intersections, nodes, and road segments; 3) Network formation: The road topology structure is converted into a network structure, in which intersections and nodes become nodes of the graph, and road segments become edges of the graph to form a road network.

[0067] Secondly, the road network map matching of the scene trajectory is performed. The map matching module combines the geometric matching algorithm and the topological relationship algorithm. Before map matching, the trajectory points need to be preprocessed. Specifically, the airport contour coordinates are used to remove all trajectory points outside the airport range. The purpose of this is to avoid the interference of trajectory points within the airport range on the subsequent map matching process. For the preprocessed trajectory point sequence (P1, P2, ..., P m ), use the point-to-line matching method to match the initial trajectory point P1 with the starting segment. Take all segments in the road network as candidate segments, calculate the Euclidean distance between the first trajectory point and all segments, and get the projection distance. After getting the projection distance, based on the maximum error distance d MAX Filter and sort all possible candidate segments. The purpose of this step is to select the most likely matching segment from all candidate segments, thereby improving the accuracy of map matching. Based on the above process, a candidate list L1 can be obtained, and the segments in the list are all segments that may match the trajectory point sequence. The formula for the Euclidean distance D is as follows:

[0068]

[0069] Among them, i is the index of the trajectory point, m is the total number of trajectory points, and x i ,y i are the horizontal and vertical coordinates of the i-th trajectory point respectively.

[0070] By traversing the candidate segments and considering the topological information of the current segment, a candidate list L2 is obtained. Continue under the distance error constraint and calculate the projection distance between the next trajectory point P2 and all candidate segments. Repeat this process, using the point-to-line matching method according to the topological information of the road network, and continuously match the road segments until the last trajectory point is reached. Finally, the complete map matching of the trajectory is completed and the trajectory image is obtained.

[0071] After completing the trajectory matching, the ResNet model is used to extract the visual features of the trajectory image, which are the outputs of the penultimate hidden layer of ResNet. Given a trajectory image V, the extracted visual features are T, expressed as:

[0072] T={v1,v2,…,v s}=ResNet(V) (3)

[0073] Among them, v s It represents the visual feature as the sth component of T.

[0074] Step S2: performing multimodal feature fusion on the text features and visual features based on an adaptive Transformer network to obtain multimodal features.

[0075] In a complex scene environment, the diversity of information and the characteristics of rapid changes in time and space make certain types of information obtained in a single domain not very resistant to interference and cannot effectively meet the requirements of environmental perception. To address this problem, the present invention provides a method for multimodal feature fusion based on an adaptive Transformer network, which combines expected trajectories and real-time instructions, and performs feature fusion through an intra-modal and inter-modal structure. It can obtain the connection between multiple modalities and provide information supplements in the time and space dimensions for the original task. The obtained multimodal features can be used to identify the expected action intentions of the aircraft and provide decision-making references. The specific steps include:

[0076] (1) Constructing an adaptive Transformer network. The present invention uses an adaptive Transformer network to fuse the features of the two modalities for the input text features and visual features. Specifically, the text features and visual features are first combined:

[0077]

[0078] Among them, F is the combined feature.

[0079] The main feature of the Transformer encoder is that it uses a self-attention mechanism, which allows the model to process all elements of the input sequence in parallel instead of processing one element at a time, which makes the Transformer more efficient and has better performance. The present invention uses a multi-head self-attention mechanism to determine the degree of association between the two modalities of text and vision. The combined features of the input are taken as a Query. In this case, the vector representations of Query, Key, and Value all come from the same input combined feature vector. The attention mechanism triple (Q F ,K F ,V F ) are all from the mapping of the same input feature, and the input of the adaptive Transformer network is calculated as follows:

[0080]

[0081] Among them, Q F , K F and V F are query, key and value respectively, W Q , W K and W V are the weights of query, key and value respectively, Q S , Q T are the textual and visual components of the query, respectively, and K S , K T are the text component and visual component of the key, V S , V T The textual and visual components of the value, respectively.

[0082] Although the existing Transformer network can process fused multimodal features, it still has limitations in some aspects. In particular, when processing the relationship between text features and visual features, it may not be able to fully express the attention similarity between the two. In order to better measure the attention similarity between text and vision, the present invention provides an adaptive Transformer network constructed by adding adaptive weight parameters to the intra-modal and inter-modal attention scores, and characterizes the similarity between features by dynamically calculating the attention scores between multimodal features. Multimodal attention can be defined as:

[0083]

[0084] Among them, λ1,λ2,λ3,λ4 are the adaptive weight parameters of the intra-modal and inter-modal data attention scores, λ1+λ2=1,λ3+λ4=1. After passing through the adaptive Transformer network, the fused multimodal features are:

[0085]

[0086] in, is a multimodal feature, is the text component of the multimodal feature, is the visual component of the multimodal feature, is the inter-modal attention of the text, is the inter-modal attention of visual information, is the inter-modal attention of text to vision, is the inter-modal attention of vision to text.

[0087] Multimodal features can be expressed as:

[0088]

[0089] Among them, f l Represents the lth representation of the multimodal feature.

[0090] During the operation of an aircraft, the changes in the environment and the information received are constantly changing. Therefore, it is necessary to fuse the features of effective air traffic control instructions and trajectory data so that the aircraft can obtain a more comprehensive fusion feature representation. This representation not only includes the content of the air traffic control instructions, but also takes into account the dynamic changes of trajectory data. Then, using the fusion features, the aircraft autonomously perceives its environment. Based on the perception of the environment, the aircraft obtains guidance on its next intention in real time. This helps the aircraft predict and plan its next actions to ensure the safety and efficiency of its operation.

[0091] like Figure 2 As shown in the figure, in the Transformer network, the input and output of the module are directly added as the final output, and this connection method is called residual connection. By using layer normalization technology, unlike other normalization techniques (for example, batch normalization), layer normalization normalizes all features of each sample instead of normalizing the same features in a batch. Next, the output feature vector is processed by a feedforward neural network, which can be used to process complex nonlinear relationships. After that, the residual connection and layer normalization operations are performed again to keep the transformed vector with the same length as the original vector, which helps prevent the occurrence of the gradient vanishing problem and improves the training stability of the model to obtain the multimodal features output by the adaptive Transformer network.

[0092] Step S3, performing intention recognition and key parameter analysis on the multimodal features respectively to obtain aircraft behavior intention and aircraft key parameters;

[0093] (1) Intent recognition. To achieve real-time and complete output of guidance information based on track-based precise guidance, the multimodal fusion features are matched with the actual control intent, and spatiotemporal parameter information is obtained to provide information for precise guidance of subsequent tracks. The fused multimodal features are input into the fully connected layer and softmax function to obtain accurate intent recognition results.

[0094] In some embodiments, the aircraft behavior intention may include push-off permission intention, turning intention, taxiing intention, take-off intention, etc. The aircraft will receive real-time prompts of the intention at its position in each state, which can assist the flight crew in performing the correct actions.

[0095] In this step, the multimodal features are fed into a fully connected layer for classification, and the output is the probability of each intent category. According to the first special representation f0 of the multimodal features, the predicted intent probability value is converted into a non-negative probability distribution through the softmax function, corresponding to the intended action to be executed, and the expression is:

[0096] y i =softmax(W i f0+b i ) (9)

[0097] Among them, y i represents the output intention probability value, softmax(·) is the activation function, W i is the intention recognition weight, b i Bias for intent identification.

[0098] In some embodiments, the behavioral intention obtained by intention recognition (such as turning intention, taxiing intention, etc.) can be output on the onboard IPAD for display and provided to the flight crew as an auxiliary module to better understand the action instructions and perceive the scene status.

[0099] (2) Key parameter analysis. The current key parameters can be analyzed from the command text combined with the language and visual information of the trajectory image. The command text modality contains key spatiotemporal operation parameters such as launch time, taxi path, and runway information. The expected trajectory modality can provide spatial direction and path guidance. After integrating the two information modalities, the action is decided based on the key parameters.

[0100] In this step, according to the remaining representations f1,…,f l Input to the softmax layer to identify the key parameters, the expression is:

[0101]

[0102] in, Indicates the lth key parameter, fl represents the lth representation of the multimodal feature, W i is the key parameter weight, b i is the key parameter bias, and L is the total number of representations of multimodal features.

[0103] Finally, the objective function of the entire adaptive Transformer network is determined as:

[0104]

[0105] Among them, y s Represents all key parameters, p(y i ,y s |x) means that given the multimodal feature x, y i ,y s The conditional probability of simultaneous occurrence, p(y i |x) represents y given the multimodal feature x i The conditional probability of occurrence, Represents given multimodal feature x The conditional probability of occurrence.

[0106] The training goal of the adaptive Transformer network is to maximize the objective function, fine-tune the model by minimizing the cross entropy loss, complete the training of the adaptive Transformer network through iterative updates, and finally output the key parameter information that the air traffic control instructions are most likely to carry.

[0107] Step S4: Using the aircraft behavior intention and aircraft key parameters as airborne situation auxiliary information to provide assistance for subsequent flight decisions.

[0108] By combining the real-time status with the expected taxiing path, the current intention of the aircraft and the key parameters of the aircraft can be obtained. The intention of the surface operation may include: push permission intention, turn intention, taxiing intention, take-off intention, etc. The key parameters of the aircraft may include the moving direction, speed and expected taxiing path of the aircraft. The aircraft behavior intention and key parameters of the aircraft are obtained as airborne situation auxiliary information, which can be provided to the flight crew as an auxiliary module to better understand the action instructions and perceive the surface status. It can also control the key nodes of the surface operation in time and space, so as to more accurately plan and adjust the operation of the aircraft to adapt to the ever-changing airport environment.

[0109] Although the specific embodiments of the present invention depict various actions or steps in a specific order, this should be understood as requiring such actions or steps to be performed in the specific order shown or in a sequential order, or requiring all illustrated actions or steps to be performed to obtain the desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination. The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0110] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for enhancing the airborne situation at an airport using multi-modal information fusion, characterized in that: The following steps are involved: Step S1, obtaining an air traffic control instruction text and an aircraft taxiing track image, and performing feature extraction on the air traffic control instruction text and the taxiing track image to obtain text features and visual features; Step S2: performing multimodal feature fusion on the text features and visual features based on an adaptive Transformer network to obtain multimodal features; Step S3, performing intention recognition and key parameter analysis on the multimodal features respectively to obtain aircraft behavior intention and aircraft key parameters; Step S4: Using the aircraft behavior intention and aircraft key parameters as airborne situation auxiliary information to provide assistance for subsequent flight decisions.

2. The method for enhancing the airport scene airborne situation by multi-modal information fusion according to claim 1 is characterized in that: Step S1 specifically includes: Step S1-1, obtaining the instruction text transcribed from the air-ground conversation or the instruction text sent by the digital control as the air traffic control instruction text, and using the BERT model to process the air traffic control instruction text to generate text features; Step S1-2: Match the airport surface network map with the expected aircraft taxiing trajectory to generate a taxiing trajectory image, and use the ResNet model to process the taxiing trajectory image to generate visual features.

3. The method for enhancing the airport scene airborne situation by multi-modal information fusion according to claim 2 is characterized in that: In step S1-1, the BERT model is used to process the air traffic control instruction text to generate text features, specifically including: Input the command text content W, and divide W into word sequences W = {w1,w2,...,w n },w n is the nth word, n represents the total number of words in W, W is input into the BERT model and converted to obtain text features S, S = {t1, t2, ..., t n },w n w n Features after conversion.

4. The method for enhancing the airport scene airborne situation by multi-modal information fusion according to claim 3 is characterized in that: Step S1-2 specifically includes: Step S1-2-1, converting the airport surface road topology structure into a network structure to form the airport surface network map; Step S1-2-2, performing Euclidean distance matching by traversing the aircraft taxiing trajectory points and the candidate road sections in the airport surface network map, and finally obtaining the taxiing trajectory image; Step S1-2-3: Use the ResNet model to extract the visual features of the trajectory image, and use the output of the penultimate hidden layer of the ResNet model as the visual features.

5. The method for enhancing the airport scene airborne situation by multi-modal information fusion according to claim 4 is characterized in that: Step S2 specifically includes: Step S2-1, combining text features and visual features to obtain combined features; and determining the input of the adaptive Transformer network based on the combined features; Step S2-2, adding adaptive weights to the attention scores within and between the text features and visual features, respectively, calculating the multimodal attention output by the adaptive Transformer network, and obtaining the multimodal features.

6. The method for enhancing the airport scene airborne situation by multi-modal information fusion according to claim 5 is characterized in that: In step S2-1, the input expression of the adaptive Transformer network is: Among them, Q F , K F and V F are query, key and value respectively, W Q , W K and W V are the weights of query, key and value respectively, Q S , Q T are the textual and visual components of the query, respectively, and K S , K T are the text component and visual component of the key, V S , V T The textual and visual components of the value, respectively.

7. The method for enhancing the airport scene airborne situation by multi-modal information fusion according to claim 6 is characterized in that: In step S2-2, the multimodal attention output by the adaptive Transformer network (Q F ,K F ,V F ) is calculated as: Among them, λ1,λ2,λ3,λ4 are the adaptive weight parameters of the intra-modal and inter-modal data attention scores; The expression of the multimodal feature is: in, is a multimodal feature, is the text component of the multimodal feature, is the visual component of the multimodal feature, is the inter-modal attention of the text, is the inter-modal attention of visual information, is the inter-modal attention of text to vision, is the inter-modal attention of vision to text; f l Represents the lth representation of the multimodal feature.

8. The method for enhancing the airport scene airborne situation by multi-modal information fusion according to claim 7 is characterized in that: Step S3 specifically includes: Step S3-1, performing a fully connected layer calculation on the first representation of the multimodal feature to obtain the aircraft behavior intention; Step S3-2: Perform full connection layer calculation on the representations of the multimodal features except the first representation to obtain key parameters of the aircraft.

9. The method for enhancing the airport scene airborne situation by multi-modal information fusion according to claim 8 is characterized in that: Step S3-1 specifically includes: According to the first special representation f0 of the multimodal feature, the predicted intention probability value is converted into an intention probability value through the softmax function as the aircraft behavior intention, which is expressed as: y i =softmax(W i f0+b i ) Among them, y i represents the probability value of intention, softmax(·) is the activation function, and W i is the intention recognition weight, b i bias for intent identification; Step S3-2 specifically includes: According to the representation of the multimodal features f1,…,f l , the key parameters of the aircraft are identified through the softmax function, and the expression is: in, Indicates the lth key parameter, f l represents the lth representation of the multimodal feature, W i is the key parameter weight, b i is the key parameter bias, and L is the total number of representations of multimodal features.

10. The method for enhancing the airport scene airborne situation by multi-modal information fusion according to claim 9, characterized in that: The objective function of the adaptive Transformer network is: Among them, y s Represents all key parameters, p(y i ,y s |x) means that given the multimodal feature x, y i ,y s The conditional probability of simultaneous occurrence, p(y i |x) represents y given the multimodal feature x i The conditional probability of occurrence, Represents given multimodal feature x Conditional probability of occurrence; The training goal of the adaptive Transformer network is to maximize the objective function, fine-tune the model by minimizing the cross entropy loss, and complete the training of the adaptive Transformer network through iterative updating.

Images