Civil aviation aircraft positioning guarantee multi-modal intelligent identification linkage method and device

Abstract

The application belongs to the technical field of aircraft safety management, and discloses a civil aviation aircraft parking guarantee multi-modal intelligent identification linkage method and device. The method comprises the following steps: collecting multi-modal data in the civil aviation aircraft parking stage to obtain standardized multi-modal feature data; detecting the confidence conflict between nodes in the graph model, and monitoring the distribution drift of each modal feature data, and calculating the comprehensive uncertainty measure of each modal feature data based on the confidence conflict and the distribution drift; inputting the multi-modal feature data and the comprehensive uncertainty measure into a multi-modal fusion decision model to generate the parking state evaluation result and the risk level of the aircraft; and deciding to execute the linkage scheduling of guarantee resources according to the parking state evaluation result and the risk level. The application can overcome the key defects of the traditional artificial mode, such as reaction lag, strong subjectivity, and the existing shallow multi-modal fusion method which is easily polluted by unreliable modes due to equal treatment of all data, and improve the safety and operation efficiency of the parking guarantee.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aircraft safety management, in particular to a multi-modal intelligent identification linkage method and device for civil aviation aircraft parking position guarantee. BACKGROUND

[0002] Civil aviation aircraft parking position guarantee is a core link of airport ground operation, and its intelligent level is directly related to safety and efficiency. Early and current most systems mainly rely on manual dominant mode and single data source decision mode.

[0003] Manual dominant mode: excessive reliance on visual observation and experience judgment of tower controllers, ground guides and the like. This method is easily disturbed by environmental factors such as rain, snow, fog and low light, has problems of delayed reaction, strong subjectivity and high labor resource cost, and is difficult to meet the needs of all-weather and high-efficiency modern airport operation.

[0004] Single data source decision mode: some systems introduce single type data such as video monitoring, scene monitoring radar or ADS-B for automated assistance. However, relying on single modal data cannot comprehensively and accurately depict complex parking position situation. For example, only video cannot work reliably in bad weather; only position data cannot intuitively perceive the relative relationship between the aircraft and the parking position marking line, and cannot judge the state of the guarantee resources, resulting in one-sided decision dimension and easy misjudgment and omission.

[0005] In order to overcome the limitations of single mode, the industry has begun to explore multi-modal fusion technology, but the existing fusion methods (such as simple feature splicing or decision-level voting) usually treat all modal data equally, lacking fine quantitative evaluation of real-time reliability of each modal data. When a mode has decreased data quality due to sensor failure, signal interference or environmental changes (such as lens contamination, radar jump), its unreliable information will directly pollute the fusion result and introduce rather than eliminate decision risk.

[0006] Therefore, under the current technical background, there is an urgent need for a civil aviation aircraft parking position guarantee scheme based on multi-modal data with higher accuracy and reliability. SUMMARY

[0007] The purpose of the present application is to provide a multi-modal intelligent identification linkage method and device for civil aviation aircraft parking position guarantee to solve the problems raised in the background art.

[0008] The present application provides a multi-modal intelligent identification linkage method for civil aviation aircraft parking position guarantee, comprising the following method steps:

[0009] Step S10, multi-modal data is collected in the civil aviation aircraft parking stage, including the parking position video data, the aircraft running state data and the guarantee node state data, the multi-modal data is cleaned, time synchronized and feature extracted, and the standardized multi-modal feature data is obtained;

[0010] Step S20, a graph model is constructed based on the space-time logical relationship between the modal feature data, the confidence conflict between the nodes in the graph is detected, and the distribution drift of the modal feature data is monitored, and the comprehensive uncertainty measure of the modal feature data is calculated based on the confidence conflict and the distribution drift;

[0011] Step S30, the standardized multi-modal feature data and the comprehensive uncertainty measure are input into a multi-modal fusion decision model, in the cross-modal attention calculation process of the multi-modal fusion decision model, the attention weight of the high-uncertainty modal feature data is dynamically suppressed by using the comprehensive uncertainty measure, and the parking state evaluation result and the risk level of the aircraft are generated;

[0012] Step S40, according to the parking state evaluation result and the risk level, the guarantee resource linkage scheduling is decided to be executed, including: when the evaluation result is left or right deviation, the trajectory adjustment prompt is sent to the ground guide; when the evaluation result is that the bridge is not in place, the bridge operator is dispatched, and the luggage trailer is stopped from approaching; when the evaluation result is speed anomaly, the alarm is sent to the ground guide and the tower to prompt the aircraft to slow down.

[0013] The application also provides a civil aviation aircraft parking guarantee multi-modal intelligent identification linkage device, the device comprising a multi-modal acquisition unit, an uncertainty measurement unit, a decision unit and a linkage scheduling unit.

[0014] The multi-modal acquisition unit is configured to collect multi-modal data in the civil aviation aircraft parking stage, including the parking position video data, the aircraft running state data and the guarantee node state data, clean, time synchronize and extract features from the multi-modal data, and obtain standardized multi-modal feature data.

[0015] The uncertainty measurement unit is configured to construct a graph model based on the space-time logical relationship between the modal feature data, detect the confidence conflict between the nodes in the graph, monitor the distribution drift of the modal feature data, and calculate the comprehensive uncertainty measure of the modal feature data based on the confidence conflict and the distribution drift.

[0016] The decision unit is configured to input the standardized multi-modal feature data and the respective comprehensive uncertainty measure into a multi-modal fusion decision model, and in the cross-modal attention calculation process of the multi-modal fusion decision model, the respective comprehensive uncertainty measure is used to dynamically suppress the attention weight of high-uncertainty modal feature data, so as to generate an entry state evaluation result and a risk level of the aircraft.

[0017] The linkage scheduling unit is configured to decide to perform linkage scheduling of guarantee resources according to the entry state evaluation result and the risk level, including: when the evaluation result is left or right trajectory deviation, sending a trajectory adjustment prompt to a ground guide; when the evaluation result is that the bridge is not in place, scheduling a bridge operator and suspending the approach of the luggage trailer; and when the evaluation result is speed anomaly, sending an alarm to the ground guide and the tower to prompt the aircraft to slow down.

[0018] By constructing a guarantee scene space-time graph model and monitoring data distribution drift, the application accurately calculates the respective modal comprehensive uncertainty measure that fuses data consistency conflict and model environment adaptability, and then deeply couples the comprehensive uncertainty measure to the cross-modal attention mechanism of the multi-modal fusion decision model, so as to dynamically and accurately suppress the attention weight of high-uncertainty modal, thereby significantly improving the decision robustness and reliability of the aircraft entry state evaluation in a complex interference environment, and finally driving the active and intelligent linkage of guarantee resources according to the high-credibility evaluation result. Therefore, the application can effectively overcome the key defects of the traditional manual mode, such as reaction lag and strong subjectivity, and the existing shallow multi-modal fusion method, which is easily polluted by unreliable modal due to equal treatment of all data, and improve the safety and operation efficiency of entry guarantee. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of a civil aviation aircraft entry guarantee multi-modal intelligent identification linkage method disclosed by an embodiment of the application;

[0020] Figure 2 A schematic diagram of a core processing process of a civil aviation aircraft entry guarantee multi-modal intelligent identification linkage method disclosed by an embodiment of the application;

[0021] Figure 3 A structural schematic diagram of a civil aviation aircraft entry guarantee multi-modal intelligent identification linkage device disclosed by an embodiment of the application. DETAILED DESCRIPTION

[0022] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0023] Please refer to Figure 1 、 Figure 2 The present application provides a multi-modal intelligent identification linkage method for civil aviation aircraft parking, comprising the following method steps:

[0024] Step S10, collecting multi-modal data in the civil aviation aircraft parking stage, including gate video data, aircraft operation state data and guarantee node state data, cleaning, time synchronizing and feature extracting the multi-modal data to obtain standardized multi-modal feature data;

[0025] In this step, in the civil aviation aircraft parking stage, through the ultra-high-definition camera deployed around the gate (such as the four corners of the gate, the bridge column, etc.), the video stream data in the aircraft parking process is collected in real time to form the gate video data; through the airport data interface (such as ADS-B receiving equipment, aircraft ACARS system, airport surface monitoring radar), the dynamic data of the aircraft is obtained in real time, including real-time position, flight speed, heading angle, etc., to form the aircraft operation state data; through the vehicle-mounted, handheld and other Internet of Things sensors, the position coordinates, working state of the ground service equipment (such as the bridge, the luggage trailer), and the real-time positioning and arrival state of the guarantee personnel are collected to form the guarantee node state data.

[0026] The collected original multi-modal data is preprocessed, for example, including:

[0027] Data cleaning and standardization: frame extraction is performed on the video stream to remove blurred frames and occluded frames; outlier elimination (such as satellite positioning jump data) and unit unification are performed on the operation state data and guarantee node data.

[0028] Time synchronization: based on the timestamp and the unified clock source of the airport, the video frame data, the aircraft operation state data and the guarantee node state data are aligned to ensure the correlation of the multi-modal data in the same time slice.

[0029] Feature extraction is performed on the preprocessed multi-modal data. Specifically, for the video frame data, the aircraft contour, gate marking line and other visual features are extracted through the convolutional neural network; the "speed-position-pose" feature sequence is extracted from the operation state data; the "device position-state" and "personnel arrival rate" feature vectors are extracted from the guarantee node data. Further, the standardized multi-modal feature data is obtained.

[0030] Step S20, constructing a graph model based on the spatio-temporal logical relationship between the modal feature data, detecting the confidence conflict between the nodes in the graph, and monitoring the distribution drift of the modal feature data, calculating the comprehensive uncertainty measure of the modal feature data based on the confidence conflict and the distribution drift;

[0031] In this step, based on the standardized multi-modal feature data extracted in the foregoing step S10, entities such as aircraft, jetty, and ground service equipment are abstracted as graph nodes, and information such as entity state and position parsed from the multi-modal feature data is given to the corresponding nodes as their attributes; at the same time, their mutual relationship in the physical space and the support process (such as relative position and docking relationship) is abstracted as an edge, thereby constructing a dynamic support scene spatio-temporal graph model. It can be understood that this graph model is not independent of the collected data, but the discrete multi-modal feature data is organized in a structured model with clear spatio-temporal and logical relationship according to its semantic attribution, and the state of the nodes and edges in the graph is updated in real time by the corresponding modal feature data.

[0032] Message passing and reasoning are performed in the graph model, each node outputs a state estimate and its confidence according to its associated modal feature data (for example, the attributes of the aircraft node are composed of its video features and running state features), and then the confidence conflict of the states of the connected nodes in the graph is detected in real time. For example, the aircraft node judges “has been stabilized” with high confidence according to the video data, but the jetty node judges “still far away” with high confidence according to its sensor data, then a strong confidence conflict occurs between these two nodes, and this conflict signal will be captured and quantified.

[0033] In parallel, the real-time distribution of each modal feature data is monitored, and the difference between its distribution and the historical data distribution during model training, i.e. the distribution drift, is calculated. For example, a sudden snowfall causes a global change in all video image features, i.e. a significant distribution drift, which indicates that the visual perception model trained based on sunny weather data has decreased reliability in the current environment.

[0034] Based on the above process, the confidence conflict intensity from the inconsistency of the neighbor nodes in the graph and the distribution drift degree indicating whether the model is adapted to the current environment of the single modal feature data are fused, and the comprehensive uncertainty measure of the modal feature data is calculated. It can be understood that the comprehensive uncertainty measure can dynamically and comprehensively reflect the credibility of the modal feature data in supporting the entire graph model reasoning process.

[0035] Step S30, input the standardized multi-modal feature data and the corresponding comprehensive uncertainty measures into a multi-modal fusion decision model. In the cross-modal attention calculation process of the multi-modal fusion decision model, the comprehensive uncertainty measures are used to dynamically suppress the attention weight of high-uncertainty modal feature data, and an entry state evaluation result and a risk level of the aircraft are generated.

[0036] In this step, the standardized multi-modal feature data obtained in step S10 and the comprehensive uncertainty measures calculated in step S20 are input into a multi-modal fusion decision model. Preferably, the multi-modal fusion decision model is constructed based on a Transformer architecture. The multi-modal fusion decision model performs cross-modal attention calculation, in which the allocation of attention weights is dynamically adjusted using the comprehensive uncertainty measures of the modal feature data. Specifically, the higher the uncertainty measure of a modal feature data, the lower its weight in the attention mechanism, so that the multi-modal fusion decision model can automatically focus on more reliable data sources and weaken or even ignore the interference of unreliable modal.

[0037] After uncertainty-guided deep fusion, the multi-modal fusion decision model outputs a comprehensive evaluation of the current entry state, including an entry state evaluation result (such as "normal entry trajectory", "X meters left deviation", "excessive speed") and a corresponding risk level (such as "no risk", "low risk", "high risk"). It can be understood that the above decision results exclude the interference of unreliable data, and their accuracy and robustness are significantly improved.

[0038] Step S40, according to the entry state evaluation result and the risk level, decision-making is performed to guarantee resource linkage scheduling, including: when the evaluation result is trajectory left or right deviation, sending a trajectory adjustment prompt to the ground guide; when the evaluation result is that the bridge is not in place, scheduling the bridge operator and suspending the approach of the luggage trailer; when the evaluation result is abnormal speed, sending an alarm to the ground guide and the tower to prompt the aircraft to slow down.

[0039] In this step, according to the evaluation result of the positioning state generated in step S30 and the risk level, the active decision and linkage scheduling of the support resources are triggered. For example, when the evaluation result is "trajectory left or right deviation (low risk)", the trajectory adjustment prompt is sent to the ground guide personnel (for example, prompting intervention by gesture or guide vehicle); when the evaluation result is "corridor bridge not in place (high risk)", the corridor bridge operator is preferentially scheduled, and the luggage trailer is temporarily stopped from approaching; when the evaluation result is "abnormal speed (too fast or too slow)", the alarm is sent to the ground guide personnel and the tower to prompt the aircraft to slow down; when the evaluation result is "normal", no response is performed. Through the airport resource scheduling system (such as the FMS system interface), the scheduling instructions are sent to the corresponding corridor bridge control system, ground equipment dispatch center and support personnel mobile terminal, realizing the active resource linkage based on the real-time situation, and improving the safety and efficiency of the positioning.

[0040] By constructing the support scene space-time graph model and monitoring the data distribution drift, the application accurately calculates the comprehensive uncertainty measure of each modality which fuses data consistency conflict and model environment adaptability, and then deeply couples the comprehensive uncertainty measure to the cross-modal attention mechanism of the multi-modal fusion decision model, realizes the dynamic and accurate suppression of the attention weight of the high uncertainty modality, thereby significantly improves the decision robustness and reliability of the aircraft positioning state evaluation in the complex interference environment, and finally drives the active and intelligent linkage of the support resources according to the high reliability evaluation result. Therefore, the application can effectively overcome the key defects of the traditional manual mode such as reaction lag, strong subjectivity and the pollution of unreliable modalities caused by the equal treatment of all data in the existing shallow multi-modal fusion method, and improve the safety and operating efficiency of the positioning support.

[0041] As an example, a graph model is constructed based on the space-time logical relationship between the feature data of each modality, the confidence conflict between the nodes in the graph is detected, and the distribution drift of the feature data of each modality is monitored, including:

[0042] In step S21, the aircraft, each support equipment and key infrastructure are taken as nodes, the spatial proximity, physical connection relationship and support process dependency relationship between the nodes are taken as edges, and a dynamically evolving space-time heterogeneous graph model is constructed.

[0043] This step aims to construct a structured model, i.e. a space-time heterogeneous graph model, which can accurately represent the complex correlation of the support scene, and the specific steps are as follows:

[0044] Node definition: the key entities in the physical world are abstracted as graph nodes, mainly including: aircraft node: representing the aircraft being positioned; each support equipment node: such as corridor bridge, luggage trailer, refueling vehicle, etc.; key infrastructure node: such as parking space marking line, wheel stop placement point, etc.

[0045] Edge relationship definition: According to the actual interaction between entities, multiple types of edges are defined to construct the spatio-temporal heterogeneous graph model, including spatial proximity edge: based on real-time location data, when the Euclidean distance between two entities (such as aircraft and bridge) is less than a set threshold, a connection is established. Physical connection relationship edge: representing the inherent physical connection between entities, for example, the fixed connection relationship between the bridge and the terminal. Security process dependency edge: based on the logical dependency established by the security operation process, for example, after the aircraft is stabilized, the wheel block security node is activated, and there is a strong dependency relationship between the two.

[0046] It should be noted that the spatio-temporal heterogeneous graph model is not static, its node attributes (such as location, state) and edge relationships (such as distance changes leading to the establishment or disappearance of spatial edges) are dynamically updated in real time with the aircraft entering process, thus accurately reflecting the instantaneous situation of the scene.

[0047] Step S22, the state information of each node and its connected neighbors is aggregated through the graph attention network to generate an enhanced state representation for each node containing neighborhood context information; based on the enhanced state representation, a confidence distribution of the state of each node is generated respectively, and the KL divergence between the confidence distributions of the node pairs directly connected through the edge is calculated; for each modal feature data, the confidence conflict measure is obtained based on the calculated KL divergence of all node pairs associated with it;

[0048] In this step, the graph attention network (Graph Attention Network, GAT) is used to perform message passing on the spatio-temporal heterogeneous graph model. For each node in the graph, GAT calculates the attention coefficient of its connection with each adjacent node through its built-in attention mechanism, which dynamically reflects the importance of different neighbor node information in the aggregation process. Based on the calculated attention coefficient, the state feature vectors of all adjacent nodes are weighted and summed, thereby generating an enhanced state representation for each node that integrates its local neighborhood context information.

[0049] Each node is processed by its corresponding state inference model (e.g., a fully connected layer and a Softmax classifier) based on the generated enhanced state representation, outputting a probability distribution about its own state (such as "position normal", "position left bias X meters", "speed abnormal", etc.), i.e., state confidence distribution. It should be noted that this confidence distribution is a comprehensive judgment made by the node after fully sensing the neighborhood context, and its reliability is better than the independent judgment based on its own modal feature data.

[0050] The Kullback-Leibler Divergence between the state confidence distributions generated by pairs of nodes directly connected by edges is calculated. The KL divergence is an asymmetric measure used in information theory to quantify the difference between two probability distributions. The larger the KL divergence value, the more significant the contradiction between the judgments of the two nodes on the associated state, even after fully perceiving the context information of each other and their surroundings. For example, when the aircraft node still judges itself to be "parked" with high confidence after perceiving the neighbor information such as the reference bridge and the stop line, and its adjacent bridge node still judges the aircraft to be "far away" with high confidence after the same context perception, the KL divergence value between the confidence distributions of the two nodes will significantly increase.

[0051] Exemplarily: Assume that at a certain time, there are two directly connected nodes in the spatiotemporal heterogeneous graph model: an aircraft node A and a bridge node B. Node A (associated with feature data such as video modalities) processes its state inference model and outputs its confidence distribution on "whether it has been parked" as: , i.e., the probability of judging "parked" is 0.95, and the probability of judging "not parked" is 0.05. Node B (associated with feature data such as support node modalities) outputs its confidence distribution on "whether the aircraft has entered the docking area" as: , i.e., the probability of judging "entered" is 0.10, and the probability of judging "not entered" is 0.90.

[0052] The KL divergence between the above two distributions is calculated to quantify the contradiction:

[0053] ;

[0054] The value is significantly greater than 0, indicating that there is a serious conflict between the judgments of node A and node B.

[0055] For each modality feature data, based on the KL divergence values calculated for all pairs of nodes associated with it, the final confidence conflict measure of the modality feature data is obtained through a predefined aggregation strategy. It can be understood that the aggregation strategy includes but is not limited to taking the maximum value, taking the average value, or weighted average, etc. For example, when the strategy of taking the maximum value is adopted, the maximum KL divergence value appearing in all node pairs related to the modality feature data is taken as the confidence conflict measure of the modality feature data, which can effectively capture the maximum consistency risk caused by the modality feature data in the system.

[0056] Exemplarily: Assume that the video modality feature data is associated with three node pairs at the current time, and the calculated KL divergence values are: , , .

[0057] If a predefined max-value aggregation strategy is adopted, the confidence conflict measure of the video modality at the current time instant is: ;

[0058] If an average-value strategy is adopted, then: .

[0059] At step S23, for each modality feature data, a high-dimensional feature is extracted by a sliding window, and a maximum mean difference between the high-dimensional feature distribution in the window and the training period reference distribution is calculated, and the maximum mean difference is taken as the distribution drift measure of the modality feature data; wherein the training period refers to the training period of the model used to extract the corresponding modality feature data.

[0060] In this step, for the real-time feature data of each modality, a sliding window of a fixed time length is used as a unit, and the corresponding trained feature extraction model (such as CNN for video and encoder for numerical data) is processed to extract the high-dimensional feature vector of each data sample in the window.

[0061] The training period reference distribution refers to the data set used to train the feature extraction model currently being used during the model development stage, wherein the overall distribution of the high-dimensional features of all samples. It can be understood that this distribution represents the data environment that the feature extraction model is optimized to adapt to and is considered reliable.

[0062] The maximum mean difference (Maximum Mean Discrepancy, MMD) between the real-time distribution of the high-dimensional features of all data samples in the current sliding window and the above-mentioned training period reference distribution is calculated. MMD is a two-sample test method based on kernel method in statistics for judging whether two samples are taken from the same distribution, which calculates the distance between the mean values of two distributions in the reproducing kernel Hilbert space, and can effectively capture the difference between high-dimensional feature distributions. The larger the MMD value, the more significant the difference between the feature distribution of the current modality feature data and the feature distribution of the data relied on during model training, that is, a serious distribution drift has occurred.

[0063] Exemplarily: after the training of the feature extraction model is completed, the feature extraction model is forward propagated on the entire training data set, and the output of a specific layer (such as the last fully connected layer) is extracted as a high-dimensional feature vector. The set of feature vectors of all training samples is denoted as the reference feature set , wherein is the total number of training samples, and each feature vector . It can be understood that the set represents the reference data distribution learned by the feature extraction model during the training stage characterizing the feature space of the training data. This set can be pre-computed and stored.

[0064] During real-time running, the following loop is executed for each modality (take video modality as an example): maintain a fixed-length sliding time window (e.g., a video frame buffer lasting 5 seconds). For each frame image within the window , forward propagation is performed using the same feature extraction model as in the training period to extract high-dimensional feature vectors of the same layer . The feature vectors of all frames within the current sliding window are collected to form a real-time feature set , where is the number of frames (samples) within the window.

[0065] After each sliding window update, the MMD2 statistic between the real-time feature set and the baseline feature set is calculated. The calculation uses an unbiased estimator, whose formula and process are as follows:

[0066] ;

[0067] where: is a positive definite kernel function, which can be selected as a Gaussian radial basis function kernel: , is the kernel parameter, is set to the median distance of all sample pairs in the two sets.

[0068] Take the square root of the calculated value and ensure it is non-negative to get the final MMD value: . This MMD value is taken as the distribution drift measure of the feature data of the current modality (video modality).

[0069] For example, when a sudden snowstorm occurs, the image feature distribution extracted by the video modality will produce a huge MMD value with the training baseline distribution under good weather. Finally, the calculated MMD value is taken as the distribution drift measure of the modality feature data, which is used to objectively characterize the environmental inadaptability of its perception model, i.e., the feature extraction model.

[0070] Illustratively: the feature extraction model is trained on a large number of airport video data under good weather. After training, these training data are re-input into the feature extraction model to extract the activation values of the last convolution or fully connected layer as high-dimensional features, forming the baseline sample set . In the inference running phase, a sudden snowstorm occurs, and the last 5 seconds of video frames are continuously collected, and their features are extracted using the same feature extraction model to form the real-time sample set . Substitute and into the MMD2 formula for calculation.

[0071] Since the image features of sunny weather (sufficient light, high contrast, clear edges) and the image features of heavy snow weather (overall white, low contrast, blurred details) are very different in the feature space, therefore: The similarity of the internal samples will be very high (because they are all sunny weather features). The similarity of the internal samples will also be very high (because they are all heavy snow weather features). But The cross-similarity between will be very low, resulting in a very large calculated value, and the final distribution shift measure (MMD value) will also be very large.

[0072] Based on the spatio-temporal heterogeneous graph model and the graph attention network, the application generates a context-aware enhanced state representation by aggregating node neighborhood information, and calculates a more reliable state confidence distribution accordingly. Then, the KL divergence is used to accurately quantify the fundamental confidence conflicts that still exist between nodes after context fusion. At the same time, the sliding window and the maximum mean difference (MMD) are used to strictly monitor the degree of drift of the real-time data distribution of each modal feature data relative to its model training period benchmark. Both of them together constitute a fine-grained uncertainty measurement mechanism that can deeply perceive data consistency conflicts and model environment mismatches, thereby providing more accurate and robust credibility basis for subsequent multi-modal fusion decision-making.

[0073] As an example, the calculation of the comprehensive uncertainty measure of each modal feature data based on the confidence conflict and the distribution drift includes:

[0074] For each modal feature data, its corresponding confidence conflict measure and distribution drift measure are weighted and fused to obtain the corresponding comprehensive uncertainty measure. The weight of the confidence conflict measure is dynamically allocated according to the degree centrality of the node corresponding to the modal feature data in the spatio-temporal heterogeneous graph model. The weight of the distribution drift measure is dynamically allocated according to the historical dependence degree of the modal feature data in the current entry stage.

[0075] In this embodiment, for each modal feature data, its comprehensive uncertainty measure is calculated by weighting and fusing its corresponding confidence conflict measure (i.e. the KL divergence value) and the distribution drift measure (i.e. the MMD value). Its calculation formula can be expressed as: comprehensive uncertainty measure = α*confidence conflict measure + β*distribution drift measure, where α and β are the dynamically allocated weights.

[0076] For the weight a: calculate the degree centrality of the node corresponding to the modal feature data in the constructed spatiotemporal heterogeneous graph model. The degree centrality is a graph theory index, the value of which is the number of edges directly connected to the node, used to measure the connectivity and influence of the node in the network. The higher the degree centrality of the node, the greater the weight a of the confidence conflict measure assigned to it. This is because, if a highly central node (for example, an aircraft node connected to a corridor bridge, a stop line, and multiple ground service equipment) has data conflicts, it means that its untrusted state will quickly spread through numerous connections and affect the state judgment of a large number of other nodes, thereby posing a greater risk to global system decision-making. Therefore, giving a higher weight to its conflicts is in line with the principle of risk control.

[0077] For the weight β: based on historical operation data, analyze the contribution of each modal feature data to the final correct decision at different parking stages (such as "remote approach", "nose alignment", "final stable stop"), i.e. the historical dependence degree. For example, through information entropy or feature importance analysis, etc. Quantitative results show that: in the "final stable stop" stage, the data of the modal feature data of the guarantee node is crucial. Identify the current parking stage, and adjust the weight β of the distribution drift measure of the modality with high historical dependence degree in this stage accordingly. In other words, the more important the modal feature data in the current task stage, the greater the risk caused by the decline in its reliability if the perception model of the modal feature data has distribution drift (i.e. "inadaptation") due to environmental changes, and therefore needs to be more strongly reflected in the comprehensive uncertainty.

[0078] By introducing the confidence conflict dynamic weighting mechanism based on node degree centrality and the distribution drift dynamic weighting mechanism based on the historical dependence degree of the parking stage, the application realizes scene adaptive calculation of the comprehensive uncertainty measure, so that the final evaluation result not only reflects the quality contradiction of the data itself, but also accurately depicts the global impact risk of node failure in the guarantee network and the key risk of perception model mismatch in a specific task stage, thereby providing a high-value credibility input with topological perception and task perception characteristics for multi-modal fusion decision-making, and improving the identification and response ability of the system to core risk sources in a complex dynamic environment.

[0079] As an example, the weight of the confidence conflict measure is dynamically allocated according to the degree centrality of the node corresponding to the modal feature data in the spatiotemporal heterogeneous graph model, comprising:

[0080] Calculate the degree centrality of each node in the spatiotemporal heterogeneous graph model, and perform logarithmic compression processing on the degree centrality; wherein the degree centrality is obtained by the ratio of the connection degree of the node to the network size;

[0081] In this step, the standardized degree centrality of each node in the spatiotemporal heterogeneous graph model is calculated. Specifically, for a node degree centrality is calculated by the following formula:

[0082] ;

[0083] wherein, denotes the degree of a node, i.e. the number of edges directly connected to the node, and n is the total number of nodes in the spatio-temporal heterogeneous graph model.

[0084] The calculated degree centrality is subjected to logarithmic compression processing: ;which is the compressed degree centrality. This processing can effectively prevent the weight distribution of a highly central node from being too large, and avoid a single node dominating the entire uncertainty calculation process. Based on the compressed degree centrality value, the initial weight of each node is calculated by a softmax function with a learnable temperature parameter, wherein the temperature parameter controls the concentration degree of the weight distribution.

[0085] In this step, based on the compressed degree centrality

[0086] calculated in the foregoing, the initial weight distribution of each node is calculated by a softmax function with a learnable temperature parameter. The specific calculation formula is:

[0087] ;

[0088] wherein, is a learnable temperature parameter for controlling the concentration degree of the weight distribution. When the value is larger, the weight distribution is more concentrated, and the important nodes obtain higher weights; when the value is smaller, the weight distribution is more gentle, and the weight difference between nodes is reduced.

[0089] Based on the weight smoothing mechanism, the initial weight is linearly interpolated with the uniformly distributed weight, and the smoothed weight is corrected according to the node type, and the corrected weight is taken as the weight of the confidence conflict measure of each modal feature data.

[0090] In this step, based on the weight smoothing mechanism, the initial weight obtained in the foregoing is linearly interpolated with the uniformly distributed weight:

[0091] ;

[0092] wherein, is a preset smoothing factor for ensuring that each modal feature data obtains a basic weight guarantee.

[0093] Then, the smoothed weight is corrected according to the node type:​

[0094] ;

[0095] wherein, is a correction coefficient related to the type of node . Specifically, the aircraft node and the key support equipment node are assigned a weight promotion coefficient , and the auxiliary node is assigned a weight suppression coefficient .

[0096] Finally, the corrected weight is used as the weight of the confidence conflict measure of each modal feature data.

[0097] The above multi-level weight adjustment mechanism adopted by this embodiment not only fully considers the topological importance of the node in the network, but also avoids the dominance of a single node in weight allocation, while ensuring the appropriate influence of key entities in decision-making, and finally realizes the reasonable, stable and business logic-compliant dynamic allocation of the confidence conflict measure weight.

[0098] As an example, in the cross-modal attention calculation process of the multi-modal fusion decision model, the attention weight of the high-uncertainty modal feature data is dynamically suppressed by using each of the comprehensive uncertainty measures, and the entry state evaluation result and the risk level of the aircraft are generated, including:

[0099] Step S31: mapping the comprehensive uncertainty measure of each modal feature data to a reliability coefficient in the interval [0, 1] through a negative exponential function; wherein the reliability coefficient is negatively correlated with the comprehensive uncertainty measure;

[0100] This step aims to convert the comprehensive uncertainty measure into a reliability coefficient suitable for attention mechanism adjustment. In specific implementation, a negative exponential function is used for mapping, wherein is the comprehensive uncertainty measure, is the reliability coefficient, and is an adjustable sensitivity parameter.

[0101] It can be understood that a negative correlation between the reliability coefficient and the comprehensive uncertainty measure is established through this mapping. When the comprehensive uncertainty measure increases, the reliability coefficient exponentially decays, thereby achieving a strong suppression effect on high-uncertainty data.

[0102] ​Step S32, when calculating the cross-modal attention weight, multiply the dot product score of the query vector and each modal feature data key vector by the reliability coefficient of the corresponding modal feature data, and then perform softmax normalization processing to obtain the uncertainty-aware attention weight;

[0103] In this step, the reliability coefficient is integrated into the standard attention calculation process. The specific calculation process is as follows:

[0104] First, calculate the original matching degree of the query vector and each modal feature data key vector according to the standard attention mechanism:

[0105]

[0106] Among them: is the query vector, is the key vector of the i-th modal feature data, is the dimension of the key vector, is the reliability coefficient of the i-th modal feature data, and is the final uncertainty-aware attention weight. Next, introduce the reliability coefficient to adjust the original score:

[0107] .

[0108] This calculation multiplies the reliability coefficient of each modal feature data with its corresponding attention score , so that the score of the high-uncertainty modal feature data (corresponding to a low value) is significantly suppressed.

[0109] Finally, normalize the adjusted score:

[0110] .

[0111] Through the softmax function, the sum of all weights is ensured to be 1, forming a complete uncertainty-aware attention distribution. This design not only maintains the differentiable nature of the attention mechanism, but also realizes adaptive feature selection based on credibility.

[0112] It should be noted that the reliability coefficient is introduced before the softmax operation, so that the weight suppression effect can affect model training through gradient propagation.

[0113] ​​In step S33, the multi-modal feature data is weighted and fused based on the attention weight of uncertainty perception, the fused feature vector is input into the classifier, and the entry state evaluation result and risk level of the aircraft are output; wherein the risk level is determined based on the comparison result of the probability distribution output by the classifier and the preset threshold.

[0114] In this step, the whole process of feature fusion and decision output is completed based on the uncertainty perception attention weight obtained in the foregoing steps, and specifically includes:

[0115] First, the reliability weighted fusion calculation of the multi-modal feature data is performed by using the attention weight, and a fused feature vector is obtained :

[0116] ;

[0117] wherein, is a value vector of the i-th modal feature data, is the uncertainty perception attention weight. The calculation can ensure that the feature data of the high-reliability modal occupies a dominant position in the fusion result. Then, the fused feature vector is input into the classifier for state recognition:

[0118]

[0119] ; wherein, W is a classifier weight matrix, b is a bias term, P is an output probability distribution vector, and each dimension corresponds to the prediction probability of a different entry state (such as “normal”, “left deviation”, “right deviation”, “abnormal speed” and the like).

[0120] Finally, the specific output is determined based on the probability distribution:

[0121] The class with the maximum probability is taken as the final state judgment, that is, state=arg max(P).

[0122] And, the risk level is divided according to the highest prediction probability value

[0123]

[0124] ; wherein,

[0125] and are preset risk thresholds. Please refer to

[0126] , the embodiment of the present application also provides a civil aviation aircraft entry guarantee multi-modal intelligent identification linkage device 200, the device comprises a multi-modal acquisition unit 201, an uncertainty measurement unit 202, a decision unit 203, a linkage scheduling unit 204; Figure 3 ​

[0127] The multi-modal acquisition unit 201 is configured to collect multi-modal data including stand video data, aircraft operation state data and support node state data during the civil aviation aircraft standing phase, clean, time synchronize and feature extract the multi-modal data to obtain standardized multi-modal feature data;

[0128] The uncertainty measurement unit 202 is configured to construct a graph model based on the spatio-temporal logical relationship between each modal feature data, detect the confidence conflict between nodes in the graph, and monitor the distribution drift of each modal feature data, and calculate the comprehensive uncertainty measure of each modal feature data based on the confidence conflict and the distribution drift;

[0129] The decision unit 203 is configured to input the standardized multi-modal feature data and the comprehensive uncertainty measure into a multi-modal fusion decision model, and in the cross-modal attention calculation process of the multi-modal fusion decision model, the attention weight of high uncertainty modal feature data is dynamically suppressed by using each comprehensive uncertainty measure, to generate the standing state evaluation result and risk level of the aircraft;

[0130] The linkage scheduling unit 204 is configured to decide to execute support resource linkage scheduling according to the standing state evaluation result and risk level, including: when the evaluation result is left or right trajectory deviation, sending a trajectory adjustment prompt to the ground guide; when the evaluation result is that the bridge is not in place, scheduling the bridge operator and suspending the approach of the luggage trailer; when the evaluation result is speed anomaly, sending an alarm to the ground guide and the tower to prompt the aircraft to slow down.

[0131] As an example, the uncertainty measurement unit 202 is specifically configured to:

[0132] A dynamic evolving spatio-temporal heterogeneous graph model is constructed with the aircraft, each support device and key infrastructure as nodes, and the spatial proximity, physical connection relationship and support process dependency relationship between nodes as edges;

[0133] The graph attention network aggregates the state information of each node and its connected neighbors to generate an enhanced state representation for each node containing neighborhood context information; based on the enhanced state representation, the confidence distribution of each node's own state is generated, and the KL divergence between the confidence distributions of the node pairs directly connected by the edges is calculated; for each modal feature data, the confidence conflict measure is obtained based on the calculated KL divergence of all node pairs associated with it;

[0134] For each modality feature data, a high-dimensional feature is extracted through a sliding window, and a maximum mean difference between the high-dimensional feature distribution in the window and a benchmark distribution in a training period is calculated, and the maximum mean difference is taken as a distribution drift measure of the modality feature data; wherein the training period refers to a training period of a model used to extract corresponding modality feature data.

[0135] As an example, the uncertainty measurement unit 202 is specifically configured to:

[0136] For each modality feature data, the corresponding confidence conflict measure and the distribution drift measure are weighted and fused to obtain the corresponding comprehensive uncertainty measure; wherein the weight of the confidence conflict measure is dynamically allocated according to the degree centrality of the node corresponding to the modality feature data in the spatio-temporal heterogeneous graph model; and the weight of the distribution drift measure is dynamically allocated according to the historical dependence degree of the modality feature data in the current entry stage.

[0137] As an example, the weight of the confidence conflict measure is dynamically allocated according to the degree centrality of the node corresponding to the modality feature data in the spatio-temporal heterogeneous graph model, including:

[0138] The degree centrality of each node in the spatio-temporal heterogeneous graph model is calculated, and the degree centrality is logarithmically compressed; wherein the degree centrality is obtained by the ratio of the connection degree of the node to the network size;

[0139] Based on the compressed degree centrality value, the initial weight of each node is calculated by a softmax function with a learnable temperature parameter, wherein the temperature parameter controls the concentration degree of the weight distribution;

[0140] Based on the weight smoothing mechanism, the initial weight is linearly interpolated with the uniform distribution weight, the smoothed weight is corrected according to the node type, and the corrected weight is taken as the weight of the confidence conflict measure of each modality feature data.

[0141] As an example, the decision unit 203 is specifically configured to:

[0142] The comprehensive uncertainty measure of each modality feature data is mapped to a reliability coefficient in the interval [0, 1] through a negative exponential function; wherein the reliability coefficient is negatively correlated with the comprehensive uncertainty measure;

[0143] When calculating the cross-modal attention weight, the dot product score of the query vector and the key vector of each modality feature data is multiplied by the reliability coefficient of the corresponding modality feature data, and then normalized by softmax to obtain the uncertainty-aware attention weight;

[0144] The multi-modal feature data is weighted and fused based on the attention weight perceived according to the uncertainty, a feature vector after the fusion is input into a classifier, and an entry state evaluation result and a risk level of the aircraft are output; wherein the risk level is determined based on a comparison result of a probability distribution output by the classifier and a preset threshold.

[0145] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or the like made based on the technical concept of the present application and the content of the specification and drawings are included in the patent protection scope of the present application.

Claims

1. A multimodal intelligent recognition and linkage method for civil aircraft positioning support, characterized in that, The methods and steps include the following: Step S10: Collect multimodal data during the civil aircraft placement phase, including stand video data, aircraft operation status data and support node status data. Clean, synchronize time and extract features from the multimodal data to obtain standardized multimodal feature data. Step S20: Construct a graph model based on the spatiotemporal logical relationship between modal feature data, detect confidence conflicts between nodes in the graph, monitor the distribution drift of each modal feature data, and calculate the comprehensive uncertainty measure of each modal feature data based on the confidence conflicts and the distribution drift. Step S30: Input the standardized multimodal feature data and each comprehensive uncertainty measure into the multimodal fusion decision model. In the cross-modal attention calculation process of the multimodal fusion decision model, use each comprehensive uncertainty measure to dynamically suppress the attention weight of high uncertainty modal feature data, and generate the aircraft's entry status assessment result and risk level. Step S40: Based on the assessment results of the positioning status and the risk level, decide to implement the coordinated scheduling of support resources, including: when the assessment result is that the trajectory is deviated to the left or right, send a trajectory adjustment prompt to the ground crew guide; when the assessment result is that the boarding bridge is not in place, dispatch the boarding bridge operator and suspend the baggage trolley approach; when the assessment result is that the speed is abnormal, send an alarm to the ground crew guide and the control tower to prompt the aircraft to decelerate. A graph model is constructed based on the spatiotemporal logical relationships between modal feature data. Confidence conflicts between nodes in the graph are detected, and the distribution drift of each modal feature data is monitored, including: Step S21: Using aircraft, various support equipment and critical infrastructure as nodes, and the spatial proximity, physical connection relationship and support process dependency relationship between nodes as edges, construct a dynamically evolving spatiotemporal heterogeneous graph model. Step S22: Aggregate the state information of each node and its connected neighbors through a graph attention network to generate an enhanced state representation for each node that includes neighborhood context information; based on the enhanced state representation, generate a confidence distribution of the state of each node, and calculate the KL divergence between the confidence distributions of node pairs directly connected by edges; for each modality feature data, derive the confidence conflict metric based on the KL divergence calculated from all its associated node pairs. Step S23: For each modal feature data, extract its high-dimensional features through a sliding window, and calculate the maximum mean difference between the high-dimensional feature distribution within the window and the baseline distribution during the training period. Use the maximum mean difference as the distribution drift measure of the modal feature data; wherein, the training period refers to the training period of the model used to extract the corresponding modal feature data.

2. The multimodal intelligent identification and linkage method for civil aircraft positioning support according to claim 1, characterized in that: Based on the confidence conflict and the distribution drift, a comprehensive uncertainty measure for each modal feature data is calculated, including: For each modal feature data, the corresponding confidence conflict measure and distribution drift measure are weighted and fused to obtain the corresponding comprehensive uncertainty measure; wherein, the weight of the confidence conflict measure is dynamically allocated according to the degree centrality of the node corresponding to the modal feature data in the spatiotemporal heterogeneous graph model; the weight of the distribution drift measure is dynamically allocated according to the historical dependence of the modal feature data in the current in-position stage.

3. The multimodal intelligent identification and linkage method for civil aircraft positioning support according to claim 2, characterized in that: The weights of the confidence conflict metric are dynamically allocated based on the degree centrality of the node corresponding to the modal feature data in the spatiotemporal heterogeneous graph model, including: Calculate the degree centrality of each node in the spatiotemporal heterogeneous graph model, and perform logarithmic compression on the degree centrality; wherein, the degree centrality is obtained by the ratio of the node's connectivity to the network size; Based on the compressed degree centrality value, the initial weights of each node are calculated using a softmax function with a learnable temperature parameter, wherein the temperature parameter controls the degree of concentration of the weight distribution. The initial weights and uniformly distributed weights are linearly interpolated based on a weight smoothing mechanism. The smoothed weights are then corrected according to the node type, and the corrected weights are used as the weights for the confidence conflict metric of each modality feature data.

4. The multimodal intelligent identification and linkage method for civil aircraft positioning support according to claim 1, characterized in that: In the cross-modal attention calculation process of this multimodal fusion decision model, the attention weights of high-uncertainty modal feature data are dynamically suppressed using the comprehensive uncertainty measures described above, generating the aircraft's arrival status assessment results and risk level, including: Step S31: The comprehensive uncertainty measure of each modal feature data is mapped to a reliability coefficient in the interval [0,1] through a negative exponential function; wherein the reliability coefficient is negatively correlated with the comprehensive uncertainty measure; Step S32: When calculating the cross-modal attention weights, the dot product score of the query vector and the key vector of each modal feature data is multiplied by the reliability coefficient of the corresponding modal feature data, and then softmax normalization is performed to obtain the attention weights for uncertainty perception. Step S33: Based on the attention weights of uncertainty perception, the multimodal feature data is weighted and fused, and the fused feature vector is input into the classifier to output the aircraft's landing status assessment result and risk level; wherein, the risk level is determined based on the comparison result between the probability distribution output by the classifier and the preset threshold.

5. A multimodal intelligent identification and linkage device for civil aircraft positioning support, characterized in that: The device includes a multimodal acquisition unit, an uncertainty measurement unit, a decision-making unit, and a linkage scheduling unit; The multimodal acquisition unit is configured to: collect multimodal data during the civil aircraft's entry phase, including stand video data, aircraft operation status data, and support node status data; clean, synchronize, and extract features from the multimodal data to obtain standardized multimodal feature data. The uncertainty measurement unit is configured to: construct a graph model based on the spatiotemporal logical relationship between modal feature data, detect confidence conflicts between nodes in the graph, monitor the distribution drift of each modal feature data, and calculate the comprehensive uncertainty measure of each modal feature data based on the confidence conflicts and the distribution drift. The decision unit is configured to: input the standardized multimodal feature data and each comprehensive uncertainty measure into the multimodal fusion decision model; during the cross-modal attention calculation process of the multimodal fusion decision model, dynamically suppress the attention weight of high uncertainty modal feature data using each comprehensive uncertainty measure, and generate the aircraft's entry status assessment result and risk level. The linkage scheduling unit is configured to: make decisions on the linkage scheduling of support resources based on the positioning status assessment results and risk level, including: when the assessment result is a left or right deviation of the trajectory, sending a trajectory adjustment prompt to the ground crew guide; when the assessment result is that the boarding bridge is not in place, dispatching the boarding bridge operator and suspending the baggage trolley approach; when the assessment result is an abnormal speed, sending an alarm to the ground crew guide and the control tower to prompt the aircraft to decelerate. The uncertainty measurement unit is specifically configured as follows: Using aircraft, various support equipment and critical infrastructure as nodes, and the spatial proximity, physical connection relationship and support process dependency relationship between nodes as edges, a dynamic spatiotemporal heterogeneous graph model is constructed. The graph attention network aggregates the state information of each node and its connected neighbors to generate an enhanced state representation for each node that includes neighborhood context information. Based on the enhanced state representation, a confidence distribution of the state of each node is generated, and the KL divergence between the node pairs directly connected by edges is calculated. For each modal feature data, the confidence conflict measure is derived based on the KL divergence calculated for all its associated node pairs. For each modal feature data, its high-dimensional features are extracted by a sliding window, and the maximum mean difference between the distribution of high-dimensional features within the window and the baseline distribution during the training period is calculated. The maximum mean difference is used as a measure of the distribution drift of the modal feature data. The training period refers to the training period of the model used to extract the corresponding modal feature data.

Images