Intelligent pilot behavior decision knowledge graph link prediction method and system for carrier-based aircraft landing
By encoding and fusing multimodal features in the time and frequency domains, the problem of missing triples in the multimodal knowledge graph during carrier-based aircraft landing was solved, enabling efficient and reliable prediction of pilot behavior decisions.
Patent Information
- Application Number
- CN202511552221.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-28
- Publication Date
- 2026-02-13
AI Technical Summary
In the process of carrier-based aircraft landing, the lack of triples in the multimodal knowledge graph leads to insufficient reasoning ability of the knowledge graph, and the spatial domain modality fusion method is complex and prone to information loss.
We employ a multimodal feature encoding, single-modal time-frequency knowledge enhancement, and multimodal time-frequency knowledge fusion approach. By using wavelet transform and gated attention mechanism, we extract and fuse features in the time-frequency domain to generate more reliable prediction results.
It improves the quality and fusion efficiency of multimodal information, reduces computational burden, enhances noise resistance and interpretability, and generates more reliable pilot behavior decision prediction results.
Smart Images
Figure CN121525818A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent pilot behavior decision prediction and auxiliary control technology, and more specifically, to an intelligent pilot behavior decision knowledge graph linking prediction method and system for carrier-based aircraft landing. Background Technology
[0002] The glide phase of carrier-based aircraft landing is a typical high-risk, highly non-stationary, and delicately controlled process. Pilots (or intelligent pilots) must comprehensively process multi-source information within an extremely short timeframe: the glide path altitude indicated by the Improved Fresnel Lens Optical Landing System (IFLOLS), the alignment deviation indicated by the lateral alignment reference lights, the angle of attack and sink rate, crosswinds and stern turbulence, the six-degree-of-freedom motion of the carrier deck, HUD / deck video, and LSO commands, to make a safe touchdown or go-around decision. Knowledge Graph (KG) is a technology for organizing and representing knowledge in a structured form. Its core idea is to model entities, concepts, and their relationships in the real world as a graph structure. In knowledge graphs, knowledge is typically stored in the form of triples (head entity-relationship-tail entity). In carrier landing scenarios, flight state / environmental constraints—pilot's corrective intentions—control actions can be modeled as a graph structure, such as <large glide slope deviation, indication, corrective action>, <corrective action, trigger, elevator correction>, etc. This structured representation enables machines to understand and reason about knowledge more efficiently. With the rapid development of artificial intelligence technology, traditional knowledge graphs based on a single text modality are no longer sufficient to meet increasingly complex practical needs. Against this backdrop, multimodal knowledge graphs (MKG), as an emerging knowledge representation paradigm, are gradually becoming a research hotspot. Multimodal knowledge graphs organically integrate multiple data modalities such as text, images, video, and audio to construct a more comprehensive, three-dimensional, and realistic knowledge representation system. The core value of multimodal knowledge graphs lies in their ability to fully leverage the complementary advantages of different modal data. By deeply integrating these different modal data, multimodal knowledge graphs can establish knowledge representation models that are richer and closer to human cognition than single-modal data, making knowledge graphs suitable for more complex application scenarios.
[0003] Due to the complexity of knowledge graph construction, missing triples are a common problem in both traditional and multimodal knowledge graphs. For carrier-based aircraft landing, multimodal knowledge graphs (MKG) further incorporate text / voice (Standard Operating Procedures (SOPs), Landing Signal Officer (LSO) commands), images / video (IFLOLS, Head-up Displays (HUDs), deck cameras), structured flight data, and environmental sensors) to achieve a knowledge representation that more closely resembles the real-world decision-making process. However, due to construction complexity and sensor gaps, incomplete triple / edge information is prevalent. Missing triples affect the reasoning ability of the knowledge graph; therefore, knowledge graph link prediction becomes a key technology for solving this problem. Link prediction aims to infer potentially missing relationships or entities based on existing triples. For example, given "large glide slope deviation, ?, correction," or "longitudinal correction, trigger, ?," the model should be able to infer the missing parts.
[0004] However, most related methods primarily design models in the spatial domain. To further improve model performance across various datasets, fusion methods have become increasingly complex. In the spatial domain, the original signals of different modalities often exhibit high heterogeneity. Direct fusion requires designing complex cross-modal interaction mechanisms, which not only increases computational burden but also easily introduces redundant parameters. Function transformations can simplify operations; for example, time-domain convolution becomes frequency-domain multiplication after Fourier transform. By decomposing the signal into orthogonal frequency (or time-frequency) components in the transform domain, the features of each modality are ultimately transformed into different distribution forms in an orthogonal function space, essentially achieving a unified representation and reconstruction of multimodal data. This unified representation allows the fusion process to bypass the differences between the original modalities and operate directly on frequencies. Furthermore, this fusion method naturally possesses noise robustness advantages because noise is usually concentrated in specific frequency bands. By selecting specific frequency bands for adaptive filtering, interference components can be effectively suppressed, avoiding the additional denoising preprocessing steps typically required by spatial domain methods. However, the simplification of fusion does not mean information loss. The ability of frequency domain analysis to extract the essential features of signals, and the ability of time-frequency domain analysis to preserve spatiotemporal information, make it possible for such methods to fully preserve the key features of the spatial domain and even more explicitly capture the deep physical correlations across modes. Summary of the Invention
[0005] The technical problem to be solved by this invention is:
[0006] To address the issue that spatial domain modal fusion is complex and results in information loss after fusion, we need to consider the following:
[0007] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:
[0008] This invention provides a knowledge graph linking and prediction method for intelligent pilot behavior decisions for carrier-based aircraft landings, comprising the following steps:
[0009] S100, Multimodal Feature Encoding, for multi-source data during the landing and glide phase, extracts multimodal knowledge representations in the spatial domain through a modal encoder;
[0010] This includes extracting modal features during the landing phase, encoding them to obtain modal embeddings, and then projecting them into a unified embedding space;
[0011] S200, single-mode time-frequency knowledge enhancement, adaptive filtering of modal time-frequency information, used to suppress intra-modal components and noise unrelated to the landing mission;
[0012] This includes extracting high-frequency and low-frequency features of a single mode through wavelet transform, then performing single-mode adaptive filtering to obtain the low-frequency and high-frequency components of the filtered mode features, and obtaining the enhanced single-mode features in the spatial domain through inverse wavelet transform of the single mode.
[0013] S300, multimodal time-frequency knowledge fusion, based on the multimodal time-frequency features extracted by wavelet transform, realizes the interaction and weighted fusion of modal time-frequency information;
[0014] This includes extracting high-frequency and low-frequency features of individual modes again through wavelet transform, dynamically generating weights for each mode through a gated attention mechanism, performing multimodal feature interaction and fusion in the time-frequency domain to form fused time-frequency domain features, and performing inverse wavelet transform on the fused time-frequency domain features to obtain fused spatial domain features.
[0015] S400, decision fusion, based on the fusion mode and single-mode spatial domain embedding representation reconstructed by inverse discrete wavelet transform, generates the final state-action link score and outputs the landing control behavior decision and its confidence level.
[0016] Further, in step S100, the following is included:
[0017] S110. Extract modal features for the landing phase. The modal features include text, images / videos, time-series data, and knowledge graph structures. The text includes landing procedures / LSO commands and mission rules. The images / videos include IFLOLS, deck camera, and HUD images. The time-series data includes flight parameters and time-series data from environmental sensors. The graph structures include entities and relationships related to flight state, intent, action, and constraints.
[0018] For the textual description information of entities, a pre-trained BERT model is used for encoding to obtain the entity sentence-level embedding representation. For the image information of entities, a pre-trained ResNet50 model is used for encoding to obtain the entity image embedding representation. The GAT model is used to model the structural information of the multimodal knowledge graph, and the output of the last GAT layer is used as the structural embedding of the entity. ;in, The dimension for text embedding; The dimension for image embedding; For the dimension of structural embedding;
[0019] S120. After obtaining the specific modal embedding using the encoder in step S110, the entity multimodal features are... Projected into a unified embedded space :
[0020] (1)
[0021] Among them, modality s represents graph structure modality, v represents image / video modality, and t represents text modality; and These are all projection layer parameters.
[0022] Further, in step S200, the following are included:
[0023] S210. Extract high-frequency and low-frequency features of a single mode using wavelet transform;
[0024] Using discrete wavelet transform to extract multimodal features By adjusting the filter with low-pass characteristics With wavelet filters that have high-pass characteristics Decomposition yields low-frequency... With high frequency :
[0025]
[0026]
[0027] Where n represents the sample index of the output subband after downsampling, j represents the subscript of the filter coefficients during convolution, and J represents the length of the filter;
[0028] S220. Based on the mission relevance of the landing phase, perform single-mode adaptive filtering. The single-mode adaptive filtering includes a gating network and an expert network, which is to analyze the low-frequency components of the modal features. With high frequency components Through a gated network, the weight vectors assigned to the expert in mode m are as follows: and :
[0029] (4)
[0030] (5)
[0031] Right now,
[0032] (6)
[0033] (7)
[0034] in, Let E be the weight matrix of the trainable gated network, and E be the number of experts.
[0035] Using weight vectors and The feature representations obtained from different expert models are aggregated, with each expert network being a 1×1 convolutional neural network, to obtain the low-frequency components of the filtered modal features. With high frequency components :
[0036] (8)
[0037] (9)
[0038] Wherein, the low-frequency component weight vector assigned to the e-th expert under mode m is: The high-frequency component weight vector assigned to the e-th expert in mode m is: , This represents the e-th expert network. ;
[0039] S230. The lifted single-mode features in the spatial domain are obtained by inverse wavelet transform of a single mode.
[0040] For the multi-scale time-frequency information after single-mode filtering, the enhanced single-mode embedding representation is obtained through inverse discrete wavelet transform. :
[0041]
[0042] (10)
[0043] in, and These are the adjusted reconstruction filter and the wavelet reconstruction filter in IDWT, respectively; IDWT stands for Discrete Wavelet Inverse Transform.
[0044] Further, in step S300, the following are included:
[0045] S310. Extract high-frequency and low-frequency features of a single mode using wavelet transform:
[0046] (11)
[0047] (12)
[0048] S320. Through a gated attention mechanism, weights are dynamically generated for each modality, enabling the graph attention network model to adaptively adjust the contribution of different modal features to the current decision at the sample level, resulting in a gated score. As shown in formula (13):
[0049] (13)
[0050] in, , , These represent the weight matrices for the structural, image, and text modalities, respectively. For bias, For activation functions; , , These represent structural, image, and text modal features, respectively.
[0051] Gating score Divide into three parts, and assign different weight features to each modality. , , It is used to dynamically adjust multimodal time-frequency information at different scales;
[0052] S330. Perform multimodal feature interaction and fusion in the time-frequency domain to obtain the fused high-frequency components. with low-frequency components :
[0053] (14)
[0054] (15)
[0055] Where i represents the position index in the vector;
[0056] Adaptive filtering operations are performed on the fused low-frequency components and the fused high-frequency components using one-dimensional convolution:
[0057] (16)
[0058] (17)
[0059] S340. Perform inverse wavelet transform on the fused time-frequency domain features to obtain the fused spatial domain features, i.e., multimodal fusion knowledge representation. :
[0060]
[0061] (18)
[0062] in, and These are the adjustment reconstruction filter and the wavelet reconstruction filter in the inverse discrete wavelet transform, respectively.
[0063] Further, in step S400, the following is included:
[0064] S410. Perform link prediction on state-action edges in the knowledge graph, and calculate the prediction loss based on the fused modality and the embedding of each single modality respectively; that is, embed the entity representation of each modality. Relational Embedding Representation The input is fed into a shared decoder to generate prediction results for each modality. , Simultaneously, multimodal fusion is embedded in the representation. Relational Embedding Representation The input is a shared decoder, which generates fused modality prediction results. Comprehensive forecast results for:
[0065] (19)
[0066] S420. The total loss is calculated by weighted summation of the losses of each modality and the fused modality. The overall training objective is to minimize this loss. The classification cross-entropy loss function is used to measure the difference between the predicted probability and the true label:
[0067] (20)
[0068] in, and This represents the binary cross-entropy loss function for single-mode and fused-mode operations.
[0069] A knowledge graph linking and prediction system for intelligent pilot behavior decisions for carrier-based aircraft landings. This system has program modules corresponding to the above steps, and executes the steps in the above-described method for intelligent pilot behavior decisions for carrier-based aircraft landings during runtime.
[0070] A computer-readable storage medium storing a computer program configured to, when invoked by a processor, implement steps of a knowledge graph-linked prediction method for intelligent pilot behavior decisions for carrier-based aircraft landings.
[0071] Compared with the prior art, the beneficial effects of the present invention are:
[0072] This invention presents a multimodal link prediction and decision-making method for carrier-based aircraft landings. This method performs single-modal adaptive filtering enhancement and lightweight cross-modal fusion in the wavelet time-frequency domain, including adaptive time-frequency feature filtering and feature fusion based on discrete wavelet transform. The adaptive time-frequency feature filtering is used to process single-modal information, improving the quality of modal information through adaptive filtering. The feature fusion based on discrete wavelet transform is used to efficiently fuse information from multiple modalities to capture complementary information between different modalities, generating more reliable prediction results. Furthermore, the feature fusion based on discrete wavelet transform has excellent modal scalability, easily fusing modal data other than structure, text, and images without significant model modifications, and performing interpretable link prediction on a knowledge graph to output state-action decisions and risk assessments. This method exhibits high modal scalability and interpretability for text / speech, image / video, structured flight data, and environmental sensors. Extensive experiments on multiple batches of high-fidelity carrier landing simulator landing datasets demonstrate that this method outperforms existing methods in terms of link prediction accuracy, noise robustness, and online inference efficiency. The modal interface of this method possesses good scalability while maintaining low fusion complexity and clear interpretability, effectively verifying the effectiveness and deployment feasibility of this invention. Attached Figure Description
[0073] Figure 1 This is a flowchart of the knowledge graph linking and prediction method for intelligent pilot behavior decision-making for carrier-based aircraft landing, as described in this embodiment of the invention.
[0074] Figure 2 This is a schematic diagram of single-mode time-frequency knowledge enhancement in an embodiment of the present invention;
[0075] Figure 3 This is a schematic diagram of multimodal time-frequency knowledge fusion in an embodiment of the present invention;
[0076] Figure 4 This is a schematic diagram of decision fusion in an embodiment of the present invention. Detailed Implementation
[0077] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0078] Specific Implementation Plan 1: Combining Figures 1 to 4 As shown, this invention provides a knowledge graph link prediction method for intelligent pilot behavior decision-making for carrier-based aircraft landing, comprising the following steps:
[0079] S100, Multimodal Feature Encoding, targets multi-source data during the landing and descent phase, extracting multimodal knowledge representations in the spatial domain using a specific modality encoder; including,
[0080] S110. Extract text, images / videos, time-series data, and knowledge graph structure modal features for the landing phase. The text includes landing procedures / LSO commands and mission rules. The images / videos include IFLOLS, deck camera, and HUD images. The time-series data includes flight parameters and time-series data from environmental sensors. The graph structure includes entities and relationships of "flight state - intent - action - constraint".
[0081] For textual descriptions, a pre-trained BERT model is used for encoding to obtain sentence-level embeddings of entities. For the image information of entities, a pre-trained ResNet50 model is used for encoding to obtain the entity image embedding representation. The Graph Attention Network (GAT) model is used to model the structural information of the multimodal knowledge graph, and the output of the last GAT layer is used as the structural embedding of the entity. ;in, The dimension of the text embedding (the feature dimension of the BERT output); The dimension of the image embedding (the feature dimension of the ResNet50 output); The dimension of the structural embedding (the feature dimension of the GAT layer output).
[0082] S120. After obtaining the specific modal embedding using the encoder described above, the entity multimodal features are... Projected into a unified embedded space , as in the formula As shown:
[0083] (1)
[0084] Among them, modality , and All are projection layer parameters; s represents the graph structure modality, v represents the image / video modality, and t represents the text modality;
[0085] S200, single-mode time-frequency knowledge enhancement, adaptive filtering of modal time-frequency information to suppress intra-modal components and noise unrelated to the landing mission; including,
[0086] S210. Extract high-frequency and low-frequency features of a single mode using wavelet transform;
[0087] Because wavelet transform has excellent time-frequency local analysis properties, it can obtain high-frequency and low-frequency information at different scales. Discrete Wavelet Transform (DWT) can be used to analyze multimodal features. By adjusting the filter with low-pass characteristics With wavelet filters that have high-pass characteristics Decomposition yields low-frequency... With high frequency , as in the formula and formula As shown:
[0088]
[0089]
[0090] Where n represents the sample index of the output subband after downsampling, j represents the subscript of the filter coefficients during convolution, and J represents the length of the filter;
[0091] S220 performs single-mode adaptive filtering based on the mission relevance of the landing phase, enhancing the characterization of control-related features, including glide slope deviation convergence, angle of attack stability, lateral centering, and deck motion compensation, and suppressing irrelevant disturbances and drifts introduced by the wake and deck motion.
[0092] Single-mode adaptive filtering includes a gating network and an expert network, including the filtering of low-frequency components of the modal features. With high frequency components Through a gated network, the weight vectors assigned to the expert in mode m are as follows: and As shown in formulas (4) and (5):
[0093] (4)
[0094] (5)
[0095] Specifically, this can be expressed as formulas (6) and (7):
[0096] (6)
[0097] (7)
[0098] in, Let E be the weight matrix of the trainable gated network, and E be the number of experts.
[0099] Using weight vectors and The feature representations obtained from different expert models are aggregated, with each expert network being a 1×1 convolutional neural network, to obtain the low-frequency components of the filtered modal features. With high frequency components As shown in formulas (8) and (9):
[0100] (8)
[0101] (9)
[0102] Wherein, the low-frequency component weight vector assigned to the e-th expert under mode m is: The high-frequency component weight vector assigned to the e-th expert in mode m is: , This represents the e-th expert network. ;
[0103] S230. The lifted single-mode features in the spatial domain are obtained by inverse wavelet transform of a single mode.
[0104] For the multi-scale time-frequency information after single-mode filtering, the enhanced single-mode embedding representation is further obtained through inverse discrete wavelet transform. As shown in formula (10):
[0105]
[0106] (10)
[0107] in, and These are the adjusted reconstruction filter and the wavelet reconstruction filter in IDWT, respectively; IDWT stands for Inverse Discrete Wavelet Transform.
[0108] S300, multimodal time-frequency knowledge fusion, based on multimodal time-frequency features extracted by wavelet transform, realizes interactive and weighted fusion of modal time-frequency information; including,
[0109] S310. Extract high-frequency and low-frequency features of a single mode using wavelet transform;
[0110] Because wavelet transform has excellent time-frequency local analysis properties, it can obtain high-frequency and low-frequency information at different scales. Discrete Wavelet Transform (DWT) can be used to analyze multimodal features. By adjusting the filter with low-pass characteristics With wavelet filters that have high-pass characteristics Decomposition yields low-frequency components With high frequency components As shown in formulas (11) and (12):
[0111] (11)
[0112] (12)
[0113] S320. By using a gating attention mechanism, weights are dynamically generated for each modality, enabling the graph attention network model to adaptively adjust the contribution of different modal features to the current decision at the sample level.
[0114] Considering the impact of modal information imbalance on the quality of multimodal embedding representations, a gated attention mechanism is used to adaptively adjust the contribution of different modal features, resulting in a gated score. As shown in formula (13):
[0115] (13)
[0116] in, , , These represent the weight matrices for the structural, image, and text modalities, respectively. For bias, For activation functions; , , These represent structural, image, and text modal features, respectively.
[0117] Gating score Divide into three parts, and assign different weight features to each modality. , , It is used to dynamically adjust multimodal time-frequency information at different scales;
[0118] S330. Perform multimodal feature interaction and fusion in the time-frequency domain to form fused time-frequency domain features;
[0119] Compared to complex modal fusion methods in the spatial domain, this invention achieves full interaction of the time-frequency features of the three modes in the high-frequency and low-frequency subbands through a maximum absolute value selection mechanism, thereby obtaining the fused high-frequency component. with low-frequency components As shown in formulas (14) and (15):
[0120] (14)
[0121] (15)
[0122] Where i represents the position index in the vector;
[0123] To effectively suppress redundant noise generated during the fusion process, adaptive filtering operations are performed on the low-frequency and high-frequency components of the fusion process using one-dimensional convolution, as shown in equations (16) and (17):
[0124] (16)
[0125] (17)
[0126] S340. Perform inverse wavelet transform on the fused time-frequency domain features to obtain the fused spatial domain features, which are used for downstream link prediction and decision output.
[0127] The filtered high-frequency fusion features and low-frequency fusion features are fused and reconstructed using inverse discrete wavelet transform to obtain a multimodal fusion knowledge representation. The specific operation is shown in formula (18):
[0128]
[0129] (18)
[0130] in, and These are the adjustment reconstruction filter and the wavelet reconstruction filter in the inverse discrete wavelet transform, respectively.
[0131] S400, decision fusion, based on the fusion mode and single-mode spatial domain embedding representation reconstructed by inverse discrete wavelet transform, generates the final state-action link score and outputs the landing control behavior decision and its confidence level; including,
[0132] S410. Perform link prediction on the state-action edge in the knowledge graph, and calculate the prediction loss based on the fused modality and the embedding of each single modality respectively; the action is a continuous control quantity (including throttle, elevator, rudder, side stick offset) and a discrete behavior category (including maintain glide, strengthen centering, correct deviation, go-around), and output the corresponding link score and confidence / risk assessment.
[0133] Decision fusion generates more reliable predictions by capturing complementary information from different modalities and their fusion modalities; specifically, it embeds entities from each modality into their representations. Relational Embedding Representation The input is fed into a shared decoder to generate prediction results for each modality. , Simultaneously, multimodal fusion is embedded in the representation. Relational Embedding Representation The input is a shared decoder, which generates fused modality prediction results. Comprehensive forecast results As shown in formula (19):
[0134] (19)
[0135] S420: The total loss is obtained by weighted summation of the losses of each modality and the fusion modality, and used for end-to-end training to improve the accuracy and robustness of intelligent pilot behavior decision prediction during the landing glide phase.
[0136] The overall training objective is to minimize the loss. The classification cross-entropy loss function is used to measure the difference between the predicted probability and the true label. The calculation formula is shown in (20):
[0137] (20)
[0138] in, and This represents the binary cross-entropy loss function for single-mode and fused-mode operations.
[0139] Specific Implementation Scheme 2: The present invention provides an intelligent pilot behavior decision-making knowledge graph linking prediction system for carrier-based aircraft landing. This system has program modules corresponding to the above steps, and executes the steps in the above-mentioned intelligent pilot behavior decision-making knowledge graph linking prediction method for carrier-based aircraft landing when running.
[0140] The other combinations and connections in this implementation scheme are the same as in Specific Implementation Scheme 1.
[0141] Specific Implementation Scheme 3: The present invention provides a computer-readable storage medium storing a computer program configured to implement, when called by a processor, the steps of a knowledge graph linking and prediction method for intelligent pilot behavior decision-making for carrier-based aircraft landing.
[0142] The other combinations and connections in this implementation scheme are the same as in Specific Implementation Scheme 1.
[0143] Simulation Experiment
[0144] The proposed method and existing models were processed on the DB15K and MKG-W datasets, and the results are shown in Table 1. MRR, Hits@1, Hits@3, and Hits@10 are the evaluation metrics. As can be seen from Table 1, the proposed method outperforms existing methods in all of the above metrics.
[0145] Table 1. Comparison of the effects of different methods and the present invention
[0146]
[0147] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.
Claims
1. A knowledge graph linking and prediction method for intelligent pilot behavior decision-making for carrier-based aircraft landing, characterized in that, Includes the following steps: S100, Multimodal Feature Encoding, for multi-source data during the landing and glide phase, extracts multimodal knowledge representations in the spatial domain through a modal encoder; This includes extracting modal features during the landing phase, encoding them to obtain modal embeddings, and then projecting them into a unified embedding space; S200, single-mode time-frequency knowledge enhancement, adaptive filtering of modal time-frequency information, used to suppress intra-modal components and noise unrelated to the landing mission; This includes extracting high-frequency and low-frequency features of a single mode through wavelet transform, then performing single-mode adaptive filtering to obtain the low-frequency and high-frequency components of the filtered mode features, and obtaining the enhanced single-mode features in the spatial domain through inverse wavelet transform of the single mode. S300, multimodal time-frequency knowledge fusion, based on the multimodal time-frequency features extracted by wavelet transform, realizes the interaction and weighted fusion of modal time-frequency information; This includes extracting high-frequency and low-frequency features of individual modes again through wavelet transform, dynamically generating weights for each mode through a gated attention mechanism, performing multimodal feature interaction and fusion in the time-frequency domain to form fused time-frequency domain features, and performing inverse wavelet transform on the fused time-frequency domain features to obtain fused spatial domain features. S400, decision fusion, based on the fusion mode and single-mode spatial domain embedding representation reconstructed by inverse discrete wavelet transform, generates the final state-action link score and outputs the landing control behavior decision and its confidence level.
2. The intelligent pilot behavior decision-making knowledge graph linking prediction method for carrier-based aircraft landing as described in claim 1, characterized in that: In step S100, the following are included: S110. Extract modal features for the landing phase. The modal features include text, images / videos, time-series data, and knowledge graph structures. The text includes landing procedures / LSO commands and mission rules. The images / videos include IFLOLS, deck camera, and HUD images. The time-series data includes flight parameters and time-series data from environmental sensors. The graph structures include entities and relationships related to flight state, intent, action, and constraints. For the textual description information of entities, a pre-trained BERT model is used for encoding to obtain the entity sentence-level embedding representation. For the image information of entities, a pre-trained ResNet50 model is used for encoding to obtain the entity image embedding representation. The GAT model is used to model the structural information of the multimodal knowledge graph, and the output of the last GAT layer is used as the structural embedding of the entity. ;in, The dimension for text embedding; The dimension for image embedding; For the dimension of structural embedding; S120. After obtaining the specific modal embedding using the encoder in step S110, the entity multimodal features are... Projected into a unified embedded space : (1) Among them, modality s represents graph structure modality, v represents image / video modality, and t represents text modality; and These are all projection layer parameters.
3. The intelligent pilot behavior decision-making knowledge graph linking prediction method for carrier-based aircraft landing as described in claim 2, characterized in that: Step S200 includes, S210. Extract high-frequency and low-frequency features of a single mode using wavelet transform; Using discrete wavelet transform to extract multimodal features By adjusting the filter with low-pass characteristics With wavelet filters that have high-pass characteristics Decomposition yields low-frequency... With high frequency : Where n represents the sample index of the output subband after downsampling, j represents the subscript of the filter coefficients during convolution, and J represents the length of the filter; S220. Based on the mission relevance of the landing phase, perform single-mode adaptive filtering. The single-mode adaptive filtering includes a gating network and an expert network, which is to analyze the low-frequency components of the modal features. With high frequency components Through a gated network, the weight vectors assigned to the expert in mode m are as follows: and : (4) (5) Right now, (6) (7) in, Let E be the weight matrix of the trainable gated network, and E be the number of experts. Using weight vectors and The feature representations obtained from different expert models are aggregated, with each expert network being a 1×1 convolutional neural network, to obtain the low-frequency components of the filtered modal features. With high frequency components : (8) (9) Wherein, the low-frequency component weight vector assigned to the e-th expert under mode m is: The high-frequency component weight vector assigned to the e-th expert in mode m is: , This represents the e-th expert network. ; S230. The lifted single-mode features in the spatial domain are obtained by inverse wavelet transform of a single mode. For the multi-scale time-frequency information after single-mode filtering, the enhanced single-mode embedding representation is obtained through inverse discrete wavelet transform. : in, and These are the adjusted reconstruction filter and the wavelet reconstruction filter in IDWT, respectively; IDWT stands for Discrete Wavelet Inverse Transform.
4. The intelligent pilot behavior decision-making knowledge graph linking prediction method for carrier-based aircraft landing as described in claim 3, characterized in that: Step S300 includes, S310. Extract high-frequency and low-frequency features of a single mode using wavelet transform: (11) (12) S320. Through a gated attention mechanism, weights are dynamically generated for each modality, enabling the graph attention network model to adaptively adjust the contribution of different modal features to the current decision at the sample level, resulting in a gated score. As shown in formula (13): (13) in, , , These represent the weight matrices for the structural, image, and text modalities, respectively. For bias, For activation functions; , , These represent structural, image, and text modal features, respectively. Gating score Divide into three parts, and assign different weight features to each modality. , , It is used to dynamically adjust multimodal time-frequency information at different scales; S330. Perform multimodal feature interaction and fusion in the time-frequency domain to obtain the fused high-frequency components. with low-frequency components : (14) (15) Where i represents the position index in the vector; Adaptive filtering operations are performed on the fused low-frequency components and the fused high-frequency components using one-dimensional convolution: (16) (17) S340. Perform inverse wavelet transform on the fused time-frequency domain features to obtain the fused spatial domain features, i.e., multimodal fusion knowledge representation. : in, and These are the adjustment reconstruction filter and the wavelet reconstruction filter in the inverse discrete wavelet transform, respectively.
5. The intelligent pilot behavior decision-making knowledge graph linking prediction method for carrier-based aircraft landing as described in claim 4, characterized in that: In step S400, the following are included: S410. Perform link prediction on state-action edges in the knowledge graph, and calculate the prediction loss based on the fused modality and the embedding of each single modality respectively; that is, embed the entity representation of each modality. Relational Embedding Representation The input is fed into a shared decoder to generate prediction results for each modality. , Simultaneously, multimodal fusion is embedded in the representation. Relational Embedding Representation The input is a shared decoder, which generates fused modality prediction results. Comprehensive forecast results for: (19) S420. The total loss is calculated by weighted summation of the losses of each modality and the fused modality. The overall training objective is to minimize this loss. The classification cross-entropy loss function is used to measure the difference between the predicted probability and the true label: (20) in, and This represents the binary cross-entropy loss function for single-mode and fused-mode operations.
6. A knowledge graph-linked prediction system for intelligent pilot behavior decisions for carrier-based aircraft landing, characterized by: The system has a program module corresponding to the steps described in any one of claims 1-5 above, and executes the steps in the above-described intelligent pilot behavior decision-making knowledge graph linking prediction method for carrier-based aircraft landing when it is run.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to, when invoked by a processor, implement the steps of the intelligent pilot behavior decision-making knowledge graph linking prediction method for carrier-based aircraft landing as described in any one of claims 1-5.