Air traffic control monitoring method and system based on multi-modal data processing

Abstract

The application discloses an air traffic control monitoring method and system based on multi-modal data processing, and comprises the following steps: acquiring multi-modal air traffic control monitoring data in real time; preprocessing the multi-modal air traffic control monitoring data, and determining preprocessed multi-modal air traffic control monitoring data; fusing the preprocessed multi-modal air traffic control monitoring data, and determining multi-modal fusion data; taking the multi-modal fusion data as air traffic control monitoring results, and performing visual display; through integration of multi-modal data, more comprehensive and rich air traffic information can be acquired, information loss or deviation possibly existing in a single data source is avoided, and a more reliable basis is provided for air traffic control monitoring.

Description

Technical Field

[0001] This invention relates to the field of air traffic control surveillance data processing, and in particular to an air traffic control surveillance method and system based on multimodal data processing. Background Technology

[0002] In recent years, the global air transport industry has flourished, with a continuous increase in the number of flights and increasingly dense air traffic, posing serious challenges to the safety and efficiency of air traffic management (ATM) systems. As a core component of the ATM system, air traffic control surveillance technology provides controllers with decision-making support by acquiring and processing real-time dynamic information about aircraft, ensuring flight safety. However, existing air traffic control surveillance technologies still have many limitations and are insufficient to meet the increasingly complex air traffic control needs.

[0003] Current mainstream air traffic control surveillance methods, such as radar surveillance and Automatic Dependent Surveillance-Broadcast (ADS-B), while providing basic information such as aircraft position and speed, each have their own limitations. Radar surveillance relies on radio wave reflection; in complex terrain (such as mountains and oceans) or severe weather (such as heavy rain and sandstorms), signals are easily interfered with or attenuated, leading to target loss or misjudgment. Secondary radar, although capable of acquiring identification information such as flight numbers, requires the cooperation of the aircraft's transponder and has blind spots when facing non-cooperative targets. ADS-B technology, based on satellite navigation and data link broadcasting, offers advantages such as high precision and low cost, but also faces security risks such as signal obstruction and cyberattacks (such as data forgery). Furthermore, single-modal data cannot comprehensively reflect the aircraft's operational status and surrounding environmental risks.

[0004] Furthermore, existing air traffic control surveillance systems typically process different types of surveillance data independently, lacking the ability to collaboratively analyze multi-source data. For example, radar data and ADS-B data are often displayed separately, requiring controllers to manually integrate the information, which is inefficient and prone to cognitive load. Auxiliary data such as weather radar and optical monitoring are not deeply integrated with core surveillance data, resulting in insufficient early warning capabilities for potential threats such as sudden weather disasters and drone intrusions. At the same time, traditional air traffic control surveillance data display methods are mostly based on two-dimensional maps or lists, with information presented in a scattered manner, making it difficult to intuitively reflect the spatial relationships between aircraft, conflict risks, and environmental connections, thus limiting the situational awareness and decision-making efficiency of controllers.

[0005] In summary, existing air traffic control surveillance technologies are limited by the limitations of single-modal data, the isolated processing of multi-source data, and inefficient data display methods, making it difficult to provide comprehensive and reliable surveillance information in complex scenarios.

[0006] Therefore, there is an urgent need for an air traffic control monitoring method and system based on multimodal data processing to solve the above-mentioned technical problems. Summary of the Invention

[0007] The present invention aims to at least partially solve one of the technical problems in the aforementioned technologies. Therefore, a first aspect of the present invention aims to propose an air traffic control surveillance method based on multimodal data processing. By integrating multimodal data, more comprehensive and richer air traffic information can be obtained, avoiding information gaps or biases that may exist with a single data source, and providing a more reliable basis for air traffic control surveillance.

[0008] A second objective of this invention is to provide an air traffic control monitoring system based on multimodal data processing.

[0009] To achieve the above objectives, a first aspect of the present invention proposes an air traffic control monitoring method based on multimodal data processing, comprising:

[0010] Real-time acquisition of multimodal air traffic control monitoring data;

[0011] The multimodal air traffic control monitoring data is preprocessed to determine the preprocessed multimodal air traffic control monitoring data;

[0012] The preprocessed multimodal air traffic control monitoring data are fused to determine the multimodal fused data;

[0013] The multimodal fusion data is used as the result of air traffic control monitoring and then visualized.

[0014] Preferably, real-time acquisition of multimodal air traffic control monitoring data includes:

[0015] Based on the mechanical scanning system of the secondary air traffic control radar, the monitoring data of the secondary air traffic control radar is acquired, and the first data is determined.

[0016] ADS-B monitoring data is acquired using an ADS-B omnidirectional antenna to determine the second data.

[0017] Based on the BeiDou / GPS / GLONASS three-in-one positioning antenna and the BeiDou antenna, acquire BeiDou surveillance data and determine the third data;

[0018] The fourth data is determined by acquiring real-time operational status data of the aircraft based on the C2 link antenna.

[0019] The first, second, third, and fourth data points are used as multimodal air traffic control monitoring data.

[0020] Preferably, the multimodal air traffic control surveillance data is preprocessed, and the preprocessed multimodal air traffic control surveillance data includes:

[0021] Select any air traffic control monitoring data under any mode as the data to be cleaned;

[0022] Obtain data points from the sequence data of each dimension in the data to be cleaned, and obtain the number of times the sequence data of each dimension is called within a preset time period;

[0023] Select any one dimension of sequence data as the sub-data to be cleaned;

[0024] The first indicator value of the sub-data to be cleaned is determined based on the number of calls;

[0025] Determine the second index value of the sub-data to be cleaned based on the data values ​​of each data point in the sub-data to be cleaned;

[0026] The product of the first index value and the second index value is used as the first data cleaning order index of the sub-data to be cleaned.

[0027] Iterate through all the sub-data to be cleaned and obtain the first data cleaning order index for each sub-data to be cleaned.

[0028] The sum of the first data cleaning order indices of all the sub-data to be cleaned is used as the second data cleaning order index of the data to be cleaned.

[0029] By traversing the multimodal air traffic control monitoring data, the second data cleaning order index of the air traffic control monitoring data under each mode is obtained;

[0030] Sort several second data cleaning order indices in descending order to generate a data cleaning list;

[0031] Obtain data cleaning rules corresponding to different modal air traffic control monitoring data;

[0032] Based on the data cleaning rules and data cleaning lists corresponding to different modes of air traffic control monitoring data, the multimodal air traffic control monitoring data is cleaned to obtain preprocessed multimodal air traffic control monitoring data.

[0033] Preferably, the second index value of the sub-data to be cleaned is determined based on the data values ​​of each data point in the sub-data to be cleaned, including:

[0034] Calculate the mean of the data points in the sub-data to be cleaned, and use it as the target mean.

[0035] Calculate the absolute difference between the data value of each data point in the sub-data to be cleaned and the target mean, and obtain several first absolute differences;

[0036] Calculate the absolute difference between the data value of each data point in the sub-data to be cleaned and the data value of the next adjacent data point to obtain several second absolute differences;

[0037] Calculate the ratio of the first absolute difference to the second absolute difference for each data point, and use it as the noise evaluation value;

[0038] The noise evaluation value is compared with a preset noise evaluation threshold, and data points whose noise evaluation value is greater than or equal to the preset noise evaluation threshold are marked to obtain several marks;

[0039] Calculate the ratio of the total number of several markers to the total number of data points in the sub-data to be cleaned, and determine the second index value of the sub-data to be cleaned.

[0040] Preferably, the preprocessed multimodal air traffic control monitoring data is fused to determine the multimodal fused data, including:

[0041] Select any mode of air traffic control monitoring data as the data to be processed;

[0042] The data to be processed is standardized to obtain standardized data to be processed.

[0043] The dimensionality of the standardized data to be processed is reduced to a preset low-dimensional space to obtain the dimensionality reduction matrix;

[0044] Principal component analysis is performed on the standardized data to be processed based on the dimensionality reduction matrix to obtain the principal component data of the standardized data to be processed.

[0045] The principal component data is segmented based on dynamically adjusting the size of a preset window to obtain several target principal component sub-data.

[0046] Feature extraction is performed on several target principal component subsets to obtain target feature data for several target principal component subsets;

[0047] By traversing all modes of air traffic control surveillance data, target feature data of several target principal component sub-data corresponding to each mode of air traffic control surveillance data are obtained.

[0048] Construct a knowledge graph for target feature recognition and reasoning;

[0049] Based on the target feature recognition reasoning knowledge graph, the association relationship between the target feature data of several target principal component sub-data is identified, and the association features between the target feature data of several target principal component sub-data are determined.

[0050] Based on the correlation features between the target feature data of several target principal component subsets, determine the correlation feature points between the target feature data of several target principal component subsets;

[0051] Based on the associated feature points, the target feature data of several target principal component sub-data are fused to obtain multimodal fused data.

[0052] Preferably, the principal component data is segmented based on dynamically adjusting the size of a preset window to obtain several target principal component sub-data, including:

[0053] Based on a preset window, the principal component data is sequentially pre-segmented from its initial position to obtain several principal component sub-data; the similarity between the first principal component sub-data and the other principal component sub-data besides the first principal component sub-data is calculated to obtain several similarity scores; the set of several similarity scores is taken as the similarity set of the first principal component sub-data.

[0054] The size of the preset window is dynamically adjusted; the principal component data is pre-segmented based on several preset windows of different sizes to obtain the similarity set of the first principal component data corresponding to several preset windows of different sizes;

[0055] The variance of similarity in each similarity set is obtained, and the evaluation value of the preset window under each size is determined to obtain several evaluation values.

[0056] The size of the preset window corresponding to the minimum value among several evaluation values ​​is used as the target window size;

[0057] The principal component data is segmented based on the target window corresponding to the target window size to obtain several target principal component sub-data.

[0058] Preferably, a target feature recognition reasoning knowledge graph is constructed, including:

[0059] Acquire knowledge in the field of air traffic control surveillance;

[0060] A feature recognition knowledge graph is constructed based on the aforementioned knowledge in the field of air traffic control surveillance.

[0061] Obtain a preset inference rule database; the preset inference rule database includes several inference rules;

[0062] Several initial feature recognition reasoning knowledge graphs are generated based on a pre-set reasoning rule database and feature recognition knowledge graph;

[0063] Several initial feature recognition reasoning knowledge graphs are fused to determine the target feature recognition reasoning knowledge graph.

[0064] Preferably, the multimodal fusion data is used as air traffic control monitoring results and visualized, including:

[0065] A visualization platform is built based on coordinate systems and map engines;

[0066] The multimodal fusion data is aligned based on a geographic coordinate system and a time axis;

[0067] The aligned multimodal fusion data is used to populate the visualization platform, and the aligned multimodal fusion data is rendered based on the map engine. The rendering result is used as the air traffic control monitoring result, and the air traffic control monitoring result is visualized.

[0068] Preferably, before using the multimodal fusion data as air traffic control monitoring results and visualizing it, the method further includes:

[0069] The fusion quality of the multimodal fusion data is evaluated, and the fusion evaluation value of the multimodal fusion data is determined.

[0070] The fusion evaluation value is compared with a preset fusion evaluation threshold;

[0071] If the fusion evaluation value is greater than or equal to the preset fusion evaluation threshold, the multimodal fusion data will be visualized.

[0072] If the fusion evaluation value is less than the preset fusion evaluation threshold, the preprocessed multimodal air traffic control monitoring data will be re-fused until the evaluation value is greater than or equal to the preset evaluation threshold.

[0073] To achieve the above objectives, a second aspect of the present invention provides an air traffic control surveillance system based on multimodal data processing, comprising:

[0074] The acquisition module is used to acquire multimodal air traffic control monitoring data in real time.

[0075] The preprocessing module is used to preprocess the multimodal air traffic control monitoring data and determine the preprocessed multimodal air traffic control monitoring data.

[0076] The data fusion module is used to fuse the preprocessed multimodal air traffic control monitoring data to determine the multimodal fused data.

[0077] The visualization module is used to visualize the multimodal fusion data as air traffic control monitoring results.

[0078] This invention provides an air traffic control surveillance method and system based on multimodal data processing. By fusing multimodal data such as radar, ADS-B, BeiDou positioning, and real-time aircraft operational status data, it effectively overcomes the limitations of single data sources. Different modal data have their own advantages in positioning accuracy and status information. Through fusion, high-precision positioning and status awareness can be achieved. Multimodal data fusion provides richer and more comprehensive information for conflict prediction models, enabling more accurate prediction of aircraft movement trends. Integrating multiple modal data enables rapid and accurate identification of emergency situations. Multimodal fused data provides a comprehensive and accurate information foundation, allowing controllers to make more scientific and reasonable decisions. In tasks such as flight planning, airspace resource allocation, and aircraft avoidance, rich data support helps reduce decision-making errors and ensures the safe and orderly operation of air traffic.

[0079] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0080] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0081] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0082] Figure 1 This is a flowchart of an air traffic control monitoring method based on multimodal data processing according to an embodiment of the present invention;

[0083] Figure 2 This is a flowchart illustrating the real-time acquisition of multimodal air traffic control monitoring data according to an embodiment of the present invention;

[0084] Figure 3 This is a block diagram of an air traffic control monitoring system based on multimodal data processing according to an embodiment of the present invention. Detailed Implementation

[0085] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0086] Example 1

[0087] like Figure 1 As shown, an air traffic control monitoring method based on multimodal data processing includes steps S1-S4:

[0088] S1: Real-time acquisition of multimodal air traffic control monitoring data;

[0089] S2: Preprocess the multimodal air traffic control monitoring data to determine the preprocessed multimodal air traffic control monitoring data;

[0090] S3: Fuse the preprocessed multimodal air traffic control monitoring data to determine the multimodal fused data;

[0091] S4: Use the multimodal fusion data as air traffic control monitoring results and visualize it.

[0092] In this embodiment, the multimodal air traffic control surveillance data includes secondary air traffic control radar surveillance data, ADS-B surveillance data, BeiDou surveillance data, and real-time operational status data of the aircraft.

[0093] The working principle of the above technical solution is as follows: by acquiring multimodal air traffic control monitoring data, preprocessing the multimodal air traffic control monitoring data, and determining the preprocessed multimodal air traffic control monitoring data; fusing the preprocessed multimodal air traffic control monitoring data to determine the multimodal fused data; and using the multimodal fused data as the air traffic control monitoring result and displaying it visually.

[0094] The beneficial effects of the above technical solution are as follows: By fusing multimodal data such as radar, ADS-B, BeiDou positioning, and real-time aircraft operational status data, the limitations of a single data source are effectively compensated for. Different modal data have their own advantages in positioning accuracy and status information. Through fusion, high-precision positioning and status awareness can be achieved. Multimodal data fusion provides richer and more comprehensive information for conflict prediction models, enabling more accurate prediction of aircraft movement trends. Integrating multiple modal data enables rapid and accurate identification of emergency situations. Multimodal fused data provides a comprehensive and accurate information foundation, allowing controllers to make more scientific and reasonable decisions. In tasks such as flight planning, airspace resource allocation, and aircraft avoidance, rich data support helps reduce decision-making errors and ensures the safe and orderly operation of air traffic.

[0095] Example 2

[0096] like Figure 2 As shown, real-time acquisition of multimodal air traffic control monitoring data includes steps S11-S15:

[0097] S11: Based on the mechanical scanning system of the secondary air traffic control radar, acquire the surveillance data of the secondary air traffic control radar and determine the first data;

[0098] S12: Acquire ADS-B monitoring data based on the ADS-B omnidirectional antenna and determine the second data;

[0099] S13: Based on the BeiDou / GPS / GLONASS three-in-one positioning antenna and the BeiDou antenna, acquire BeiDou surveillance data and determine the third data;

[0100] S14: Obtain real-time operational status data of the aircraft based on the C2 link antenna to determine the fourth data;

[0101] S15: Use the first data, second data, third data, and fourth data as multimodal air traffic control monitoring data.

[0102] The beneficial effects of the above technical solution are as follows: data from different data sources complement each other. Air traffic control secondary radar has traditional detection advantages, while ADS-B surveillance data provides richer flight status information; combining the two can more accurately determine the aircraft's position, altitude, speed, and other information, reducing positioning errors and thus improving the accuracy of air traffic control surveillance; the fusion of multi-source data enhances the system's fault tolerance; if one data source fails or is interfered with, other data sources can still provide some key information; comprehensive multimodal data allows for a more complete understanding of the aircraft's operational status; BeiDou surveillance data can provide high-precision positioning information, and real-time aircraft operational status data acquired by the C2 link antenna can reflect the operational status of the aircraft's internal systems; integrating these data with air traffic control secondary radar data and ADS-B data allows for understanding the aircraft's status from multiple dimensions, including its external position and internal system status, helping air traffic control departments better predict the aircraft's trajectory and potential problems, thereby enabling them to make reasonable scheduling and command decisions in advance.

[0103] Example 3

[0104] The multimodal air traffic control surveillance data is preprocessed, and the preprocessed multimodal air traffic control surveillance data includes:

[0105] Select any air traffic control monitoring data under any mode as the data to be cleaned;

[0106] Obtain data points from the sequence data of each dimension in the data to be cleaned, and obtain the number of times the sequence data of each dimension is called within a preset time period;

[0107] Select any one dimension of sequence data as the sub-data to be cleaned;

[0108] The first indicator value of the sub-data to be cleaned is determined based on the number of calls;

[0109] Determine the second index value of the sub-data to be cleaned based on the data values ​​of each data point in the sub-data to be cleaned;

[0110] The product of the first index value and the second index value is used as the first data cleaning order index of the sub-data to be cleaned.

[0111] Iterate through all the sub-data to be cleaned and obtain the first data cleaning order index for each sub-data to be cleaned.

[0112] The sum of the first data cleaning order indices of all the sub-data to be cleaned is used as the second data cleaning order index of the data to be cleaned.

[0113] By traversing the multimodal air traffic control monitoring data, the second data cleaning order index of the air traffic control monitoring data under each mode is obtained;

[0114] Sort several second data cleaning order indices in descending order to generate a data cleaning list;

[0115] Obtain data cleaning rules corresponding to different modal air traffic control monitoring data;

[0116] Based on the data cleaning rules and data cleaning lists corresponding to different modes of air traffic control monitoring data, the multimodal air traffic control monitoring data is cleaned to obtain preprocessed multimodal air traffic control monitoring data.

[0117] In this embodiment, the first indicator value is used to determine the importance of the sub-data to be cleaned based on the number of times the sub-data to be cleaned is called, and the indicator value is expressed as a numerical indicator.

[0118] In this embodiment, the second index value is used to judge the abnormal situation of the data points in the sub-data to be cleaned and to express it in an index-based numerical value.

[0119] In this embodiment, obtaining data cleaning rules corresponding to different modal air traffic control monitoring data can be understood as follows: for some modal air traffic control monitoring data values ​​that are relatively sensitive to continuity, numerical data cleaning rules can be obtained for data cleaning.

[0120] The working principle of the above technical solution is as follows: Take any air traffic control monitoring data from one modality as the data to be cleaned. Use the frequency of calls to the sub-data to be cleaned as an evaluation index of data importance and judge the anomalies of data points in the sub-data to be cleaned to determine the data cleaning order of the sub-data to be cleaned, which serves as the data cleaning order within the modality. Use the sum of the first data cleaning order indices of all the sub-data to be cleaned as the second data cleaning order index of the data to be cleaned. Sort several second data cleaning order indices in descending order to generate a data cleaning list. Sort the cleaning list for the entire multimodal data cleaning process to obtain the data cleaning rules corresponding to different modal air traffic control monitoring data. Based on the data cleaning rules and the data cleaning list, perform data cleaning on the multimodal air traffic control monitoring data to obtain preprocessed multimodal air traffic control monitoring data.

[0121] The beneficial effects of the above technical solution are as follows: By analyzing the sequence data of each dimension in the data to be cleaned, including considering the number of calls and the values ​​of data points, it is possible to accurately identify data that may have problems; the data to be cleaned is sorted according to the calculated first data cleaning order index and the second data cleaning order index to generate a data cleaning list; it avoids the waste of resources caused by indiscriminate cleaning, and also makes the cleaning process more efficient, reducing unnecessary calculation and processing steps; since the data cleaning rules corresponding to different modes of air traffic control surveillance data are considered, the characteristics of each mode of data can be fully respected during the cleaning process; for example, different modes of data such as air traffic control secondary radar data, ADS-B data, Beidou surveillance data, and C2 link antenna data may have their own formats, value ranges, and error ranges; this method of cleaning according to the characteristics of different modes of data can better retain the effective information of each mode of data, ensuring that the cleaned data still has usability and accuracy in their respective modes, and improving the adaptability of data to different air traffic control surveillance modes.

[0122] Example 4

[0123] The second indicator value of the sub-data to be cleaned is determined based on the data values ​​of each data point in the sub-data to be cleaned, including:

[0124] Calculate the mean of the data points in the sub-data to be cleaned, and use it as the target mean.

[0125] Calculate the absolute difference between the data value of each data point in the sub-data to be cleaned and the target mean, and obtain several first absolute differences;

[0126] Calculate the absolute difference between the data value of each data point in the sub-data to be cleaned and the data value of the next adjacent data point to obtain several second absolute differences;

[0127] Calculate the ratio of the first absolute difference to the second absolute difference for each data point, and use it as the noise evaluation value;

[0128] The noise evaluation value is compared with a preset noise evaluation threshold, and data points whose noise evaluation value is greater than or equal to the preset noise evaluation threshold are marked to obtain several marks;

[0129] Calculate the ratio of the total number of several markers to the total number of data points in the sub-data to be cleaned, and determine the second index value of the sub-data to be cleaned.

[0130] The working principle of the above technical solution is as follows: By analyzing the absolute difference between the data value of each data point in the sub-data to be cleaned and the target mean, the focus is on observing the fluctuation of the data point's value relative to the target mean; by analyzing the absolute difference between the data value of each data point in the sub-data to be cleaned and the data value of the next adjacent data point, the focus is on observing the fluctuation of the data point's value relative to the next adjacent data point; by calculating the ratio of the first absolute difference to the second absolute difference for each data point, a noise evaluation value is determined. This noise evaluation value is not for noise reduction, but rather reflects abnormal conditions in the sub-data to be cleaned. The overall solution aims to address the order of data cleaning; specific data cleaning will be performed according to the data types of different modalities, selecting appropriate data cleaning rules to complete the data cleaning process.

[0131] The beneficial effects of the above technical solution are as follows: By calculating the ratio of the absolute difference between a data value and the mean, and the absolute difference between the data value and the values ​​of adjacent data points, the noise evaluation value is determined. This method comprehensively considers the deviation of data points from the overall data distribution and the changes between data points. Compared to a single calculation method, it can more comprehensively reflect the noise characteristics of data points, thus more accurately quantifying the noise situation in the data. Comparing the noise evaluation value with a preset noise evaluation threshold and marking the data points can effectively identify those data points that may be abnormal or deviate from the normal data pattern. For example, in air traffic control monitoring data, if the noise evaluation value of a certain data point is high, it may be abnormal data caused by equipment failure, interference, etc. This marking method helps to filter out these abnormal data points for subsequent targeted data cleaning. A second indicator value is determined based on the ratio of the total number of marked data points to the total number of data points, which is then used to determine the data cleaning order. This makes data cleaning more targeted; sub-data points containing more potentially abnormal data points will be prioritized for cleaning or given more attention. This avoids indiscriminate cleaning of the overall data, improves the efficiency and effectiveness of data cleaning, and better preserves valid data.

[0132] Example 5

[0133] The preprocessed multimodal air traffic control monitoring data are fused to determine the multimodal fused data, including:

[0134] Select any mode of air traffic control monitoring data as the data to be processed;

[0135] The data to be processed is standardized to obtain standardized data to be processed.

[0136] The dimensionality of the standardized data to be processed is reduced to a preset low-dimensional space to obtain the dimensionality reduction matrix;

[0137] Principal component analysis is performed on the standardized data to be processed based on the dimensionality reduction matrix to obtain the principal component data of the standardized data to be processed.

[0138] The principal component data is segmented based on dynamically adjusting the size of a preset window to obtain several target principal component sub-data.

[0139] Feature extraction is performed on several target principal component subsets to obtain target feature data for several target principal component subsets;

[0140] By traversing all modes of air traffic control surveillance data, target feature data of several target principal component sub-data corresponding to each mode of air traffic control surveillance data are obtained.

[0141] Construct a knowledge graph for target feature recognition and reasoning;

[0142] Based on the target feature recognition reasoning knowledge graph, the association relationship between the target feature data of several target principal component sub-data is identified, and the association features between the target feature data of several target principal component sub-data are determined.

[0143] Based on the correlation features between the target feature data of several target principal component subsets, determine the correlation feature points between the target feature data of several target principal component subsets;

[0144] Based on the associated feature points, the target feature data of several target principal component sub-data are fused to obtain multimodal fused data.

[0145] In this embodiment, Principal Component Analysis (PCA) is a commonly used dimensionality reduction technique. It projects high-dimensional data into a low-dimensional space through linear transformation while preserving the main information of the data. The core idea of ​​PCA is to find the directions with the largest variance in the data (i.e., the principal components) and project the data onto these directions.

[0146] In this embodiment, the purpose of segmenting the principal component data based on dynamically adjusting the size of the preset window is to segment according to the characteristics of the data as much as possible, thereby ensuring the integrity of the data features. By dynamically adjusting the size of the preset window, the segmentation size of each data is dynamically tested. Two different data may be segmented into different sizes, avoiding the destruction of the data structure caused by the traditional one-size-fits-all approach.

[0147] The working principle and beneficial effects of the above technical solution are as follows: After arbitrarily selecting air traffic control monitoring data of a given mode as the data to be processed, standardization is performed to obtain standardized data. The purpose of this step is to eliminate differences in the dimensions and numerical ranges of different data, enabling subsequent processing to be carried out on the same scale. The standardized data to be processed is then dimensionality-reduced to a preset low-dimensional space to obtain a dimensionality-reduced matrix. Principal component analysis is then performed based on this dimensionality-reduced matrix to obtain principal component data. Dimensionality reduction can reduce data complexity and remove redundant information. Principal component analysis, on the other hand, finds the main component directions of the data and projects the original data onto these principal component directions. The resulting principal component data can reduce the dimensionality of the data while retaining most of the original information. The principal component data is then segmented based on dynamically adjusted preset window sizes to obtain several target principal component sub-data. Feature extraction is then performed on these sub-data to obtain target feature data. Dynamically adjusting the window size can adaptively divide the data into segments according to the characteristics of the data, in order to better capture the features of the data in different local areas. Feature extraction aims to extract representative information from each sub-data set. After traversing all modalities of air traffic control surveillance data and repeating the above steps to obtain target feature data for each modality, a target feature recognition and reasoning knowledge graph is constructed. This knowledge graph describes the logical relationships and hierarchical structure between features of different modalities. Based on the target feature recognition and reasoning knowledge graph, correlation relationships are identified in the target feature data to determine associated features and then associated feature points. Some features in different modalities may be correlated in specific flight phases or airspace areas. Finally, the target feature data is fused based on the associated feature points to obtain multimodal fused data. The fused multimodal data integrates the advantages of each modality, providing more comprehensive and accurate air traffic control surveillance information.

[0148] Example 6

[0149] The principal component data is segmented based on dynamically adjusting the size of a preset window to obtain several target principal component sub-data, including:

[0150] Based on a preset window, the principal component data is sequentially pre-segmented from its initial position to obtain several principal component sub-data; the similarity between the first principal component sub-data and the other principal component sub-data besides the first principal component sub-data is calculated to obtain several similarity scores; the set of several similarity scores is taken as the similarity set of the first principal component sub-data.

[0151] The size of the preset window is dynamically adjusted; the principal component data is pre-segmented based on several preset windows of different sizes to obtain the similarity set of the first principal component data corresponding to several preset windows of different sizes;

[0152] The variance of similarity in each similarity set is obtained, and the evaluation value of the preset window under each size is determined to obtain several evaluation values.

[0153] The size of the preset window corresponding to the minimum value among several evaluation values ​​is used as the target window size;

[0154] The principal component data is segmented based on the target window corresponding to the target window size to obtain several target principal component sub-data.

[0155] In this embodiment, the variance of similarity in each similarity set is obtained, and the variance reflects the degree of dispersion of similarity in each similarity set.

[0156] In this embodiment, as the size of the preset window is adjusted, each size corresponds to a similarity set of the first principal component subdata after pre-segmentation of that size.

[0157] The working principle and beneficial effects of the above technical solution are as follows: Based on a preset window, the principal component data is sequentially pre-segmented from its initial position to obtain several principal component sub-data. This step divides the principal component data into multiple sub-data blocks according to a preset window size. Then, the similarity between the first principal component sub-data and other principal component sub-data is calculated, resulting in several similarity scores that form a similarity set for the first principal component sub-data. Calculating similarity is to measure the degree of similarity between these sub-data, which can reflect the correlation or consistency of data between different segmentation parts. The size of the preset window is dynamically adjusted, and the principal component data is pre-segmented again based on the preset window at different sizes to obtain a similarity set for the first principal component sub-data at each size. Dynamically adjusting the window size is to find a segmentation scale that best suits the data features. Different window sizes will lead to different sub-data segmentation results, thus affecting the similarity between sub-data. This approach allows us to explore the internal structure and correlations of data at different segmentation scales. We obtain the variance of similarity in each similarity set to determine the evaluation value of the preset window at each size. Variance reflects the dispersion of data; here, the variance of similarity in the similarity set measures the fluctuation of similarity between the first principal component data and other data. A smaller variance indicates that the similarity between the first principal component data and other data is relatively stable at that window size, meaning the data segmentation at this scale may be more reasonable. The preset window size corresponding to the minimum of several evaluation values ​​is taken as the target window size, which best maintains stable similarity between data. Finally, the principal component data is segmented based on the target window corresponding to the target window size, resulting in several target principal component data. The target principal component data obtained in this way is optimized in terms of segmentation scale, better reflecting the internal structure and feature relationships of the data, and providing a more reasonable data foundation for subsequent feature extraction, data fusion, and other operations.

[0158] Example 7

[0159] Constructing a knowledge graph for target feature recognition and reasoning, including:

[0160] Acquire knowledge in the field of air traffic control surveillance;

[0161] A feature recognition knowledge graph is constructed based on the aforementioned knowledge in the field of air traffic control surveillance.

[0162] Obtain a preset inference rule database; the preset inference rule database includes several inference rules;

[0163] Several initial feature recognition reasoning knowledge graphs are generated based on a pre-set reasoning rule database and feature recognition knowledge graph;

[0164] Several initial feature recognition reasoning knowledge graphs are fused to determine the target feature recognition reasoning knowledge graph.

[0165] In this embodiment, several initial feature recognition and reasoning knowledge graphs are fused to determine the target feature recognition and reasoning knowledge graph. This includes: first, aligning the entities corresponding to the several initial feature recognition and reasoning knowledge graphs. This can be done based on name matching, using a string similarity algorithm to match entity names; or based on attribute matching, using entity attributes (such as type, description, and context); or based on graph structure matching, using neighbor or path information of entities. Second, defining mapping rules between relationships, which can be done manually or using ontology alignment tools; merging semantically identical relationships into a unified relationship; finally, assigning credibility weights to each knowledge graph, selecting information with higher weights, and selecting the majority vote if multiple knowledge graphs provide conflicting information; for critical conflicts, manual review by domain experts can be conducted; merging the aligned entities and mapped relationships into the target knowledge graph, removing duplicate entities and relationships, and optimizing the graph structure.

[0166] The beneficial effects of the above technical solution are as follows: by acquiring knowledge in the air traffic control surveillance domain to construct a feature recognition knowledge graph, it is possible to ensure that the content of the graph is based on professional knowledge in this domain; by integrating scattered knowledge in the air traffic control surveillance domain into a knowledge graph, a structured knowledge system is formed; the reasoning rules in the pre-set reasoning rule database can add logical reasoning capabilities to the knowledge graph; since there are multiple reasoning rules, multiple initial feature recognition reasoning knowledge graphs can be generated, which increases the flexibility of knowledge graph construction; by fusing several initial feature recognition reasoning knowledge graphs to determine the target feature recognition reasoning knowledge graph, the advantages of each initial graph can be combined.

[0167] Example 8

[0168] The multimodal fusion data is used as air traffic control monitoring results and visualized, including:

[0169] A visualization platform is built based on coordinate systems and map engines;

[0170] The multimodal fusion data is aligned based on a geographic coordinate system and a time axis;

[0171] The aligned multimodal fusion data is used to populate the visualization platform, and the aligned multimodal fusion data is rendered based on the map engine. The rendering result is used as the air traffic control monitoring result, and the air traffic control monitoring result is visualized.

[0172] The beneficial effects of the above technical solution are as follows: The visualization platform built based on a coordinate system and map engine provides an intuitive framework for displaying air traffic control surveillance results. In the field of air traffic control surveillance, the geographic coordinate system can accurately locate the positions of monitored objects such as aircraft, while the map engine can display this location information in an intuitive map format; aligning multimodal fusion data based on the geographic coordinate system and time axis ensures the consistency of data in time and space; filling the visualization platform with the aligned multimodal fusion data and rendering it based on the map engine enables a comprehensive display of multimodal data.

[0173] Example 9

[0174] Before using the multimodal fusion data as air traffic control monitoring results and visualizing it, the following steps are also included:

[0175] The fusion quality of the multimodal fusion data is evaluated, and the fusion evaluation value of the multimodal fusion data is determined.

[0176] The fusion evaluation value is compared with a preset fusion evaluation threshold;

[0177] If the fusion evaluation value is greater than or equal to the preset fusion evaluation threshold, the multimodal fusion data will be visualized.

[0178] If the fusion evaluation value is less than the preset fusion evaluation threshold, the preprocessed multimodal air traffic control monitoring data will be re-fused until the evaluation value is greater than or equal to the preset evaluation threshold.

[0179] In this embodiment, the fusion evaluation algorithm includes:

[0180]

[0181] Where Z represents the fusion evaluation value of the multimodal fusion data; n represents the total number of multimodal fusion data; m represents the number of evaluation indicators in the multimodal fusion data; w i，j f represents the weight value of the j-th evaluation index among the m evaluation indicators corresponding to the i-th multimodal fusion data; i，j Let represent the value of the j-th evaluation index among the m evaluation indicators corresponding to the i-th multimodal fusion data; α represents the adjustment coefficient, α∈[0,1].

[0182] like Figure 3 As shown, to achieve the above objectives, a second aspect of the present invention proposes an air traffic control surveillance system based on multimodal data processing, comprising:

[0183] The acquisition module is used to acquire multimodal air traffic control monitoring data in real time.

[0184] The preprocessing module is used to preprocess the multimodal air traffic control monitoring data and determine the preprocessed multimodal air traffic control monitoring data.

[0185] The data fusion module is used to fuse the preprocessed multimodal air traffic control monitoring data to determine the multimodal fused data.

[0186] The visualization module is used to visualize the multimodal fusion data as air traffic control monitoring results.

[0187] The beneficial effects of the above technical solution are as follows: By fusing multimodal data such as radar, ADS-B, BeiDou positioning, and real-time aircraft operational status data, the limitations of a single data source are effectively compensated for. Different modal data have their own advantages in positioning accuracy and status information. Through fusion, high-precision positioning and status awareness can be achieved. Multimodal data fusion provides richer and more comprehensive information for conflict prediction models, enabling more accurate prediction of aircraft movement trends. Integrating multiple modal data enables rapid and accurate identification of emergency situations. Multimodal fused data provides a comprehensive and accurate information foundation, allowing controllers to make more scientific and reasonable decisions. In tasks such as flight planning, airspace resource allocation, and aircraft avoidance, rich data support helps reduce decision-making errors and ensures the safe and orderly operation of air traffic.

[0188] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An air traffic control monitoring method based on multimodal data processing, characterized in that, include: Real-time acquisition of multimodal air traffic control monitoring data; The multimodal air traffic control monitoring data is preprocessed to determine the preprocessed multimodal air traffic control monitoring data; The preprocessed multimodal air traffic control monitoring data are fused to determine the multimodal fused data; The multimodal fusion data is used as air traffic control monitoring results and visualized. The multimodal air traffic control surveillance data is preprocessed to determine the preprocessed multimodal air traffic control surveillance data, including: Select any air traffic control monitoring data under any mode as the data to be cleaned; Obtain data points from the sequence data of each dimension in the data to be cleaned, and obtain the number of times the sequence data of each dimension is called within a preset time period; Select any one dimension of sequence data as the sub-data to be cleaned; The first indicator value of the sub-data to be cleaned is determined based on the number of calls; Determine the second index value of the sub-data to be cleaned based on the data values ​​of each data point in the sub-data to be cleaned; The product of the first index value and the second index value is used as the first data cleaning order index of the sub-data to be cleaned. Iterate through all the sub-data to be cleaned and obtain the first data cleaning order index for each sub-data to be cleaned. The sum of the first data cleaning order indices of all the sub-data to be cleaned is used as the second data cleaning order index of the data to be cleaned. By traversing the multimodal air traffic control monitoring data, the second data cleaning order index of the air traffic control monitoring data under each mode is obtained; Sort several second data cleaning order indices in descending order to generate a data cleaning list; Obtain data cleaning rules corresponding to different modal air traffic control monitoring data; Based on the data cleaning rules and data cleaning lists corresponding to different modes of air traffic control monitoring data, the multimodal air traffic control monitoring data is cleaned to obtain preprocessed multimodal air traffic control monitoring data.

2. The air traffic control monitoring method based on multimodal data processing as described in claim 1, characterized in that, Real-time acquisition of multimodal air traffic control monitoring data, including: Based on the mechanical scanning system of the secondary air traffic control radar, the monitoring data of the secondary air traffic control radar is acquired, and the first data is determined. ADS-B monitoring data is acquired using an ADS-B omnidirectional antenna to determine the second data. Based on the BeiDou / GPS / GLONASS three-in-one positioning antenna and the BeiDou antenna, acquire BeiDou surveillance data and determine the third data; The fourth data is determined by acquiring real-time operational status data of the aircraft based on the C2 link antenna. The first, second, third, and fourth data points are used as multimodal air traffic control monitoring data.

3. The air traffic control monitoring method based on multimodal data processing as described in claim 1, characterized in that, The second indicator value of the sub-data to be cleaned is determined based on the data values ​​of each data point in the sub-data to be cleaned, including: Calculate the mean of the data points in the sub-data to be cleaned, and use it as the target mean. Calculate the absolute difference between the data value of each data point in the sub-data to be cleaned and the target mean, and obtain several first absolute differences; Calculate the absolute difference between the data value of each data point in the sub-data to be cleaned and the data value of the next adjacent data point to obtain several second absolute differences; Calculate the ratio of the first absolute difference to the second absolute difference for each data point, and use it as the noise evaluation value; The noise evaluation value is compared with a preset noise evaluation threshold, and data points whose noise evaluation value is greater than or equal to the preset noise evaluation threshold are marked to obtain several marks; Calculate the ratio of the total number of several markers to the total number of data points in the sub-data to be cleaned, and determine the second index value of the sub-data to be cleaned.

4. The air traffic control monitoring method based on multimodal data processing as described in claim 1, characterized in that, The preprocessed multimodal air traffic control monitoring data are fused to determine the multimodal fused data, including: Select any mode of air traffic control monitoring data as the data to be processed; The data to be processed is standardized to obtain standardized data to be processed. The dimensionality of the standardized data to be processed is reduced to a preset low-dimensional space to obtain the dimensionality reduction matrix; Principal component analysis is performed on the standardized data to be processed based on the dimensionality reduction matrix to obtain the principal component data of the standardized data to be processed. The principal component data is segmented based on dynamically adjusting the size of a preset window to obtain several target principal component sub-data. Feature extraction is performed on several target principal component subsets to obtain target feature data for several target principal component subsets; By traversing all modes of air traffic control surveillance data, target feature data of several target principal component sub-data corresponding to each mode of air traffic control surveillance data are obtained. Construct a knowledge graph for target feature recognition and reasoning; Based on the target feature recognition reasoning knowledge graph, the association relationship between the target feature data of several target principal component sub-data is identified, and the association features between the target feature data of several target principal component sub-data are determined. Based on the correlation features between the target feature data of several target principal component subsets, determine the correlation feature points between the target feature data of several target principal component subsets; Based on the associated feature points, the target feature data of several target principal component sub-data are fused to obtain multimodal fused data.

5. The air traffic control monitoring method based on multimodal data processing as described in claim 4, characterized in that, The principal component data is segmented based on dynamically adjusting the size of a preset window to obtain several target principal component sub-data, including: Based on a preset window, the principal component data is sequentially pre-segmented from its initial position to obtain several principal component sub-data; the similarity between the first principal component sub-data and the other principal component sub-data besides the first principal component sub-data is calculated to obtain several similarity scores; the set of several similarity scores is taken as the similarity set of the first principal component sub-data. The size of the preset window is dynamically adjusted; the principal component data is pre-segmented based on several preset windows of different sizes to obtain the similarity set of the first principal component data corresponding to several preset windows of different sizes; The variance of similarity in each similarity set is obtained, and the evaluation value of the preset window under each size is determined to obtain several evaluation values. The size of the preset window corresponding to the minimum value among several evaluation values ​​is used as the target window size; The principal component data is segmented based on the target window corresponding to the target window size to obtain several target principal component sub-data.

6. The air traffic control monitoring method based on multimodal data processing as described in claim 4, characterized in that, Constructing a knowledge graph for target feature recognition and reasoning, including: Acquire knowledge in the field of air traffic control surveillance; A feature recognition knowledge graph is constructed based on the aforementioned knowledge in the air traffic control surveillance field; Obtain a preset inference rule database; the preset inference rule database includes several inference rules; Several initial feature recognition reasoning knowledge graphs are generated based on a pre-set reasoning rule database and feature recognition knowledge graph; Several initial feature recognition reasoning knowledge graphs are fused to determine the target feature recognition reasoning knowledge graph.

7. The air traffic control monitoring method based on multimodal data processing as described in claim 1, characterized in that, The multimodal fusion data is used as air traffic control monitoring results and visualized, including: A visualization platform is built based on coordinate systems and map engines; The multimodal fusion data is aligned based on a geographic coordinate system and a time axis; The aligned multimodal fusion data is used to populate the visualization platform, and the aligned multimodal fusion data is rendered based on the map engine. The rendering result is used as the air traffic control monitoring result, and the air traffic control monitoring result is visualized.

8. The air traffic control monitoring method based on multimodal data processing as described in claim 1, characterized in that, Before using the multimodal fusion data as air traffic control monitoring results and visualizing it, the following steps are also included: The fusion quality of the multimodal fusion data is evaluated, and the fusion evaluation value of the multimodal fusion data is determined. The fusion evaluation value is compared with a preset fusion evaluation threshold; If the fusion evaluation value is greater than or equal to the preset fusion evaluation threshold, the multimodal fusion data will be visualized. If the fusion evaluation value is less than the preset fusion evaluation threshold, the preprocessed multimodal air traffic control monitoring data will be re-fused until the evaluation value is greater than or equal to the preset evaluation threshold.

9. An air traffic control surveillance system employing the multimodal data processing-based air traffic control surveillance method as described in any one of claims 1-8, characterized in that, include: The acquisition module is used to acquire multimodal air traffic control monitoring data in real time. The preprocessing module is used to preprocess the multimodal air traffic control monitoring data and determine the preprocessed multimodal air traffic control monitoring data. The data fusion module is used to fuse the preprocessed multimodal air traffic control monitoring data to determine the multimodal fused data. The visualization module is used to visualize the multimodal fusion data as air traffic control monitoring results.

Images