Civil aviation flight big data centralized service system supporting multi-tenancy and multi-source access

Abstract

The application discloses a kind of civil aviation flight big data intensive service system of supporting multi-tenancy and multi-source access.System includes multi-source data access adaptation, data processing, engineering parameter, analysis parameter, analysis, multi-tenancy management and flow calculation module.Through multi-source data access, heterogeneous data is uniformly checked;Using data processing and engineering parameter module, decoding analysis, HDF5 export and engineering parameter intelligent mapping and normalization binding are completed;Based on analysis parameter and Spark, Kudu engine is used to build standard and tenant-specific analysis parameter library covering all flight segments;Combined with multi-tenancy management, logical isolation and account on-demand are realized;Using flow calculation, data timeliness is improved.The application solves the problems of high cost of traditional system repeated construction, cannot be shared, slow mapping and poor scalability, improves data processing real-time, security and resource utilization.

Description

Technical Field

[0001] This invention relates to the field of civil aviation flight data processing, specifically to a multi-tenant service system for the centralized storage, processing and use of multi-airline, multi-source civil aviation flight operation data (such as QAR data) in the civil aviation field. Background Technology

[0002] Currently, mainstream aircraft models on the market include Boeing 737-700, 737-800, 777-300ER, 787, Airbus A320, A320Neo, A330, A350, A380, and C919, among many others. These aircraft are all equipped with Quick Access Recorders (QARs). QAR devices typically have a capacity of 128MB, using MO (Magneto Optical) magneto-optical discs or PCMCIA cards as storage media. They can continuously record pilot actions, aircraft status, weather, and other flight parameters for up to 600 hours. The advent of wireless QARs further enhances the speed and convenience of data acquisition. Wireless QAR data downloads are not limited by the removal of solid-state storage media. After landing, when conditions such as engine shutdown, cabin door opening, and the airport meeting external conditions such as network and hardware infrastructure requirements, wireless QAR data is automatically downloaded via the local network.

[0003] QAR data is arranged and stored in binary format and needs to be logically converted into intuitive engineering data that engineers can understand. This process is called QAR data decoding. Currently, commercially available decoding software commonly used in China includes AirFASE, AGS, and GRAF. These commercial decoding software programs also have flight data analysis capabilities, primarily using parameter analysis to issue warnings for exceeding limits (such as recording a speeding event during driving using a photograph), but they do not process the QAR data for the entire process under normal circumstances.

[0004] With the increasing demand for in-depth utilization of flight data in civil aviation operations, existing flight data processing methods and systems have revealed the following prominent defects and problems: 1. Difficulty in unifying parameters across multiple aircraft models and time-consuming and labor-intensive matching: Due to significant differences in the data types, collection frequencies, and parameter names of QAR records for different aircraft models, and even different production years of the same aircraft model, and because the decoded data and flight operation system data (such as pilot information, take-off and landing airports, weather runways, etc.) are scattered in different systems, traditional systems lack effective intelligent mapping and normalization mechanisms, resulting in a large amount of manual and time-consuming data matching when combining specific flight data with the flight operation environment.

[0005] 2. High investment and operation and maintenance costs due to redundant construction in the industry: Because traditional systems only support access to a single data source from a single airline and lack a multi-tenant logical isolation mechanism, each airline needs to independently build, operate and maintain its own QAR data storage and processing system, resulting in a large amount of redundant construction within the civil aviation industry, long system construction cycles and serious waste of funds and hardware resources.

[0006] 3. Poor system scalability and low resource utilization: Because traditional systems cannot adapt to QAR data from multiple airlines, aircraft types, and manufacturers, as well as third-party civil aviation auxiliary data sources, each airline's system is independent of each other, forming data silos. This results in advanced data analysis models not being able to be reused or shared under the premise of secure isolation, which seriously restricts the improvement of the overall flight data analysis capabilities of the civil aviation industry. Summary of the Invention

[0007] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0008] The purpose of this invention is to solve the above-mentioned problems and provide a civil aviation flight big data centralized service system that supports multi-tenant and multi-source access. It can effectively and centrally store and process QAR data of civil aviation flight operations from multiple airlines and multiple sources, forming a database of analytical parameters that can meet the needs of various business operations. At the same time, it realizes multi-tenant isolation, instant account opening and use, supports multiple airlines to share a single system, reduces the industry's redundant construction costs, and improves the convenience of using flight operation data and the resource utilization rate.

[0009] The technical solution of this invention is as follows: This invention discloses a civil aviation flight big data centralized service system supporting multi-tenancy and multi-source access. The system includes a multi-source data access adaptation module, a data processing module, an engineering parameter module, an analysis parameter module, an analysis module, a multi-tenant management module, and a streaming computing module, wherein: The multi-source data access adaptation module is used to access data from different airlines, aircraft types, and transmission methods, as well as civil aviation auxiliary data sources, to achieve unified access and preliminary verification of multi-source data. The data processing module is connected to the output end of the multi-source data access adapter module. It is used to decode and parse the full parameters of multiple aircraft models, write big data and match and associate information with the flight operation system, export a file containing all parameters, and complete data cleaning, storage and matching and association with flight operation system information. The engineering parameter module, connected to the output of the data processing module, is used to normalize the engineering parameters of different aircraft models to support the subsequent establishment of an analysis parameter library. The analysis parameter module, connected to the output of the engineering parameter module, includes a standard analysis parameter library unit and an analysis parameter editing unit. The standard analysis parameter library unit is used to construct a statistical parameter system covering the entire flight segment, and the analysis parameter editing unit is used to provide the algorithm syntax for calculating flight big data indicator analysis parameters. The analysis module, connected to the output of the analysis parameter module, is used to establish an analysis based on the combination of analysis parameters and flight elements, and supports outputting multi-dimensional and multi-scenario analyses according to tenant needs; The multi-tenant management module connects to various data storage and processing modules in the system, enabling logical isolation of multiple airline tenants, instant account activation, and support for data sharing while ensuring data security and compliance.

[0010] According to an embodiment of the civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access of the present invention, the system further includes: The streaming computing module connects the analysis parameter module and the analysis module, and is used for streaming computing of real-time flight data.

[0011] According to an embodiment of the civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access of the present invention, the system further includes: The version control module is connected to the output of the data processing module and the input of the engineering parameter module. It is used to manage the engineering parameter library throughout its entire lifecycle, record the addition of parameters and changes in parameter names in each version library, prevent historical parameter data from being overwritten, and ensure the traceability of historical parameters.

[0012] According to an embodiment of the civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access of the present invention, the system further includes: The sensitive flight management module, connected to the analysis module, is used to mask sensitive flight information, specifically the flight number and tail number of the sensitive flights that have been set up.

[0013] According to an embodiment of the civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access of the present invention, the data processing module is further configured as follows: The system automatically accesses the QAR data of each flight, decodes the file, generates a log file, and exports an HDF5 file containing all parameters. It backs up the QAR data file and flight information from the flight operation system, determines the primary key of the data, transforms the decoded flight data into columnar storage and stores it in the HBase cluster, and compresses the flight data. Finally, it performs preliminary cleaning and filtering of the flight data according to the configured cleaning rules.

[0014] According to an embodiment of the civil aviation flight big data centralized service system supporting multi-tenancy and multi-source access of the present invention, the cleaned and filtered data automatically generates a log file after being stored in the database. The log file records the name of the original QAR data file and the name of the full parameter file. It is determined whether the full parameter file is the result of correct decoding. If not, the corresponding QAR data file is re-stored in the database under manual intervention based on the information in the log file.

[0015] According to an embodiment of the civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access of the present invention, the engineering parameter module is configured to perform secondary cleaning and filtering on flight data according to the configured cleaning rules to obtain aircraft parameter database metadata.

[0016] According to an embodiment of the civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access of the present invention, the normalization processing of the engineering parameter module is to map and match the metadata of different aircraft model parameter libraries with standard engineering parameters, and further includes: The first level of automatic matching is performed using name regular expression matching to find model parameters with the same or similar names, while verifying the data output range. After the first layer of automatic matching is completed, the second layer of automatic matching is performed on the unmatched model parameters through semantic similarity matching; Calculate the credibility of the matching results after automatic matching at the first and second layers; The remaining unmatched model parameters are manually verified, and adjustments and parameter matching are made manually based on the calculated confidence level. Set a standard frequency, and automatically increase the frequency by filling in the average value for parameters that are lower than the standard frequency. Manual frequency setting is also supported.

[0017] According to an embodiment of the civil aviation flight big data centralized service system supporting multi-tenancy and multi-source access of the present invention, the engineering parameter module is further configured to provide an interface for custom functions and custom scripts for engineering parameters. Through this interface, custom functions are used to edit new custom engineering parameters, process the metadata of the aircraft parameter library to obtain the engineering parameter library, and the engineering parameter library supports personalized expansion according to tenant needs, serving as the basis for analyzing the parameter library.

[0018] According to an embodiment of the civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access of the present invention, the analysis parameter module is configured as follows: Based on the civil aviation business logic, and using the unified standard engineering parameters in the engineering parameter module, the system functions in the algorithm syntax provided in the analysis parameter editing unit are used to calculate the analysis parameters, i.e., the normalized engineering parameters, through user-defined scripts. At the same time, it supports assigning exclusive analysis parameter editing and query permissions to different tenants according to their permissions, so as to realize the isolation and reuse of analysis parameters between tenants.

[0019] According to an embodiment of the civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access of the present invention, the analysis parameter module adopts the Spark distributed computing engine. The calculation request of the analysis parameter is sent to the Spark distributed computing environment based on YARN resource scheduling. The Spark distributed computing engine divides the data into flight-analysis parameter units and performs parallel calculations. Each core in the Spark distributed computing engine is responsible for the calculation of one smallest unit. After any core has completed the calculation of one analysis parameter of a flight, it will be assigned a new calculation task. The computing resources are dynamically allocated according to the tenant quota to ensure the performance stability when multiple tenants use the data concurrently.

[0020] According to an embodiment of the civil aviation flight big data intensive service system supporting multi-tenant and multi-source access of the present invention, the analysis parameter module stores the calculation results of Spark operation scripts in a columnar database and establishes the analysis parameter library in the system. The analysis parameter library includes a standard analysis parameter library and a tenant-specific analysis parameter library. The multiple analysis parameters in the standard analysis parameter library cover flight indicators throughout the entire flight process. The tenant-specific analysis parameter library is used to store the analysis parameters and calculation results customized by the tenant, thereby realizing tenant data isolation.

[0021] According to an embodiment of the civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access of the present invention, the analysis module is configured as follows: The system receives the flight or analysis parameter filtering conditions selected by the user on the query page, generates the corresponding SQL statement in the background and submits it to the database query engine, which executes the SQL statement. In the analysis and processing, it performs multi-dimensional and multi-scenario aggregate analysis based on flight operation data and the calculation results of the analysis parameter module. It also supports the generation of personalized analysis reports according to tenant needs, and only displays data and analysis results within the tenant's permission scope.

[0022] Compared with the prior art, the present invention has the following advantages: The system of this invention can effectively store massive amounts of multi-source aircraft operation data, establish a standard engineering parameter library for the engineering parameters of different aircraft models or aircraft of the same model at different production times, and appropriately add custom analysis parameters based on actual operation conditions to establish an analysis parameter library with civil aviation logic and good scalability. The data in the analysis parameter library can be widely used in various subsequent application scenarios.

[0023] Meanwhile, the core innovation of this invention lies in solving the pain point of redundant construction of systems in the civil aviation industry. By adding a multi-source data access adaptation module and a multi-tenant management module, it realizes unified access of multi-airline and multi-source data, as well as logical isolation of multi-tenants and instant account use. It supports multiple airlines to share a single system, significantly reducing the investment of each airline in building its own system, reducing industry operation and maintenance costs, and realizing intensive use of resources.

[0024] In detail, the system of this invention provides a data frame-like approach to process big data during flight, while incorporating multi-tenancy, multi-source access, and intensive design concepts, specifically reflected in the following aspects: 1. Similar to object-oriented programming, the flight process data is treated as a dataset (table): Unlike traditional decoding software that uses row-level computation as its process, processing each row of flight process data after decoding, this system retrieves the entire result set and operates on the entire result set. The advantage of this approach is that it eliminates the need for defining numerous variables to record the results of the previous row's calculations, as is common in traditional decoding software.

[0025] 2. Improved Flight Process Data Processing Through System Functions: Flight process data is highly complex, involving parameters with multiple frames per second, requiring consideration of user-defined frequency downsampling, frequency upsampling, adding parameters of different frequencies, frame skipping, and user-specified sampling frequencies. These complexities are addressed through system functions within the algorithm syntax provided in the analysis parameter editing unit. The results are then compared with those from traditional decoding software to verify their correctness, thus simplifying data processing.

[0026] 3. Provides interfaces for mutual calling of a large number of analysis parameters through system functions: The system functions in the algorithm syntax provided in the analysis parameter editing unit integrate the advantages of various traditional processing methods, which not only makes the system calculation results closer to the results of traditional decoding software, but also improves the flexibility of parameter reuse. At the same time, it supports the customization of analysis parameters according to the needs of tenants through user-defined scripts, adapting to the personalized needs of different airlines.

[0027] 4. Normalization processing via the engineering parameter module: The engineering parameter module is connected to the output of the data processing module and is used to perform normalization processing to retrieve, match, bind, and verify engineering parameters and original parameters through engineering parameter mapping. Since the normalization of flight parameters for multiple aircraft models has been completed at the engineering parameter level, when designing analysis parameter algorithms for different aircraft models, only the normalized data needs to be used, and there is no need to design an algorithm for each aircraft model, which greatly speeds up the efficiency of flight algorithms.

[0028] 5. Achieving unified access and optimized export format for multi-source data through a multi-source data access and adaptation module: The multi-source data access and adaptation module is used to access data from different airlines, aircraft types, and transmission methods, as well as civil aviation auxiliary data sources. It achieves unified access and preliminary verification of multi-source data, supports multiple transmission methods such as third-party civil aviation data interfaces, and is compatible with QAR data formats for various aircraft types such as Boeing, Airbus, and 919. It is also compatible with multiple types of civil aviation auxiliary data sources such as flight operations, meteorology, and aircraft maintenance, realizing unified access and standardized processing of multi-source data and solving the problems of single data source and poor scalability in existing systems. The data processing module is configured to export HDF5 format files containing all parameters, organize full-parameter data by flight dimension, and support block compression and multi-format conversion, which greatly improves the efficient reading and writing capabilities and cross-platform adaptability of data, meeting the data format requirements of multi-airline and multi-scenario data analysis.

[0029] 6. Achieve centralized deployment of multi-tenants through the multi-tenant management module: The multi-tenant management module connects with various data storage and processing modules of the system to achieve logical isolation of multi-airline tenants, instant account access, support for adding, canceling, and managing the status of tenants, creating multiple accounts, allocating permissions, and fine-grained control. This enables multiple airlines to share a single system, with data not visible between tenants and no interference between operations, reducing redundant construction costs and improving resource utilization.

[0030] 7. Improve the real-time performance of data processing through the streaming computing module: The streaming computing module connects the analysis parameter module and the analysis module for real-time streaming computing of flight data. This optimizes the original method of periodically batch processing of flight data into a streaming computing mode, with a single flight as the smallest processing unit. After the aircraft lands, the QAR data is uploaded and verified, triggering the data processing flow. This achieves real-time processing of "processing one flight at a time," significantly reducing data processing latency. Combined with the unified access capability of multi-source data, the system can efficiently handle the high-concurrency QAR data access and processing needs of multiple airlines, solving the problems of data lag and inability to adapt to the data volume of multiple airlines in the existing batch processing mode, and improving the timeliness of flight big data processing.

[0031] 8. Achieving full lifecycle traceability of engineering parameters through the version control module: The version control module is connected to the output end of the data processing module and the input end of the engineering parameter module to perform full lifecycle management of the engineering parameter library. It records the addition of parameters and changes in parameter names in each version library, and establishes a dedicated engineering parameter version sub-library for each tenant to record key information such as parameter addition, modification, and deletion. This avoids the overwriting of historical parameter data, ensures the traceability of historical parameters, and completely solves the industry problem of untraceable and easily confused parameter versions in existing systems. Parameter operations are linked with the version library in real time to ensure the accuracy and timeliness of parameter records. This provides a precise parameter foundation for flight data decoding, engineering parameter matching, and analysis parameter calculation, ensuring the accuracy and traceability of the entire flight big data processing flow.

[0032] 9. Enhance data security and compliance through the multi-tenant management module: The multi-tenant management module achieves deep logical isolation between tenants, partitions data storage based on unique tenant identifiers, and implements fine-grained access control to ensure that different tenant accounts cannot access data across tenants. It achieves complete isolation of data between airlines from the bottom layer, ensuring that data between tenants is invisible and operations do not interfere with each other. This lays a solid foundation for cross-airline data security for multiple airlines to share a single system, eliminating airlines' core security concerns about data sharing and platform sharing.

[0033] 10. Implement autonomous control of sensitive flight information through the sensitive flight management module: By connecting the sensitive flight management module with the analysis module, sensitive flight information can be masked. The key information such as flight number and tail number of the set sensitive flights can be masked by desensitization, hash encryption and other masking processes. At the same time, it supports administrators to customize the types of sensitive information and exclusive viewing permissions, so as to realize the refined autonomous control of sensitive flight data within the airline and further improve the airline's data privacy and security level.

[0034] 11. Secure sharing of excellent models through the model publishing function of the analysis module: The analysis module establishes intelligent analysis based on the combination of analysis parameters and flight elements, and adds a model publishing function. Under the premise of ensuring data isolation among multiple tenants, a system-level model sharing library is built. When a tenant publishes an excellent analysis model, the system automatically removes the original sensitive data from the model, retaining only reusable information such as model structure, algorithm parameters, and training results. Other tenants can view and download the model in the shared library for learning and use. This not only realizes the interoperability and sharing of excellent data analysis models among airlines, but also avoids the leakage of original data, solves the problems of isolated models and low resource utilization in the existing system, and promotes the improvement of the overall flight data analysis capabilities of the civil aviation industry.

[0035] 12. Intelligent mapping and matching of engineering parameters through the engineering parameter module: The engineering parameter module performs secondary cleaning and filtering of flight data according to the configured cleaning rules, obtaining aircraft parameter database metadata as the original parameters for retrieval and matching. First-level automatic matching is performed through name regular expression matching, and second-level automatic matching is performed through semantic similarity matching. The credibility of the matching results is calculated, and the remaining unmatched parameters are manually verified and their frequencies adjusted. A keyword index library and a vector feature library are constructed. Rapid matching of engineering parameters and original parameters is achieved through keyword retrieval and vector similarity calculation. After matching, data range and frequency verification are automatically completed, and parameter binding is achieved, significantly improving the efficiency and accuracy of parameter matching. Compared with the existing manual and basic automatic matching modes, this significantly reduces manual intervention and lowers labor costs. It can quickly adapt to the parameter integration needs of multiple airlines and aircraft types, providing efficient parameter adaptation support for new airlines to quickly access the system.

[0036] In summary, the technical solution of this invention has the characteristics of integration, real-time performance, security, traceability, and high scalability. It not only solves the single-airline adaptation deficiency of existing flight big data processing systems, but also conforms to the integration development trend of the civil aviation industry. It provides multiple airlines with an efficient, economical, and secure solution for the storage, processing, and use of flight big data, and has significant technological advancements and industry application value. Attached Figure Description

[0037] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0038] Figure 1 This is a schematic diagram of an embodiment of the civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access of the present invention. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0040] Figure 1 This illustrates the principle of an embodiment of the civil aviation flight big data centralized service system supporting multi-tenancy and multi-source access of the present invention.

[0041] refer to Figure 1The system in this embodiment includes the following modules: multi-source data access adaptation module, data processing module, engineering parameter module, analysis parameter module, analysis module, multi-tenant management module, and streaming computing module.

[0042] The multi-source data access adaptation module is used to access data from different airlines, aircraft types, and transmission methods, as well as civil aviation auxiliary data sources, to achieve unified access and preliminary verification of multi-source data.

[0043] In practical implementation, the multi-source data access unified access of the multi-source data access adaptation module refers to supporting data access through various transmission methods such as wireless QAR, wired QAR, and third-party civil aviation data interfaces, adapting to the QAR data formats of Boeing and Airbus aircraft models, as well as the formats of auxiliary data sources such as flight operation data, meteorological data, and aircraft maintenance data.

[0044] The initial verification of the multi-source data access adaptation module includes: verifying the integrity of the accessed data (verifying whether the data is missing key fields or whether there are outliers), standardizing the format conversion (converting QAR data and auxiliary data of different formats into a format that the system can uniformly recognize), filtering invalid data, and transmitting valid data to the subsequent data processing module to provide a unified and standardized data source for subsequent data processing.

[0045] The data processing module connects to the output of the multi-source data access adapter module. It is used to decode and parse full parameters of multiple aircraft models, write large data, match and associate information with flight operation system information, export files containing full parameters, and complete data cleaning, storage, and matching and association with flight operation system information.

[0046] The decoding and parsing process uses traditional decoding software to decode the QAR data for each flight, exporting an HDF5 format file containing all parameters (hereinafter referred to as the "full parameter file"). Simultaneously, it extracts the log file automatically generated by the decoding software after decoding and data entry. The log file records the name of the original QAR data file and the name of the decoded full parameter file. Then, it determines whether the full parameter file is the correctly decoded result. If not, based on the information in the decoding software's log file, the corresponding original QAR data file is manually re-decoded and re-entered into the database. If it is correct, subsequent modules continue processing. The decoded HDF5 format data is referred to as flight data.

[0047] Because QAR data needs to be matched and correlated with existing flight operation systems (which store other basic flight information such as weather, runway, company, and personnel) based on flight departure time, tail number, and airport / departure / arrival factors, the original QAR data file, full parameter file, and flight information from the flight operation system are correlated according to given rules. The primary key is determined, and the flight data is transformed into columnar storage (converting traditional row-based storage to columnar storage) and stored in an HBase cluster. The Snappy development package is used to compress the flight data. Other matching information (such as personnel, airport, and weather information for matching flights) is stored using Oracle. Simultaneously, based on the tenant identifier assigned by the multi-tenant management module, data from different tenants is logically partitioned for storage, ensuring data isolation between tenants.

[0048] In addition, the data processing module also provides an interface to improve the quality of flight data. Flight data is extracted through this interface, and cleaning rules (jump processing, threshold processing, etc.) are configured according to the aircraft system characteristics to perform preliminary cleaning and filtering of the flight data.

[0049] The engineering parameters module connects to the output of the data processing module and is used to normalize the engineering parameters of different aircraft models to support the subsequent establishment of an analysis parameter library.

[0050] The engineering parameter module is configured to perform secondary cleaning and filtering on flight data according to the configured cleaning rules to obtain aircraft parameter database metadata.

[0051] Normalization involves mapping and matching metadata from different aircraft model parameter libraries with standard engineering parameters, and further includes the following processing steps: The first layer of automatic matching is performed using name regular expression matching to find model parameters with the same or similar names. Simultaneously, the data output range is verified. For example, when a new model is added to the system with a new parameter list, the system compares the names in this new parameter list with existing engineering parameters in the engineering parameter module. If the names are the same or similar, an automatic match is made. The data output range verification process is as follows: First, sample data A is sorted, and data between the 25th and 75th positions is considered the valid range of A. The standard deviation of sample A is calculated. Second, sample B with the same parameter name is sorted, and data between the 25th and 75th positions is considered the valid range of B. Sample B is then mixed with sample A. If, after mixing, the fluctuation of the new sample parameters is less than three times the standard deviation of sample A, then the data ranges of samples A and B are considered consistent. After the first layer of automatic matching is completed, a second layer of automatic matching is performed for unmatched aircraft model parameters using semantic similarity matching. Currently, most engineering parameters are described by the aircraft manufacturers. By splitting the description fields into multiple words and using word matching, the default matching method that finds the closest words in terms of frequency and order is used to solve some Airbus and Boeing parameter issues. The credibility of the matching results after the first and second layers of automatic matching is calculated. For example, the calculated credibility is divided into four levels: extremely high, high, relatively high, and medium. Extremely high credibility indicates that the name, frequency, and parameter range are completely identical, and the parameter descriptions are identical or have minor differences, such as punctuation. High credibility indicates that the name and parameter range are the same; the frequency and parameter descriptions have some differences. Relatively high credibility indicates that the name can be fuzzily matched using regular expressions, and the parameter range is the same. Medium credibility indicates that the parameter name cannot be matched, the parameter range is the same, but the parameter descriptions use similar words. The remaining unmatched model parameters require manual verification and processing. The remaining unmatched model parameters are manually verified, and adjustments and parameter matching are made manually based on the calculated confidence level. Set a standard frequency. Parameters below the standard frequency (the system defaults to the highest frequency among all model parameters) will be automatically increased by filling with the average value. Manual frequency setting is also supported. Because parameter frequencies can vary significantly between different models, a unified frequency setting is required after the above parameter matching is completed.

[0052] Furthermore, the engineering parameter module is configured to provide interfaces for custom functions and custom engineering parameter scripts. These interfaces allow users to edit new custom engineering parameters using custom functions, process the model parameter library metadata to obtain the engineering parameter library, and support personalized expansion based on tenant needs, serving as the foundation for the analysis parameter library. Different tenants can customize their own engineering parameters based on the standard engineering parameter library according to their own business needs, and the customized engineering parameters between tenants are isolated and do not interfere with each other.

[0053] The analysis parameter module, connected to the output of the engineering parameter module, includes a standard analysis parameter library unit and an analysis parameter editing unit. The standard analysis parameter library unit is used to build a system of statistical parameters of interest covering the entire flight segment, while the analysis parameter editing unit provides the algorithm syntax for calculating flight big data indicator analysis parameters. It also supports the isolation and reuse of analysis parameters among tenants based on tenant permissions.

[0054] Based on the civil aviation business logic, and using the unified standard engineering parameters in the engineering parameter module, the system functions in the algorithm syntax provided in the analysis parameter editing unit are used to calculate the analysis parameters, i.e., the normalized engineering parameters, through user-defined scripts. At the same time, it supports assigning exclusive analysis parameter editing and query permissions to different tenants according to their permissions, so as to realize the isolation and reuse of analysis parameters between tenants.

[0055] The analysis parameter module uses the Spark distributed computing engine. Calculation requests for analysis parameters are sent to the Spark distributed computing environment (Spark on YARN) based on YARN resource scheduling. The Spark distributed computing engine divides the computation into flight-analysis parameter units and performs parallel computations. Each core in the Spark distributed computing engine is responsible for the computation of one of the smallest units. After any core has completed the computation of one analysis parameter for a flight, it will be assigned a new computation task. Computing resources are dynamically allocated according to tenant quotas to ensure performance stability when multiple tenants use the service concurrently.

[0056] The analysis parameter module stores the calculation results of Spark computation scripts in a columnar database such as Kudu. It establishes an analysis parameter library in the system, which includes a standard analysis parameter library and a tenant-specific analysis parameter library. The standard analysis parameter library contains multiple analysis parameters that cover flight indicators throughout the entire flight process. The tenant-specific analysis parameter library is used to store the tenant's custom analysis parameters and calculation results, thereby achieving tenant data isolation and ensuring that the analysis parameter data of different tenants do not interfere with each other.

[0057] The user-defined scripts described above use Groovy as their scripting language, supporting all basic syntax. Java support is integrated into the Groovy runtime context, allowing calls to Java's basic class packages. The system functions mentioned above mainly include: 1. `filter`: The most commonly used flight calculation function, a typical data frame-based SQL filtering method. This method can remove data that the user does not need from a large amount of flight data. For example: `filter("'FLIGHT_PHASE' = 3 and 'RALTC' < 1000")` means that the user only cares about the data where the flight phase is 3 and the radio altitude is below 1000, and other data is removed.

[0058] 2. runScript: A parameter reference within the same parameter library. For example: _.runScript(“Time Point 1”); means referencing the analysis parameter for time point 1 in the current parameter library.

[0059] 3. foreach: The foreach method in engineering calculation parameters provides users with a lower-level processing logic. Users can process the calculation logic themselves after obtaining the data for each frame in the internal function.

[0060] Regarding the computing engine, considering the need to balance computational complexity and performance, the Spark distributed computing engine is preferred for the analysis parameter module. The Spark ecosystem is a popular big data processing technology, compatible with traditional big data storage HDFS and resource management framework Yarn, making it ideal for handling massive amounts of complex flight process analysis parameters in the analysis parameter module. In this embodiment, the Spark distributed computing engine reads the configuration of the analysis parameter script to be calculated, transforms the calculation of the analysis parameters into a set of tasks for a directed acyclic graph (DAG), and completes the calculation of complex flight process analysis parameters through rich SQL statements and a rich set of machine learning algorithms, thus meeting the complexity requirements. This allows for the calculation of multiple flight parameters. The calculation task with multiple parameters (Cartesian product) generates a task configuration file. The main program reads the task configuration file and distributes the calculation task to different running nodes. It leverages the advantages of clustering to perform batch calculations, thereby meeting performance requirements.

[0061] Regarding storage technology, considering that the analysis parameter module needs to continuously write the calculation results of Spark operations at high concurrency, such as for user-defined analysis parameter algorithms, and also needs to support real-time querying and analysis, such as querying event-related continuous engineering parameters, Kudu storage technology is preferred. This technology is an open-source columnar storage technology running on the Hadoop platform, compatible with the Hadoop ecosystem, and suitable for random read / write and batch scanning scenarios. In the analysis parameter module, Kudu's database table design adopts a partitioned approach, with each table storing a large number of analysis parameters. Through this partitioning strategy, the system can theoretically support an unlimited number of analysis parameters for expansion. Simultaneously, partitioning management is based on tenant identifiers, achieving isolated storage of tenant-specific analysis parameter libraries.

[0062] The analysis module connects to the output of the analysis parameter module and is used to establish analyses based on the combination of analysis parameters and flight elements. It supports outputting multi-dimensional and multi-scenario analyses according to tenant requirements.

[0063] Based on the aforementioned flight big data, this embodiment of the system preferably applies business intelligence principles to the aviation field. As described in the preceding modules, the underlying big data not only stores flight process big data but also integrates various factors such as flight operation data, scheduling data, meteorological data, runway data, and aircraft maintenance data with the flight process big data into a unified big data environment, and pre-matches them, solving the problems of insufficient data and difficulty in matching. The analysis module allows analysts to perform analysis using multiple preset standard analysis parameters, and also supports analysts to define new analysis parameters in the analysis parameter module, solving the problem of insufficient flight analysis data.

[0064] Considering the need for front-end analysis and querying to handle large amounts of data while maintaining fast response times, the analysis module uses Impala as its underlying database query engine. Impala is an interactive SQL query framework based on in-memory computing, offering high performance and low latency. It is compatible with native data warehouse queries within the Hadoop ecosystem and is the preferred real-time query and analysis engine for petabyte-scale big data on the CDH platform. The analysis module is configured to receive flight or analysis parameter filters selected by the user on the query page, generate corresponding SQL statements in the background, and submit them to the Impala engine. The Impala engine executes the SQL statements and performs multi-dimensional, multi-scenario aggregation analysis based on flight operation data and the calculation results from the analysis parameter module. It also supports generating personalized analysis reports according to tenant needs, displaying only data and analysis results within the tenant's access permissions. The Impala engine achieves sub-second response times for queries linking the calculation results of the analysis parameter module with flight operation data, allowing users to achieve complex analysis content and report results simply through system page-based analysis configurations, reducing the learning curve for users.

[0065] The analysis module allows pilots to freely combine analytical parameters with factors such as pilot and environmental conditions. Any analytical parameter or flight attribute can be dragged onto the X, Y, or Z axes of a chart. The module automatically identifies whether the parameter is a dimension type (non-numerical) or a measure type (numerical) and performs the corresponding conversion to generate the analytical chart. Currently supported chart types include line charts, bar charts, stacked charts, scatter plots, 3D charts, and pie charts, greatly improving the efficiency of the analysis process.

[0066] The multi-tenant management module connects to various data storage and processing modules within the system, enabling logical isolation between multiple airline tenants, immediate account activation, and ensuring data security and compliance while supporting data sharing. Specific configurations are as follows: (1) Tenant Management: Supports the addition, cancellation and status control of tenants, assigns a unique tenant identifier to each tenant, and all tenants share a set of system hardware and software resources, but achieve logical isolation based on the tenant identifier, and the data and operations between tenants do not interfere with each other; (2) Account Management: Supports the creation and permission allocation of multiple accounts under a tenant, realizes the binding of accounts and tenants, and different tenant accounts cannot access data across tenants. Different accounts under the same tenant can be assigned different operation permissions (such as read-only, edit, administrator, etc.). (3) Access Control: Based on tenant identifiers and account permissions, the data access and operation permissions of each module are finely controlled to ensure that tenants can only access and operate data and functions within their own permission scope, thus protecting tenant data security and privacy; (4) Quick activation: Provides tenant accounts that can be used immediately. New airline tenants do not need to build the system again. They only need to complete tenant registration, account activation and permission configuration to use all the functions of the system, which greatly shortens the deployment cycle and reduces the cost of repeated construction.

[0067] In summary, the multi-source data access and adaptation module enables unified access and adaptation of multi-source data, providing the system with a unified and standardized data source; the data processing module decodes, cleans, and stores multi-source QAR data, and implements logical data partitioning based on tenant identifiers; the engineering parameter module normalizes engineering parameters for different aircraft models and supports personalized extensions for tenants; the analysis parameter module generates standard and tenant-specific analysis parameter libraries based on normalized engineering parameters, achieving isolation and reuse between tenants; the analysis module provides personalized intelligent analysis services to tenants based on analysis parameters and flight data; and the multi-tenant management module runs throughout the entire system, enabling full-process control of multiple tenants, supporting multiple airlines to share a single system, and achieving centralized deployment.

[0068] The entire system addresses the pain points of existing systems, such as single data source, inability to be shared by multiple airlines, and high costs of redundant construction. It enables multi-source data access, multi-tenant isolation, instant account access, and intensive resource utilization, significantly reducing the redundant construction and maintenance costs in the civil aviation industry. At the same time, it improves the ease of use and scalability of flight data and adapts to the personalized business needs of multiple airlines.

[0069] In addition, the system in this embodiment also includes a streaming computing module, which connects the analysis parameter module and the analysis module, for streaming computing of real-time flight data. The transmission characteristics of civil aviation QAR data determine that it is transmitted discretely from flight data. For the continuous flow of flight files from business operations, the system adopts a streaming pipeline for seamless processing. That is, when the multi-source data access adaptation module receives the QAR data of a certain flight, the streaming computing module captures the arrival event of the flight data, drives the analysis parameter module (Spark distributed engine) to divide the task into units of "flight - analysis parameter" and perform parallel calculations. The calculation results are written to the columnar database (Kudu) in real time, and the analysis module and visualization page can query and present the analysis results of the flight immediately.

[0070] In addition, the system in this embodiment also includes: a version library management module, which is connected to the output end of the data processing module and the input end of the engineering parameter module. It is used to perform full lifecycle management of the engineering parameter library, record the addition of parameters and changes of parameter names in each version library, avoid the overwriting of historical parameter data, and ensure the traceability of historical parameters.

[0071] In addition, the system in this embodiment also includes a sensitive flight management module, which is connected to the analysis module and is used to shield sensitive flight information, and to shield key information such as flight number and tail number of the sensitive flights that have been set.

[0072] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0073] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0074] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein can be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0075] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0076] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0077] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A civil aviation flight big data centralized service system supporting multi-tenancy and multi-source access, characterized in that, The system includes a multi-source data access adaptation module, a data processing module, an engineering parameter module, an analysis parameter module, an analysis module, a multi-tenant management module, and a streaming computing module, wherein: The multi-source data access adaptation module is used to access data from different airlines, aircraft types, and transmission methods, as well as civil aviation auxiliary data sources, to achieve unified access and preliminary verification of multi-source data. The data processing module is connected to the output end of the multi-source data access adapter module. It is used to decode and parse the full parameters of multiple aircraft models, write big data and match and associate information with the flight operation system, export a file containing all parameters, and complete data cleaning, storage and matching and association with flight operation system information. The engineering parameter module, connected to the output of the data processing module, is used to normalize the engineering parameters of different aircraft models to support the subsequent establishment of an analysis parameter library. The analysis parameter module, connected to the output of the engineering parameter module, includes a standard analysis parameter library unit and an analysis parameter editing unit. The standard analysis parameter library unit is used to construct a statistical parameter system covering the entire flight segment, and the analysis parameter editing unit is used to provide the algorithm syntax for calculating flight big data indicator analysis parameters. The analysis module, connected to the output of the analysis parameter module, is used to establish an analysis based on the combination of analysis parameters and flight elements, and supports outputting multi-dimensional and multi-scenario analyses according to tenant needs; The multi-tenant management module connects to various data storage and processing modules in the system, enabling logical isolation of multiple airline tenants, instant account activation, and support for data sharing while ensuring data security and compliance.

2. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 1, characterized in that, The system also includes: The streaming computing module connects the analysis parameter module and the analysis module, and is used for streaming computing of real-time flight data.

3. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 1, characterized in that, The system also includes: The version control module is connected to the output of the data processing module and the input of the engineering parameter module. It is used to manage the engineering parameter library throughout its entire lifecycle, record the addition of parameters and changes in parameter names in each version library, prevent historical parameter data from being overwritten, and ensure the traceability of historical parameters.

4. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 1, characterized in that, The system also includes: The sensitive flight management module, connected to the analysis module, is used to mask sensitive flight information, specifically the flight number and tail number of the sensitive flights that have been set up.

5. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 1, characterized in that, The data processing module is further configured as follows: The system automatically accesses the QAR data of each flight, decodes the file, generates a log file, and exports an HDF5 file containing all parameters. It backs up the QAR data file and flight information from the flight operation system, determines the primary key of the data, transforms the decoded flight data into columnar storage and stores it in the HBase cluster, and compresses the flight data. Finally, it performs preliminary cleaning and filtering of the flight data according to the configured cleaning rules.

6. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 5, characterized in that, After the cleaned and filtered data is entered into the database, a log file is automatically generated. The log file records the name of the original QAR data file and the name of the full parameter file. It is determined whether the full parameter file is the result of correct decoding. If not, the corresponding QAR data file is re-entered into the database under manual intervention based on the information in the log file.

7. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 1, characterized in that, The engineering parameter module is configured to perform secondary cleaning and filtering on flight data according to the configured cleaning rules to obtain aircraft parameter database metadata.

8. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 7, characterized in that, The normalization process of the engineering parameter module involves mapping and matching metadata from different model parameter libraries with standard engineering parameters, and further includes: The first level of automatic matching is performed using name regular expression matching to find model parameters with the same or similar names, while verifying the data output range. After the first layer of automatic matching is completed, the second layer of automatic matching is performed on the unmatched model parameters through semantic similarity matching; Calculate the credibility of the matching results after automatic matching at the first and second layers; The remaining unmatched model parameters are manually verified, and adjustments and parameter matching are made manually based on the calculated confidence level. Set a standard frequency, and automatically increase the frequency by filling in the average value for parameters that are lower than the standard frequency. Manual frequency setting is also supported.

9. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 8, characterized in that, The engineering parameter module is also configured to provide an interface for custom functions and custom scripts for engineering parameters. Through this interface, custom functions can be used to edit new custom engineering parameters, process the metadata of the model parameter library to obtain the engineering parameter library, and the engineering parameter library can be personalized and expanded according to the tenant's needs, serving as the basis for the analysis parameter library.

10. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 1, characterized in that, The analysis parameter module is configured as follows: Based on the civil aviation business logic, and using the unified standard engineering parameters in the engineering parameter module, the system functions in the algorithm syntax provided in the analysis parameter editing unit are used to calculate the analysis parameters, i.e., the normalized engineering parameters, through user-defined scripts. At the same time, it supports assigning exclusive analysis parameter editing and query permissions to different tenants according to their permissions, so as to realize the isolation and reuse of analysis parameters between tenants.

11. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 10, characterized in that, The analysis parameter module uses the Spark distributed computing engine. Calculation requests for analysis parameters are sent to the Spark distributed computing environment based on YARN resource scheduling. The Spark distributed computing engine divides the computation into flight-analysis parameter units and performs parallel computations. Each core in the Spark distributed computing engine is responsible for the computation of one unit. After any core has completed the computation of one analysis parameter for a flight, it will be assigned a new computation task. Computing resources are dynamically allocated according to tenant quotas to ensure performance stability when multiple tenants use the service concurrently.

12. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 11, characterized in that, The analysis parameter module stores the calculation results of the Spark operation script in a columnar database and establishes the analysis parameter library in the system. The analysis parameter library includes a standard analysis parameter library and a tenant-specific analysis parameter library. The standard analysis parameter library contains multiple analysis parameters that cover flight indicators throughout the entire flight process. The tenant-specific analysis parameter library is used to store the analysis parameters and calculation results customized by the tenant, thereby achieving tenant data isolation.

13. The civil aviation flight big data intensive service system supporting multi-tenancy and multi-source access according to claim 1, characterized in that, The analysis module is configured as follows: The system receives the flight or analysis parameter filtering conditions selected by the user on the query page, generates the corresponding SQL statement in the background and submits it to the database query engine, which executes the SQL statement. In the analysis and processing, it performs multi-dimensional and multi-scenario aggregate analysis based on flight operation data and the calculation results of the analysis parameter module. It also supports the generation of personalized analysis reports according to tenant needs, and only displays data and analysis results within the tenant's permission scope.

Images