Data real-time stream processing method and system based on civil aviation data intermediate station

Abstract

The invention relates to the technical field of civil aviation data processing, in particular to a data real-time stream processing method and system based on a civil aviation data intermediate station, and the method comprises the steps: obtaining a civil aviation multi-source data stream; performing adaptive window division on the civil aviation multi-source data stream based on a data type; carrying out processing priority distribution on the data flow by adopting a priority weighting algorithm; constructing a layered pipeline processing architecture and executing parallel computing; performing real-time data aggregation analysis by using a sliding time window; identifying abnormal data by adopting an abnormal detection algorithm based on a statistical model; generating predictive maintenance early warning information; and dynamically adjusting and calculating the resource allocation proportion according to the data flow characteristics. According to the method, real-time processing of high-speed data streams is realized through self-adaptive window division and a priority processing mechanism, system response delay is remarkably reduced, and civil aviation operation situation awareness and decision support capabilities are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of civil aviation data processing, and particularly relates to a data real-time stream processing method and system based on a civil aviation data center. BACKGROUND

[0002] As a core platform for integrating, managing and analyzing multi-source data in the civil aviation industry, the civil aviation data center bears the task of converging and processing massive heterogeneous data. With the expansion of civil aviation business scale and the improvement of intelligent degree, various real-time data streams are increasing, including flight status data, onboard equipment data, flight operation data and passenger flow data, etc. These data have obvious real-time characteristics, which puts higher requirements on the response speed and throughput capacity of the processing system.

[0003] The traditional civil aviation data processing method is mainly based on the batch processing mode, which periodically analyzes the accumulated data in batches and cannot meet the needs of real-time business scenarios. For example, the health status monitoring of onboard equipment, flight delay prediction and air traffic flow management, etc. key businesses all rely on the rapid processing and analysis of real-time data streams; in addition, different types of data streams have significant differences in rate, importance and processing complexity, and a unified processing strategy cannot balance efficiency and resource utilization. At present, although there are some stream processing technologies in the industry, their application in the civil aviation field still faces many challenges: first, they cannot effectively cope with the complexity of civil aviation multi-source heterogeneous data streams; second, they lack adaptive processing mechanisms for different types of data streams; third, they lack the ability to respond to sudden peak traffic; fourth, they cannot balance the relationship between processing delay and throughput under limited resources.

[0004] Therefore, how to build a method and system that can efficiently process civil aviation multi-source real-time data streams and dynamically adjust the processing strategy according to the data characteristics has become a technical problem to be solved. SUMMARY

[0005] (1) Technical problem to be solved The purpose of the present application is to provide a data real-time stream processing method and system based on a civil aviation data center, to solve the problems in the prior art that civil aviation multi-source heterogeneous real-time data streams cannot be effectively processed, the processing delay is high, the resource utilization is low, and there is a lack of adaptive ability, etc.

[0006] (2) Technical scheme To achieve the above-mentioned purpose, the present application provides a data real-time stream processing method based on a civil aviation data center, which comprises the following steps: S1, acquiring civil aviation multi-source data streams, the civil aviation multi-source data streams including flight status data streams, onboard equipment data streams, flight operation data streams and passenger flow data streams.

[0007] S2, performing adaptive window division on the civil aviation multi-source data stream based on data types, the adaptive window division method comprising: using a time-based sliding window for high-speed data stream, using a quantity-based rolling window for medium-speed data stream, and using an event-triggered cumulative window for low-speed data stream.

[0008] S3, performing processing priority allocation on the civil aviation multi-source data stream using a priority weighting algorithm, the processing priority allocation being based on data timeliness, safety criticality, and business value.

[0009] S4, constructing a hierarchical pipeline processing architecture and performing multi-source data stream parallel computing, the hierarchical pipeline processing architecture comprising a data access layer, a preprocessing layer, an analysis processing layer, and a result output layer, each layer using multi-node parallel computing.

[0010] S5, performing real-time data aggregation analysis on data streams of the same type using a sliding time window to generate state indicators, trend changes, and correlation characteristics.

[0011] S6, performing monitoring on real-time data streams using an abnormality detection algorithm based on a statistical model to identify abnormal data deviating from normal patterns and sending real-time early warnings when abnormal data is detected.

[0012] S7, generating maintenance warning information using a predictive model based on the analysis results of the on-board equipment data stream, the maintenance warning information comprising component fault risk assessment, optimal maintenance time window recommendation, and resource allocation suggestion.

[0013] S8, dynamically adjusting the allocation proportion of computing resources according to the data stream variation characteristics, automatically expanding the resource configuration of the corresponding processing unit when the data traffic of a certain type surges, and ensuring that the overall processing capacity of the system matches the data traffic variation.

[0014] Further, the high-speed data stream is a data stream with a data generation frequency ; the medium-speed data stream is a data stream with a data generation frequency between ; and the low-speed data stream is a data stream with a data generation frequency .

[0015] Further, in the method of performing adaptive window division on the civil aviation multi-source data stream based on data types, the calculation method of the sliding window size of the high-speed data stream is as follows: ; wherein represents the data stream rate, with the unit of piece / second; represents the average processing time of a single piece of data, with the unit of second; represents a window adjustment coefficient, with the value range of .

[0016] Further, the calculation formula of the priority weighting algorithm is: ; wherein, represents the processing priority of the first class data stream; represents the timeliness score of the first class data stream, and the value range is ; represents the safety criticality score of the first class data stream, and the value range is ; represents the service value score of the first class data stream, and the value range is ; , , respectively are the weight coefficients of timeliness, safety criticality and service value, and satisfy .

[0017] Further, the parallel computing resource allocation of each layer in the hierarchical pipeline processing architecture follows the constraint condition: ; and ; wherein, represents the layer number of the processing architecture, , respectively correspond to the data access layer, the preprocessing layer, the analysis processing layer and the result output layer; represents the number of computing resources allocated to the first layer; represents the total number of system computing resources; represents the minimum computing resource number of each layer.

[0018] Further, the dynamic adjustment formula of the sliding step length of the sliding time window is: ; wherein, represents the expected window data overlap rate, and the value range is ; represents the step length adjustment factor, and the value range is .

[0019] Further, the abnormality detection algorithm based on the statistical model includes the following steps: S6.1, extracting feature vectors from historical data to build a multi-dimensional normal behavior model; S6.2, calculating the Mahalanobis distance between real-time data points and the normal behavior model: , wherein, represents the feature vector of the real-time data point, represents the mean vector of the normal behavior model, a covariance matrix representing the normal behavior model; S6.3, setting an adaptive threshold , the data point is marked as abnormal; S6.4, clustering analysis is performed on the continuously occurring abnormal points to evaluate the severity of the abnormality and generate graded warning information.

[0020] Further, the formula for calculating the risk of component failure by the predictive model is: wherein, represents the time when the internal component fails; represents the time at which the failure rate function, ; wherein, represents the baseline failure rate; represents the value of the th influencing factor at time ; represents the weight coefficient of the th influencing factor; represents the total number of influencing factors.

[0021] Further, the method for dynamically adjusting the allocation ratio of computing resources comprises: S8.1, monitoring the real-time traffic and processing delay of each type of data stream; S8.2, calculating the resource utilization : ; wherein, represents the set of data streams processed by the th layer; represents the amount of computation required to process one piece of data stream of the th type; represents the processing capacity of the th layer per unit resource; represents the amount of resources allocated to the th layer at time ; S8.3, when or , increase the resource allocation of the th layer: ; S8.4, when and all , reduce the resource allocation of the th layer: ; wherein,​ represents a resource utilization upper limit; represents a resource utilization lower limit; represents a maximum allowed processing delay; represents a target processing delay; represents a maximum proportion of a single resource adjustment.

[0022] Based on the same inventive concept, in another aspect, the present application also provides a civil aviation data center-based data real-time stream processing system, which comprises a data access module, a window division module, a priority allocation module, a parallel processing module, a data analysis module, an anomaly detection module, a prediction and early warning module, and a resource scheduling module.

[0023] The data access module is configured to obtain civil aviation multi-source data streams, wherein the civil aviation multi-source data streams comprise flight state data streams, airborne equipment data streams, flight operation data streams, and passenger flow data streams.

[0024] The window division module is configured to perform adaptive window division on the civil aviation multi-source data streams based on data types.

[0025] The priority allocation module is configured to perform priority allocation on the civil aviation multi-source data streams by using a priority weighting algorithm.

[0026] The parallel processing module is configured to construct a hierarchical pipeline processing architecture and perform parallel computation.

[0027] The data analysis module is configured to perform real-time data aggregation analysis on data streams of the same type by using a sliding time window, and generate state indicators, trend changes, and correlation features.

[0028] The anomaly detection module is configured to monitor real-time data streams by using an anomaly detection algorithm based on a statistical model, identify abnormal data deviating from a normal mode, and send real-time early warning to a related system when abnormal data is detected.

[0029] The prediction and early warning module is configured to generate maintenance early warning information by using a predictive model based on analysis results of the airborne equipment data streams.

[0030] The resource scheduling module is configured to dynamically adjust a computing resource allocation proportion according to data stream variation characteristics.

[0031] (3) Advantages Compared with the prior art, the present application has the following advantages: 1. Through the adaptive window division mechanism, different window processing strategies are adopted for different types and rates of data streams, realizing efficient processing of high-speed data streams, medium-speed data streams and low-speed data streams, and improving the overall processing capacity of the system. High-speed data streams adopt time-based sliding windows, which can capture transient changes in data streams in time; medium-speed data streams adopt quantity-based rolling windows, balancing processing timeliness and resource utilization; low-speed data streams adopt event-triggered cumulative windows, avoiding unnecessary processing overhead.

[0032] 2. The application priority weighting algorithm is applied, the data timeliness, safety criticality and business value are comprehensively considered, different processing priorities are allocated to different data streams, the key business data is ensured to be processed preferentially, the response delay of the system to safety critical data is significantly reduced, and the response ability of the system to emergency is improved; a hierarchical pipeline processing architecture is constructed, the data processing task is divided into four stages of data access, preprocessing, analysis processing and result output, and multi-node parallel computing is applied in each stage, the parallel processing capacity of hardware resources is fully utilized, the performance bottleneck problem in the traditional serial processing mode is solved, and high-throughput, low-delay real-time processing capability is realized.

[0033] 3. The dynamic resource allocation mechanism can intelligently adjust the allocation proportion of computing resources according to the real-time change characteristics of data streams, optimize the resource utilization rate while ensuring the processing performance. When a certain type of data flow increases, the system can automatically increase the resource configuration of the corresponding processing unit; and during the low load period, the excess resources are released for other tasks, realizing the economic and efficient use of computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A flow chart of a data real-time stream processing method based on a civil aviation data middle station of the present application; Figure 2 A composition schematic diagram of a data real-time stream processing system based on a civil aviation data middle station of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the present application is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0036] Embodiment 1 As shown in Figure 1 As shown in Figure 1As shown, the embodiment provides a data real-time flow processing method based on a civil aviation data center, which comprises the following steps: S1, acquiring civil aviation multi-source data flow, which comprises flight state data flow, airborne equipment data flow, flight operation data flow and passenger flow data flow.

[0037] Among them, the flight state data flow includes real-time flight parameters such as aircraft position, height, speed and attitude, and the data generation frequency is about 10Hz; the airborne equipment data flow includes engine parameters, hydraulic system pressure, electrical system state and other equipment operation parameters, and the data generation frequency is about 5Hz; the flight operation data flow includes flight plan, actual take-off and landing time, delay information and other operation data, and the data update frequency is triggered according to the event, with an average of about 0.05Hz; the passenger flow data flow includes boarding gate passenger quantity, security channel passenger flow and other information, and the data update frequency is about 0.1Hz.

[0038] S2, based on the data type, the civil aviation multi-source data flow is adaptively windowed, and the adaptive windowing method comprises: for high-speed data flow, a time-based sliding window is used, for medium-speed data flow, a quantity-based rolling window is used, and for low-speed data flow, an event-triggered cumulative window is used.

[0039] According to the characteristics of the data flow, the above data flow is divided into three categories: high-speed data flow (frequency>1Hz), medium-speed data flow (0.1Hz≤frequency≤1Hz) and low-speed data flow (frequency<0.1Hz). For high-speed data flow (such as flight state data flow, airborne equipment data flow), a time-based sliding window is used, and the window size is dynamically adjusted according to the data flow rate and processing capacity. For example, for flight state data flow, the window size is set to 5 seconds, and about 50 data records are contained in the window; for medium-speed data flow (such as passenger flow data flow), a quantity-based rolling window is used, and the number of data records in the fixed window is 100; for low-speed data flow (such as flight operation data flow), an event-triggered cumulative window is used, which triggers data processing after a specific event (such as flight state change) occurs.

[0040] The calculation method of the sliding window size of high-speed data flow is: ; wherein, represents the data flow rate, with units of pieces / second; represents the average processing time of a single data, with units of seconds; represents the window adjustment coefficient, with a value range of .

[0041] ​S3, the priority weighting algorithm is adopted to assign processing priority to the civil aviation multi-source data stream, and the processing priority assignment is based on comprehensive calculation of data timeliness, safety criticality and business value.

[0042] Based on the three dimensions of data timeliness, safety criticality and business value, different data streams are assigned processing priority; the calculation formula of the priority weighting algorithm is: ; wherein, represents the processing priority of the data stream of the class; represents the timeliness score of the data stream of the class, and the value range is ; represents the safety criticality score of the data stream of the class, and the value range is ; represents the business value score of the data stream of the class, and the value range is ; , , are the weight coefficients of timeliness, safety criticality and business value respectively, and satisfy .

[0043] For example, for the flight state data stream, , , , the weight coefficient , the processing priority is calculated to be 0.905; for the passenger flow data stream, , the processing priority is calculated to be 0.41.

[0044] S4, a hierarchical pipeline processing architecture is constructed and multi-source data stream parallel computing is performed, and the hierarchical pipeline processing architecture includes a data access layer, a preprocessing layer, an analysis processing layer and a result output layer, and each layer adopts multi-node parallel computing.

[0045] A four-layer pipeline processing architecture is constructed, including: Data access layer: responsible for receiving multi-source data stream, protocol analysis and data format conversion; Preprocessing layer: responsible for data cleaning, deduplication, missing value completion, etc.; Analysis processing layer: performs core processing tasks such as real-time data analysis, anomaly detection, prediction model calculation, etc.; Result output layer: responsible for formatting, storing and distributing the processing results.

[0046] Each layer adopts multi-node parallel computing, and the number of nodes is allocated according to the task complexity and data flow of the layer; for example, 8 parallel nodes are configured for the data access layer, 12 parallel nodes are configured for the preprocessing layer, 20 parallel nodes are configured for the analysis processing layer, and 6 parallel nodes are configured for the result output layer.

[0047] The parallel computing resource allocation of each layer in the hierarchical pipeline processing architecture follows the constraint condition: ; and ; wherein, represents the hierarchical number of the processing architecture, , respectively corresponding to the data access layer, the preprocessing layer, the analysis processing layer and the result output layer; represents the number of computing resources allocated to the first layer; represents the total number of system computing resources, for example, set to 50 computing nodes; represents the minimum number of computing resources of each layer, for example, set to 2 computing nodes.

[0048] S5, real-time data aggregation analysis is performed on the same type of data stream using a sliding time window to generate state indicators, trend changes and correlation characteristics.

[0049] For example, for engine parameters in the airborne equipment data stream, the mean, standard deviation, change rate and other statistical indicators of parameters such as vibration frequency, fuel flow and exhaust temperature are calculated within a 10-second sliding window; the correlation between parameters is analyzed, such as the correlation coefficient of fuel flow and exhaust temperature; the trend of parameters is identified, such as the growth trend of vibration amplitude.

[0050] The dynamic adjustment formula of the sliding step length of the sliding time window is: ; wherein, represents the expected window data overlap rate, which is in the range of ; represents the step length adjustment factor, which is in the range of .

[0051] S6, a statistical model-based anomaly detection algorithm is used to monitor real-time data streams to identify abnormal data that deviates from the normal mode, and send real-time early warning when abnormal data is detected.

[0052] The statistical model-based anomaly detection algorithm includes the following steps: S6.1, extract feature vectors from historical data to build a multi-dimensional normal behavior model; for example, for engine data, extract 10 key features such as vibration frequency, fuel flow, exhaust temperature and speed, and build a multi-dimensional normal behavior model based on the historical data of the past 30 days.

[0053] S6.2, Calculate the Mahalanobis distance between the real-time data points and the normal behavior model: ,in, The feature vector representing real-time data points, such as [vibration frequency = 120Hz, fuel flow rate = 850kg / h, exhaust temperature = 580℃, ...]; This represents the mean vector of the normal behavior model, i.e., the historical average value of each feature; The covariance matrix represents the normal behavior model and reflects the correlation between features.

[0054] S6.3, Set adaptive threshold ,when When this happens, the data point is marked as an anomaly; threshold The false alarm rate is automatically calculated based on the distribution characteristics of historical data and the expected false alarm rate, for example, set to the 99.7 percentile of the Mahalanobis distance distribution of normal samples.

[0055] S6.4 performs cluster analysis on consecutively occurring anomalies to assess the severity of the anomalies and generate tiered early warning information.

[0056] For example, a high-level warning is generated when three consecutive data points exceed the threshold and the Mahalanobis distance shows an increasing trend; a low-level warning is generated when a single data point exceeds the threshold but subsequently returns to normal.

[0057] S7. Based on the analysis results of the airborne equipment data stream, a predictive model is used to generate maintenance early warning information, which includes component failure risk assessment, optimal maintenance time window recommendation, and resource allocation suggestions.

[0058] For example, a predictive model is applied to the analysis results of airborne equipment data streams to calculate component failure risks and generate maintenance early warning information: The formula for calculating component failure risk using a predictive model is as follows: ,in, Indicates time The probability of internal component failure; Indicates time The failure rate function at the location, ;in, Indicates the baseline failure rate; Indicates the first Each influencing factor over time The value at; Indicates the first The weighting coefficients of each influencing factor; This indicates the total number of influencing factors.

[0059] For a certain type of engine turbine blade, the baseline failure rate / hour, influencing factors include cumulative operating time, exhaust temperature deviation, vibration frequency deviation, etc.; when the probability of failure in the next 7 days is calculated. When the rate exceeds 5%, the system generates a maintenance warning and recommends that an inspection be carried out during the next appropriate flight interval.

[0060] S8 dynamically adjusts the allocation ratio of computing resources based on the characteristics of data flow changes. When a certain type of data traffic surges, it automatically expands the resource configuration of the corresponding processing unit to ensure that the overall processing capacity of the system matches the changes in data traffic.

[0061] Methods for dynamically adjusting the allocation ratio of computing resources include: S8.1 monitors the real-time traffic of various data streams. and processing delay The system calculates the inflow rate and average processing latency of each data stream every 10 seconds.

[0062] S8.2, Calculate resource utilization rate : ;in, Indicates the first The data stream collection processed by the layer; Indicates processing a first The computational cost required for data streams; Indicates the first Processing capacity of resources per layer; express Time assigned to the first The amount of resources in the layer.

[0063] S8.3, when or At that time, add the first Layer resource allocation: For example, setting , When the resource utilization rate of the analysis and processing layer At that time, the system will increase the resource allocation for that layer by approximately 12%.

[0064] S8.4, when And all At that time, reduce the number of Layer resource allocation: ;in, Indicates the upper limit of resource utilization; Indicates the lower limit of resource utilization; Indicates the maximum allowed processing delay; This indicates a delay in target processing; This indicates the maximum percentage that can be adjusted in a single instance.

[0065] For example, set , milliseconds, when the data access layer resource utilization and all data flow processing delay is less than the target value, the system will reduce the resource allocation of this layer, but not less than the minimum resource amount .

[0066] Embodiment 2 As Figure 2 shown, the embodiment provides a data real-time stream processing system based on civil aviation data center, which comprises a data access module, a window division module, a priority allocation module, a parallel processing module, a data analysis module, an anomaly detection module, a prediction and early warning module and a resource scheduling module.

[0067] The data access module is configured to obtain civil aviation multi-source data streams, wherein the civil aviation multi-source data streams comprise flight state data streams, airborne equipment data streams, flight operation data streams and passenger flow data streams.

[0068] The window division module is configured to perform adaptive window division on the civil aviation multi-source data streams based on data types.

[0069] The priority allocation module is configured to perform priority allocation on the civil aviation multi-source data streams by using a priority weighting algorithm.

[0070] The parallel processing module is configured to construct a hierarchical pipeline processing architecture and perform parallel computing.

[0071] The data analysis module is configured to perform real-time data aggregation analysis on data streams of the same type by using a sliding time window, and generate state indicators, trend changes and correlation characteristics.

[0072] The anomaly detection module is configured to monitor real-time data streams by using an anomaly detection algorithm based on a statistical model, identify abnormal data deviating from a normal mode, and send real-time early warning to a related system when detecting abnormal data.

[0073] The prediction and early warning module is configured to generate maintenance early warning information by using a predictive model based on analysis results of the airborne equipment data streams.

[0074] The resource scheduling module is configured to dynamically adjust a computing resource allocation ratio according to data stream variation characteristics.

[0075] It should be noted that, as for the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment related to the method, and will not be described in detail here.

[0076] The above detailed description of the specific embodiments of the present application has been given to understand the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A real-time data stream processing method based on a civil aviation data platform, characterized in that, The method includes the following steps: S1, acquire civil aviation multi-source data streams, which include flight status data streams, airborne equipment data streams, flight operation data streams, and passenger flow data streams; S2, perform adaptive window partitioning on the civil aviation multi-source data stream based on data type. The adaptive window partitioning method includes: using a time-based sliding window for high-speed data streams, using a quantity-based rolling window for medium-speed data streams, and using an event-triggered cumulative window for low-speed data streams. S3, a priority weighting algorithm is used to allocate processing priority to the multi-source data stream of civil aviation. The processing priority allocation is based on a comprehensive calculation of data timeliness, safety criticality and business value. S4, construct a hierarchical pipeline processing architecture and execute multi-source data stream parallel computing. The hierarchical pipeline processing architecture includes a data access layer, a preprocessing layer, an analysis and processing layer, and a result output layer. Each layer adopts multi-node parallel computing. S5 utilizes a sliding time window to perform real-time data aggregation and analysis on data streams of the same type, generating status indicators, trend changes, and correlation characteristics. S6 uses a statistical model-based anomaly detection algorithm to monitor the real-time data stream, identify abnormal data that deviates from the normal pattern, and send a real-time warning when abnormal data is detected. S7. Based on the analysis results of the airborne equipment data stream, a predictive model is used to generate maintenance early warning information, which includes component failure risk assessment, optimal maintenance time window recommendation, and resource allocation suggestions. S8 dynamically adjusts the allocation ratio of computing resources based on the characteristics of data flow changes. When a certain type of data traffic surges, it automatically expands the resource configuration of the corresponding processing unit to ensure that the overall processing capacity of the system matches the changes in data traffic.

2. The real-time data stream processing method based on a civil aviation data platform as described in claim 1, characterized in that, The high-speed data stream is the data generation frequency. The data stream; the medium-speed data stream is the data generation frequency at... The data stream between; the low-speed data stream is the data generation frequency. The data stream.

3. The real-time data stream processing method based on a civil aviation data platform as described in claim 2, characterized in that, In the method for adaptive window partitioning of multi-source civil aviation data streams based on data type, the sliding window size of the high-speed data stream... The calculation method is as follows: ;in, Indicates the data stream rate, measured in messages per second; This represents the average processing time for a single data entry, in seconds. This represents the window adjustment coefficient, with a value range of [value missing]. .

4. The real-time data stream processing method based on a civil aviation data platform as described in claim 3, characterized in that, The calculation formula for the priority weighting algorithm is as follows: ;in, Indicates the first Processing priority for data streams; Indicates the first The timeliness score for data streams ranges from [value range missing]. ; Indicates the first The security criticality score for data streams ranges from [value range missing]. ; Indicates the first The business value score for data streams ranges from [value range missing]. ; , , The weighting coefficients are timeliness, security criticality, and business value, respectively, and satisfy the following conditions: .

5. The real-time data stream processing method based on a civil aviation data platform as described in claim 4, characterized in that, The allocation of parallel computing resources in each layer of the hierarchical pipelined processing architecture follows the following constraints: ;and ;in, Indicates the hierarchical number of the processing architecture. These correspond to the data access layer, preprocessing layer, analysis and processing layer, and result output layer, respectively. Indicates assignment to the first The amount of computing resources required for each layer; This indicates the total amount of computing resources in the system; This indicates the minimum number of computing resources required for each layer.

6. The real-time data stream processing method based on a civil aviation data platform as described in claim 5, characterized in that, The sliding step size of the sliding time window The dynamic adjustment formula is: ;in, This represents the desired window data overlap rate, with a value range of [value missing]. ; This represents the step size adjustment factor, with a value range of [value range missing]. .

7. The real-time data stream processing method based on a civil aviation data platform as described in claim 6, characterized in that, The anomaly detection algorithm based on the statistical model includes the following steps: S6.1 Extract feature vectors from historical data to construct a multidimensional normal behavior model; S6.2, Calculate the Mahalanobis distance between the real-time data points and the normal behavior model: ,in, The feature vector representing a real-time data point. This represents the mean vector of the normal behavior model. The covariance matrix representing the normal behavior model; S6.3, Set adaptive threshold ,when When this happens, mark the data point as an anomaly; S6.4 performs cluster analysis on consecutively occurring anomalies to assess the severity of the anomalies and generate graded early warning information.

8. The real-time data stream processing method based on a civil aviation data platform as described in claim 7, characterized in that, The formula for calculating component failure risk using a predictive model is as follows: ,in, Indicates time The probability of internal component failure; Indicates time The failure rate function at the location, ;in, Indicates the baseline failure rate; Indicates the first Each influencing factor over time The value at; Indicates the first The weighting coefficients of each influencing factor; This indicates the total number of influencing factors.

9. A real-time data stream processing method based on a civil aviation data platform as described in claim 8, characterized in that, The method for dynamically adjusting the resource allocation ratio includes: S8.1 monitors the real-time traffic of various data streams. and processing delay ; S8.2, Calculate resource utilization rate : ;in, Indicates the first The data stream collection processed by the layer; Indicates processing one of the first The computational cost required for data streams; Indicates the first Processing capacity of resources per layer; express Time assigned to the first Layer resource quantity; S8.3, when or At that time, add the first Layer resource allocation: ； S8.4, when And all At that time, reduce the number of Layer resource allocation: ； in, Indicates the upper limit of resource utilization; Indicates the lower limit of resource utilization; Indicates the maximum allowed processing delay; This indicates a delay in target processing; This indicates the maximum percentage that can be adjusted in a single instance.

10. A real-time data stream processing system based on a civil aviation data platform, used to execute the method described in any one of claims 1-9, characterized in that, The system includes: a data access module, a window partitioning module, a priority allocation module, a parallel processing module, a data analysis module, an anomaly detection module, a prediction and early warning module, and a resource scheduling module; The data access module is used to acquire multi-source civil aviation data streams, which include flight status data streams, airborne equipment data streams, flight operation data streams, and passenger flow data streams. The window partitioning module is used to adaptively partition the civil aviation multi-source data stream based on the data type; The priority allocation module is used to allocate processing priorities for civil aviation multi-source data streams using a priority weighting algorithm; The parallel processing module is used to construct a hierarchical pipelined processing architecture and perform parallel computing; The data analysis module is used to perform real-time data aggregation and analysis on data streams of the same type using a sliding time window, and to generate status indicators, trend changes, and correlation characteristics. The anomaly detection module is used to monitor the real-time data stream using an anomaly detection algorithm based on a statistical model, identify abnormal data that deviates from the normal pattern, and send a real-time warning to the relevant system when abnormal data is detected. The prediction and early warning module is used to generate maintenance early warning information based on the analysis results of airborne equipment data streams using a predictive model. The resource scheduling module is used to dynamically adjust the allocation ratio of computing resources based on the changing characteristics of the data flow.

Images