A data real-time stream processing method and system based on a civil aviation data center platform

Abstract

The application relates to the technical field of civil aviation data processing, in particular to a data real-time flow processing method and system based on a civil aviation data center, which comprises the following steps: acquiring civil aviation multi-source data flow; performing self-adaptive window division on the civil aviation multi-source data flow based on data types; adopting a priority weighting algorithm to perform priority distribution on the data flow; constructing a hierarchical pipeline processing architecture and performing parallel calculation; performing real-time data aggregation analysis by using a sliding time window; adopting an abnormality detection algorithm based on a statistical model to identify abnormal data; generating predictive maintenance early warning information; and dynamically adjusting a calculation resource allocation proportion according to data flow characteristics. Through the self-adaptive window division and the priority processing mechanism, the application realizes real-time processing of high-speed data flow, significantly reduces system response delay, and improves civil aviation operation situation awareness and decision support capability.

Description

Technical Field

[0001] This invention relates to the field of civil aviation data processing technology, specifically a real-time data stream processing method and system based on a civil aviation data platform. Background Technology

[0002] As the core platform for integrating, managing, and analyzing multi-source data in the civil aviation industry, the civil aviation data platform undertakes the task of aggregating and processing massive amounts of heterogeneous data. With the expansion of civil aviation operations and the improvement of its intelligence level, various real-time data streams are increasing daily, including flight status data, onboard equipment data, flight operation data, and passenger flow data. These data have obvious real-time characteristics, placing high demands on the response speed and throughput capacity of the processing system.

[0003] Traditional civil aviation data processing methods are primarily based on batch processing, periodically analyzing accumulated data in batches, which cannot meet the needs of real-time business scenarios. For example, critical operations such as onboard equipment health monitoring, flight delay prediction, and air traffic flow management all rely on the rapid processing and analysis of real-time data streams. Furthermore, different types of data streams vary significantly in rate, importance, and processing complexity, making it difficult to balance efficiency and resource utilization with a uniform processing strategy. Currently, while some stream processing technologies exist in the industry, their application in the civil aviation sector still faces numerous challenges: first, they cannot effectively handle the complexity of multi-source, heterogeneous data streams in civil aviation; second, they lack adaptive processing mechanisms for different types of data streams; third, their ability to cope with sudden peak traffic is insufficient; and fourth, it is difficult to balance processing latency and throughput under limited resource conditions.

[0004] Therefore, how to build a method and system that can efficiently process multi-source real-time data streams from civil aviation and dynamically adjust processing strategies based on data characteristics has become an urgent technical problem to be solved. Summary of the Invention

[0005] (1) Technical problems to be solved

[0006] The purpose of this invention is to provide a real-time data stream processing method and system based on a civil aviation data platform, in order to solve the problems in the prior art such as the inability to effectively process multi-source heterogeneous real-time data streams in civil aviation, high processing latency, low resource utilization, and lack of adaptive capabilities.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention provides a real-time data stream processing method based on a civil aviation data platform, the method comprising the following steps:

[0009] S1, acquire civil aviation multi-source data streams, which include flight status data streams, airborne equipment data streams, flight operation data streams, and passenger flow data streams.

[0010] S2, perform adaptive window partitioning on the civil aviation multi-source data stream based on data type. The adaptive window partitioning method includes: using a time-based sliding window for high-speed data streams, using a quantity-based rolling window for medium-speed data streams, and using an event-triggered cumulative window for low-speed data streams.

[0011] S3. A priority weighting algorithm is used to allocate processing priorities for multi-source data streams in civil aviation. The processing priority allocation is based on a comprehensive calculation of data timeliness, safety criticality, and business value.

[0012] S4. Construct a hierarchical pipelined processing architecture and execute parallel computing of multi-source data streams. The hierarchical pipelined processing architecture includes a data access layer, a preprocessing layer, an analysis and processing layer, and a result output layer. Each layer adopts multi-node parallel computing.

[0013] S5 utilizes a sliding time window to perform real-time data aggregation and analysis on data streams of the same type, generating status indicators, trend changes, and correlation characteristics.

[0014] S6 employs an anomaly detection algorithm based on a statistical model to monitor the real-time data stream, identify abnormal data that deviates from the normal pattern, and send a real-time alert when abnormal data is detected.

[0015] S7. Based on the analysis results of the airborne equipment data stream, a predictive model is used to generate maintenance early warning information, which includes component failure risk assessment, optimal maintenance time window recommendation, and resource allocation suggestions.

[0016] S8 dynamically adjusts the allocation ratio of computing resources based on the characteristics of data flow changes. When a certain type of data traffic surges, it automatically expands the resource configuration of the corresponding processing unit to ensure that the overall processing capacity of the system matches the changes in data traffic.

[0017] Furthermore, the high-speed data stream refers to the data generation frequency. The data stream; the medium-speed data stream is the data generation frequency at... The data stream between; the low-speed data stream is the data generation frequency. The data stream.

[0018] Furthermore, in the method for adaptive window partitioning of multi-source civil aviation data streams based on data type, the sliding window size of the high-speed data stream... The calculation method is as follows: ;in, Indicates the data stream rate, measured in messages per second; This represents the average processing time for a single data entry, in seconds. This represents the window adjustment coefficient, with a value range of [value missing]. .

[0019] Furthermore, the calculation formula for the priority weighting algorithm is as follows: ;in, Indicates the first Processing priority for data streams; Indicates the first The timeliness score for data streams ranges from [value range missing]. ; Indicates the first The security criticality score for data streams ranges from [value range missing]. ; Indicates the first The business value score for data streams ranges from [value range missing]. ; , , The weighting coefficients are timeliness, security criticality, and business value, respectively, and satisfy the following conditions: .

[0020] Furthermore, the allocation of parallel computing resources in each layer of the hierarchical pipelined processing architecture follows the following constraints:

[0021] ;and ;in, Indicates the hierarchical number of the processing architecture. These correspond to the data access layer, preprocessing layer, analysis and processing layer, and result output layer, respectively. Indicates assignment to the first The amount of computing resources required for each layer; This indicates the total amount of computing resources in the system; This indicates the minimum number of computing resources required for each layer.

[0022] Furthermore, the sliding step size of the sliding time window The dynamic adjustment formula is: ;in, This represents the desired window data overlap rate, with a value range of [value missing]. ; This represents the step size adjustment factor, with a value range of [value range missing]. .

[0023] Furthermore, the anomaly detection algorithm based on the statistical model includes the following steps:

[0024] S6.1 Extract feature vectors from historical data to construct a multidimensional normal behavior model;

[0025] S6.2, Calculate the Mahalanobis distance between the real-time data points and the normal behavior model: ,in, The feature vector representing a real-time data point. This represents the mean vector of the normal behavior model. The covariance matrix representing the normal behavior model;

[0026] S6.3, Set adaptive threshold ,when When this happens, mark the data point as an anomaly;

[0027] S6.4 performs cluster analysis on consecutively occurring anomalies to assess the severity of the anomalies and generate graded early warning information.

[0028] Furthermore, the formula for calculating component failure risk using the predictive model is as follows: ,in, Indicates time The probability of internal component failure; Indicates time The failure rate function at the location, ;in, Indicates the baseline failure rate; Indicates the first Each influencing factor over time The value at; Indicates the first The weighting coefficients of each influencing factor; This indicates the total number of influencing factors.

[0029] Furthermore, the method for dynamically adjusting the allocation ratio of computing resources includes:

[0030] S8.1 monitors the real-time traffic of various data streams. and processing delay ;

[0031] S8.2, Calculate resource utilization rate : ;in, Indicates the first The data stream collection processed by the layer; Indicates processing one of the first The computational cost required for data streams; Indicates the first Processing capacity of resources per layer; express Time assigned to the first Layer resource quantity;

[0032] S8.3, when or At that time, add the first Layer resource allocation:

[0033] ;

[0034] S8.4, when And all At that time, reduce the number of Layer resource allocation:

[0035] ;

[0036] in, Indicates the upper limit of resource utilization; Indicates the lower limit of resource utilization rate; Indicates the maximum allowed processing delay; This indicates a delay in target processing; This indicates the maximum percentage that can be adjusted in a single instance.

[0037] Based on the same inventive concept, this invention also provides a real-time data stream processing system based on a civil aviation data platform. The system includes: a data access module, a window partitioning module, a priority allocation module, a parallel processing module, a data analysis module, an anomaly detection module, a prediction and early warning module, and a resource scheduling module.

[0038] The data access module is used to acquire multi-source civil aviation data streams, which include flight status data streams, airborne equipment data streams, flight operation data streams, and passenger flow data streams.

[0039] The window partitioning module is used to adaptively partition the civil aviation multi-source data stream based on the data type.

[0040] The priority allocation module is used to allocate processing priorities for civil aviation multi-source data streams using a priority weighting algorithm.

[0041] The parallel processing module is used to construct a hierarchical pipelined processing architecture and perform parallel computing.

[0042] The data analysis module is used to perform real-time data aggregation analysis on data streams of the same type using a sliding time window, and to generate status indicators, trend changes, and correlation characteristics.

[0043] The anomaly detection module is used to monitor the real-time data stream using an anomaly detection algorithm based on a statistical model, identify abnormal data that deviates from the normal pattern, and send a real-time warning to the relevant system when abnormal data is detected.

[0044] The prediction and early warning module is used to generate maintenance early warning information based on the analysis results of airborne equipment data streams using a predictive model.

[0045] The resource scheduling module is used to dynamically adjust the allocation ratio of computing resources based on the changing characteristics of the data flow.

[0046] (3) Beneficial effects

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] 1. By employing an adaptive windowing mechanism, different windowing strategies are used for different types and rates of data streams, achieving efficient processing of high-speed, medium-speed, and low-speed data streams and improving the overall processing capability of the system. High-speed data streams utilize a time-based sliding window to capture instantaneous changes in the data stream; medium-speed data streams use a quantity-based rolling window to balance processing timeliness and resource utilization; and low-speed data streams use an event-triggered cumulative window to avoid unnecessary processing overhead.

[0049] 2. By applying a priority-weighted algorithm that comprehensively considers data timeliness, security criticality, and business value, differentiated processing priorities are assigned to different data streams. This ensures that critical business data is processed first, significantly reducing the system's response latency to security-critical data and improving the system's responsiveness to emergencies. A hierarchical pipeline processing architecture is constructed, decomposing data processing tasks into four stages: data access, preprocessing, analysis, and result output. Multi-node parallel computing is applied in each stage, fully utilizing the parallel processing capabilities of hardware resources. This solves the performance bottleneck problem of the traditional serial processing mode and achieves high throughput and low latency real-time processing capabilities.

[0050] 3. The dynamic resource allocation mechanism can intelligently adjust the allocation ratio of computing resources based on the real-time changes in data flow, optimizing resource utilization while ensuring processing performance. When a surge in a certain type of data traffic occurs, the system can automatically increase the resource configuration of the corresponding processing unit; while during periods of low load, excess resources are released for other tasks, achieving economical and efficient utilization of computing resources. Attached Figure Description

[0051] Figure 1 This is a flowchart of a real-time data stream processing method based on a civil aviation data platform according to the present invention;

[0052] Figure 2 This is a schematic diagram of the composition of a real-time data stream processing system based on a civil aviation data platform according to the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of this invention, not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0054] Example 1

[0055] like Figure 1 As shown, Figure 1 As shown in the figure, this embodiment provides a real-time data stream processing method based on a civil aviation data platform. The method includes the following steps:

[0056] S1, acquire civil aviation multi-source data streams, which include flight status data streams, airborne equipment data streams, flight operation data streams, and passenger flow data streams.

[0057] The flight status data stream includes real-time flight parameters such as aircraft position, altitude, speed, and attitude, with a data generation frequency of approximately 10Hz; the airborne equipment data stream includes equipment operating parameters such as engine parameters, hydraulic system pressure, and electrical system status, with a data generation frequency of approximately 5Hz; the flight operation data stream includes operational data such as flight schedules, actual takeoff and landing times, and delay information, with a data update frequency based on event triggers, averaging approximately 0.05Hz; and the passenger flow data stream includes information such as the number of passengers at boarding gates and passenger flow at security checkpoints, with a data update frequency of approximately 0.1Hz.

[0058] S2, perform adaptive window partitioning on the civil aviation multi-source data stream based on data type. The adaptive window partitioning method includes: using a time-based sliding window for high-speed data streams, using a quantity-based rolling window for medium-speed data streams, and using an event-triggered cumulative window for low-speed data streams.

[0059] Based on data stream characteristics, the above data streams are divided into three categories: high-speed data streams (frequency > 1Hz), medium-speed data streams (0.1Hz ≤ frequency ≤ 1Hz), and low-speed data streams (frequency < 0.1Hz). For high-speed data streams (such as flight status data streams and airborne equipment data streams), a time-based sliding window is used, with the window size dynamically adjusted according to the data stream rate and processing capacity. For example, for flight status data streams, the window size is set to 5 seconds, containing approximately 50 data records. For medium-speed data streams (such as passenger flow data streams), a quantity-based scrolling window is used, with a fixed number of data records per window (100). For low-speed data streams (such as flight operation data streams), an event-triggered cumulative window is used, triggering data processing after a specific event (such as a flight status change) occurs.

[0060] Sliding window size for high-speed data streams The calculation method is as follows: ;in, Indicates the data stream rate, measured in messages per second; This represents the average processing time for a single data entry, in seconds. This represents the window adjustment coefficient, with a value range of [value missing]. .

[0061] S3. A priority weighting algorithm is used to allocate processing priorities for multi-source data streams in civil aviation. The processing priority allocation is based on a comprehensive calculation of data timeliness, safety criticality, and business value.

[0062] Based on three dimensions—data timeliness, security criticality, and business value—processing priorities are assigned to different data streams; the calculation formula for the priority weighting algorithm is as follows: ;in, Indicates the first Processing priority for data streams; Indicates the first The timeliness score for data streams ranges from [value range missing]. ; Indicates the first The security criticality score for data streams ranges from [value range missing]. ; Indicates the first The business value score for data streams ranges from [value range missing]. ; , , The weighting coefficients are timeliness, security criticality, and business value, respectively, and satisfy the following conditions: .

[0063] For example, for flight status data streams, , , Weighting coefficient The calculated processing priority is 0.905; for passenger flow data stream, The calculated processing priority is 0.41.

[0064] S4. Construct a hierarchical pipelined processing architecture and execute parallel computing of multi-source data streams. The hierarchical pipelined processing architecture includes a data access layer, a preprocessing layer, an analysis and processing layer, and a result output layer. Each layer adopts multi-node parallel computing.

[0065] Construct a four-layer pipelined processing architecture, including:

[0066] Data access layer: responsible for receiving multi-source data streams, performing protocol parsing, and data format conversion;

[0067] Preprocessing layer: responsible for data cleaning, deduplication, and filling in missing values;

[0068] Analysis and processing layer: performs core processing tasks such as real-time data analysis, anomaly detection, and predictive model calculation;

[0069] Output layer: responsible for formatting, storing and distributing results.

[0070] Each layer employs multi-node parallel computing, with the number of nodes allocated based on the complexity of the layer's tasks and the amount of data traffic. For example, the data access layer is configured with 8 parallel nodes, the preprocessing layer with 12 parallel nodes, the analysis and processing layer with 20 parallel nodes, and the result output layer with 6 parallel nodes.

[0071] In a layered pipelined processing architecture, the allocation of parallel computing resources in each layer follows certain constraints:

[0072] ;and ;in, Indicates the hierarchical number of the processing architecture. These correspond to the data access layer, preprocessing layer, analysis and processing layer, and result output layer, respectively. Indicates assignment to the first The amount of computing resources required for each layer; This indicates the total number of computing resources in the system, for example, set to 50 computing nodes; This indicates the minimum number of computing resources per layer, for example, set to 2 computing nodes.

[0073] S5 utilizes a sliding time window to perform real-time data aggregation and analysis on data streams of the same type, generating status indicators, trend changes, and correlation characteristics.

[0074] For example, for engine parameters in the airborne equipment data stream, the mean, standard deviation, and rate of change of parameters such as vibration frequency, fuel flow rate, and exhaust temperature are calculated within a 10-second sliding window; the correlation between parameters is analyzed, such as the correlation coefficient between fuel flow rate and exhaust temperature; and the trend of parameter changes is identified, such as the increasing trend of vibration amplitude.

[0075] Sliding step of the sliding time window The dynamic adjustment formula is: ;in, This represents the desired window data overlap rate, with a value range of [value missing]. ; This represents the step size adjustment factor, with a value range of [value range missing]. .

[0076] S6 employs an anomaly detection algorithm based on a statistical model to monitor the real-time data stream, identify abnormal data that deviates from the normal pattern, and send a real-time alert when abnormal data is detected.

[0077] Anomaly detection algorithms based on statistical models include the following steps:

[0078] S6.1 Extract feature vectors from historical data to construct a multidimensional normal behavior model; for example, extract 10 key features such as vibration frequency, fuel flow, exhaust temperature, and speed from engine data, and construct a multidimensional normal behavior model based on historical data from the past 30 days.

[0079] S6.2, Calculate the Mahalanobis distance between the real-time data points and the normal behavior model: ,in, The feature vector representing real-time data points, such as [vibration frequency = 120Hz, fuel flow rate = 850kg / h, exhaust temperature = 580℃, ...]; This represents the mean vector of the normal behavior model, i.e., the historical average value of each feature; The covariance matrix represents the normal behavior model and reflects the correlation between features.

[0080] S6.3, Set adaptive threshold ,when When this happens, the data point is marked as an anomaly; threshold The false alarm rate is automatically calculated based on the distribution characteristics of historical data and the expected false alarm rate, for example, set to the 99.7 percentile of the Mahalanobis distance distribution of normal samples.

[0081] S6.4 performs cluster analysis on consecutively occurring anomalies to assess the severity of the anomalies and generate graded early warning information.

[0082] For example, a high-level warning is generated when three consecutive data points exceed the threshold and the Mahalanobis distance shows an increasing trend; a low-level warning is generated when a single data point exceeds the threshold but subsequently returns to normal.

[0083] S7. Based on the analysis results of the airborne equipment data stream, a predictive model is used to generate maintenance early warning information, which includes component failure risk assessment, optimal maintenance time window recommendation, and resource allocation suggestions.

[0084] For example, a predictive model is applied to the analysis results of airborne equipment data streams to calculate component failure risks and generate maintenance early warning information:

[0085] The formula for calculating component failure risk using a predictive model is: ,in, Indicates time The probability of internal component failure; Indicates time The failure rate function at the location, ;in, Indicates the baseline failure rate; Indicates the first Each influencing factor over time The value at; Indicates the first The weighting coefficients of each influencing factor; This indicates the total number of influencing factors.

[0086] For a certain type of engine turbine blade, the baseline failure rate / hour, influencing factors include cumulative operating time, exhaust temperature deviation, vibration frequency deviation, etc.; when the probability of failure in the next 7 days is calculated. When the rate exceeds 5%, the system generates a maintenance warning and recommends that an inspection be carried out during the next appropriate flight interval.

[0087] S8 dynamically adjusts the allocation ratio of computing resources based on the characteristics of data flow changes. When a certain type of data traffic surges, it automatically expands the resource configuration of the corresponding processing unit to ensure that the overall processing capacity of the system matches the changes in data traffic.

[0088] Methods for dynamically adjusting the allocation ratio of computing resources include:

[0089] S8.1 monitors the real-time traffic of various data streams. and processing delay The system calculates the inflow rate and average processing latency of each data stream every 10 seconds.

[0090] S8.2, Calculate resource utilization rate : ;in, Indicates the first The data stream collection processed by the layer; Indicates processing one of the first The computational cost required for data streams; Indicates the first Processing capacity of resources per layer; express Time assigned to the first The amount of resources in the layer.

[0091] S8.3, when or At that time, add the first Layer resource allocation:

[0092] For example, setting , When the resource utilization rate of the analysis and processing layer At that time, the system will increase the resource allocation for that layer by approximately 12%.

[0093] S8.4, when And all At that time, reduce the number of Layer resource allocation:

[0094] ;in, Indicates the upper limit of resource utilization; Indicates the lower limit of resource utilization rate; Indicates the maximum allowed processing delay; This indicates a delay in target processing; This indicates the maximum percentage that can be adjusted in a single instance.

[0095] For example, setting , Milliseconds, when the data access layer resource utilization rate Furthermore, if the processing latency of all data streams is less than the target value, the system will reduce the resource allocation for that layer, but not below the minimum resource requirement. .

[0096] Example 2

[0097] like Figure 2 As shown in the figure, this embodiment provides a real-time data stream processing system based on a civil aviation data platform. The system includes: a data access module, a window partitioning module, a priority allocation module, a parallel processing module, a data analysis module, an anomaly detection module, a prediction and early warning module, and a resource scheduling module.

[0098] The data access module is used to acquire multi-source civil aviation data streams, which include flight status data streams, airborne equipment data streams, flight operation data streams, and passenger flow data streams.

[0099] The window partitioning module is used to adaptively partition the civil aviation multi-source data stream based on the data type.

[0100] The priority allocation module is used to allocate processing priorities for civil aviation multi-source data streams using a priority weighting algorithm.

[0101] The parallel processing module is used to construct a hierarchical pipelined processing architecture and perform parallel computing.

[0102] The data analysis module is used to perform real-time data aggregation analysis on data streams of the same type using a sliding time window, and to generate status indicators, trend changes, and correlation characteristics.

[0103] The anomaly detection module is used to monitor the real-time data stream using an anomaly detection algorithm based on a statistical model, identify abnormal data that deviates from the normal pattern, and send a real-time warning to the relevant system when abnormal data is detected.

[0104] The prediction and early warning module is used to generate maintenance early warning information based on the analysis results of airborne equipment data streams using a predictive model.

[0105] The resource scheduling module is used to dynamically adjust the allocation ratio of computing resources based on the changing characteristics of the data flow.

[0106] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0107] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time data stream processing method based on a civil aviation data platform, characterized in that, The method includes the following steps: S1, acquire civil aviation multi-source data streams, which include flight status data streams, airborne equipment data streams, meteorological environment data streams, flight operation data streams, and passenger flow data streams; S2, perform adaptive window partitioning on the civil aviation multi-source data stream based on data type. The adaptive window partitioning method includes: using a time-based sliding window for high-speed data streams, using a quantity-based rolling window for medium-speed data streams, and using an event-triggered cumulative window for low-speed data streams. S3, a priority weighting algorithm is used to allocate processing priority to the multi-source data stream of civil aviation. The processing priority allocation is based on a comprehensive calculation of data timeliness, safety criticality and business value. S4, construct a hierarchical pipeline processing architecture and execute multi-source data stream parallel computing. The hierarchical pipeline processing architecture includes a data access layer, a preprocessing layer, an analysis and processing layer, and a result output layer. Each layer adopts multi-node parallel computing. S5 utilizes a sliding time window to perform real-time data aggregation and analysis on data streams of the same type, generating status indicators, trend changes, and correlation characteristics. S6 uses a statistical model-based anomaly detection algorithm to monitor the real-time data stream, identify abnormal data that deviates from the normal pattern, and send a real-time warning when abnormal data is detected. S7. Based on the analysis results of the airborne equipment data stream, a predictive model is used to generate maintenance early warning information, which includes component failure risk assessment, optimal maintenance time window recommendation, and resource allocation suggestions. S8 dynamically adjusts the allocation ratio of computing resources according to the characteristics of data flow changes. When a certain type of data traffic surges, it automatically expands the resource configuration of the corresponding processing unit to ensure that the overall processing capacity of the system matches the changes in data traffic. The method for dynamically adjusting the resource allocation ratio includes: S8.1 monitors the real-time traffic of various data streams. and processing delay ; S8.2, Calculate resource utilization rate : ;in, Indicates the first The data stream collection processed by the layer; Indicates processing one of the first The computational cost required for data streams; Indicates the first Processing capacity of resources per layer; express Time assigned to the first Layer resource quantity; S8.3, when or At that time, add the first Layer resource allocation: ； S8.4, when And all At that time, reduce the number of Layer resource allocation: ； in, Indicates the upper limit of resource utilization; Indicates the lower limit of resource utilization rate; Indicates the maximum allowed processing delay; This indicates a delay in target processing; This indicates the maximum percentage that can be adjusted in a single instance.

2. The real-time data stream processing method based on a civil aviation data platform as described in claim 1, characterized in that, The high-speed data stream is the data generation frequency. The data stream; the medium-speed data stream is the data generation frequency at... The data stream between; the low-speed data stream is the data generation frequency. The data stream.

3. The real-time data stream processing method based on a civil aviation data platform as described in claim 2, characterized in that, Adaptive window partitioning is performed on the multi-source civil aviation data streams based on data type, and the sliding window size of the high-speed data stream is adjusted. The calculation method is as follows: ;in, Indicates the data stream rate, measured in messages per second; This represents the average processing time for a single data entry, in seconds. This represents the window adjustment coefficient, with a value range of [value missing]. .

4. The real-time data stream processing method based on a civil aviation data platform as described in claim 3, characterized in that, The calculation formula for the priority weighting algorithm is as follows: ;in, Indicates the first Processing priority for data streams; Indicates the first The timeliness score for data streams ranges from [value range missing]. ; Indicates the first The security criticality score for data streams ranges from [value range missing]. ; Indicates the first The business value score for data streams ranges from [value range missing]. ; , , The weighting coefficients are timeliness, security criticality, and business value, respectively, and satisfy the following conditions: .

5. The real-time data stream processing method based on a civil aviation data platform as described in claim 4, characterized in that, The allocation of parallel computing resources in each layer of the hierarchical pipelined processing architecture follows the following constraints: ;and ;in, Indicates the hierarchical number of the processing architecture. These correspond to the data access layer, preprocessing layer, analysis and processing layer, and result output layer, respectively. Indicates assignment to the first The amount of computing resources required for each layer; This indicates the total amount of computing resources in the system; This indicates the minimum number of computing resources required for each layer.

6. The real-time data stream processing method based on a civil aviation data platform as described in claim 5, characterized in that, The sliding step size of the sliding time window The dynamic adjustment formula is: ;in, This represents the desired window data overlap rate, with a value range of [value missing]. ; This represents the step size adjustment factor, with a value range of [value range missing]. .

7. The real-time data stream processing method based on a civil aviation data platform as described in claim 6, characterized in that, The anomaly detection algorithm based on the statistical model includes the following steps: S6.1 Extract feature vectors from historical data to construct a multidimensional normal behavior model; S6.2, Calculate the Mahalanobis distance between the real-time data points and the normal behavior model: ,in, The feature vector representing a real-time data point. This represents the mean vector of the normal behavior model. The covariance matrix representing the normal behavior model; S6.3, Set adaptive threshold ,when When this happens, mark the data point as an anomaly; S6.4 performs cluster analysis on consecutively occurring anomalies to assess the severity of the anomalies and generate graded early warning information.

8. The real-time data stream processing method based on a civil aviation data platform as described in claim 7, characterized in that, The formula for calculating component failure risk using a predictive model is as follows: ,in, Indicates time The probability of internal component failure; Indicates time The failure rate function at the location, ;in, Indicates the baseline failure rate; Indicates the first Each influencing factor over time The value at; Indicates the first The weighting coefficients of each influencing factor; This indicates the total number of influencing factors.

9. A real-time data stream processing system based on a civil aviation data platform, used to execute the method described in any one of claims 1-8, characterized in that, The system includes: a data access module, a window partitioning module, a priority allocation module, a parallel processing module, a data analysis module, an anomaly detection module, a prediction and early warning module, and a resource scheduling module; The data access module is used to acquire multi-source civil aviation data streams, which include flight status data streams, airborne equipment data streams, meteorological environment data streams, flight operation data streams, and passenger flow data streams. The window partitioning module is used to adaptively partition the civil aviation multi-source data stream based on the data type; The priority allocation module is used to allocate processing priorities for civil aviation multi-source data streams using a priority weighting algorithm; The parallel processing module is used to construct a hierarchical pipelined processing architecture and perform parallel computing; The data analysis module is used to perform real-time data aggregation and analysis on data streams of the same type using a sliding time window, and to generate status indicators, trend changes, and correlation characteristics. The anomaly detection module is used to monitor the real-time data stream using an anomaly detection algorithm based on a statistical model, identify abnormal data that deviates from the normal pattern, and send a real-time warning to the relevant system when abnormal data is detected. The prediction and early warning module is used to generate maintenance early warning information based on the analysis results of airborne equipment data streams using a predictive model. The resource scheduling module is used to dynamically adjust the allocation ratio of computing resources based on the changing characteristics of the data flow.

Images