Airport flight data fusion processing method and system based on timestamps

Abstract

The invention belongs to the technical field of airport operation data processing, and discloses an airport flight data fusion processing method and system based on timestamps. According to the method, data streams of different data sources are captured in real time through a message queue, and a database is constructed; the captured database is analyzed, key data fields are extracted, and preprocessing operation is carried out; synchronizing and sorting the analyzed data by using a timestamp; performing data processing in real time according to a preset fusion rule; and storing the fused real-time data into a database, and carrying out data analysis and monitoring on a fusion result to find potential problems. According to the method, the consistency and accuracy of the data in the fusion process are ensured, and data conflicts and errors are avoided. According to the method, configuration of various data sources and fusion rules is supported, and a user can flexibly select the data sources and set the fusion rules according to actual requirements so as to meet data processing requirements in different scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of airport operation data processing, and in particular relates to a timestamp-based airport flight data fusion processing method and system. Background Art

[0002] In today's digital age, with the widespread application of technologies such as big data, cloud computing, and the Internet of Things, the amount of data generated by various data sources has exploded. These data sources may come from different systems, devices, or networks, and their data formats, quality, and timestamp standards vary. In order to achieve effective use and in-depth analysis of data, it is necessary to fuse these data from multiple channels. However, traditional data fusion methods often have problems such as low processing efficiency and difficulty in ensuring data consistency, especially when processing data with complex time attributes.

[0003] Data fusion technology is particularly important in the fields of distributed systems, database systems, log processing, version control systems, etc. These systems are often composed of multiple nodes or components, each of which may generate a large amount of data. In order to ensure the consistency and integrity of the data, it is necessary to effectively fuse the data. The timestamp-based data fusion technology was developed to meet this demand.

[0004] Through the above analysis, the problems and defects of the existing technology are: the existing data fusion technology has problems such as low processing efficiency and poor data consistency, and improves the accuracy and real-time performance of data fusion. Summary of the invention

[0005] In order to overcome the problems existing in the related technology, the disclosed embodiment of the present invention provides an airport flight data fusion processing method and system based on timestamp, specifically relates to a cost calculation processing method based on abnormal airport flights, which can solve the interference of multi-source data access on airport flights and ensure data consistency.

[0006] The technical solution is as follows: A method for fusion processing of airport flight data based on timestamps, comprising: S1, API interface documents of different data sources use secure communication protocols to establish connection channels, obtain data from different data sources, and tag the flight messages with data sources and timestamps and send them to the message queue to achieve real-time capture of data streams from different data sources; S2, using the sequential nature of the message queue and using the single consumer mode to store the collected messages in the database for parsing, extracting key data fields in the message including flight date, flight number, departure station, arrival station, scheduled departure time, and data source identifier as the unique key of the data, defining other data items in the message as structured data and storing them in a relational table for preprocessing operations; S3, synchronizes and sorts the parsed data using timestamps; S4, real-time data processing according to preset fusion rules; S5, stores the fused real-time data in the database, analyzes and monitors the fusion results, and identifies potential problems.

[0007] In step S1 , data streams from different data sources are captured in real time, including: collecting flight data from multiple data sources in real time or periodically, including timestamp information.

[0008] In step S2, a preprocessing operation is performed, including: preliminary cleaning and verification of the data.

[0009] In step S3, the parsed data is synchronized and sorted using timestamps, including: timestamp alignment, based on timestamp information, time alignment of data from different data sources; wherein time alignment includes processing time differences and data delays.

[0010] In step S4, data processing is performed in real time according to preset fusion rules, including: processing the data using a fusion algorithm based on time alignment; the fusion algorithm includes data item weighting and key data update methods to eliminate data redundancy, reduce errors, and generate comprehensive flight information.

[0011] In step S5, data analysis and monitoring are performed on the fusion results, including: in the process of data fusion, the data conflicts are adjudicated by using the formulated conflict resolution strategy.

[0012] Furthermore, the formulated conflict resolution strategy is used for arbitration, including arbitration based on data source reliability and time freshness factors.

[0013] After step S5, the results need to be output and displayed: the integrated flight data is output in the form of structured data tables, API interfaces, CSV, JSON, and XML data formats, and displayed through a visual interface to monitor changes in basic flight information, resource usage information, and service guarantee information in real time, and to promptly detect and handle abnormal situations.

[0014] Furthermore, after the results are output and displayed, log recording and auditing are carried out to record the data source, processing time, and fusion result information during the data fusion process, extract data fusion result monitoring indicators, and use a visual interface to display the real-time data fusion processing trend. For fusion information that fails to be processed, configure monitoring alarm rules and push processing exception information in real time via email and DingTalk instant messaging.

[0015] Another object of the present invention is to provide a timestamp-based airport flight data fusion processing system, which implements the timestamp-based airport flight data fusion processing method, and the system includes: The database construction module captures data streams from different data sources in real time through message queues and builds databases; The data preprocessing module is used to parse the captured database, extract key data fields, and perform preprocessing operations; The data synchronization and sorting module uses timestamps to synchronize and sort the parsed data to ensure that the data is processed in chronological order; The data fusion module is used to process data in real time according to preset fusion rules to ensure the accuracy and consistency of the fusion results; The data analysis and monitoring module is used to store the fused real-time data in the database, analyze and monitor the fusion results, and discover potential problems.

[0016] In combination with all the above technical solutions, the beneficial effects of the present invention are as follows: The present invention significantly improves the efficiency of data processing and reduces processing delays by optimizing the links of real-time data stream capture, parsing and preprocessing, timestamp synchronization and sorting, and real-time data fusion algorithm. The present invention utilizes timestamp information and preset fusion rules to improve accuracy, ensure the consistency and accuracy of data during the fusion process, and avoid data conflicts and errors. The present invention adopts a modular design concept. In terms of scalability, the system is divided into multiple independent modules or services, which is convenient for expansion and upgrading according to actual needs. The present invention supports the configuration of multiple data sources and fusion rules. In terms of flexibility, users can flexibly select data sources and set fusion rules according to actual needs to meet data processing needs in different scenarios.

[0017] The present invention provides more comprehensive and accurate flight information, saves the system docking cost of the data demander, facilitates the airport to grasp the flight operation status more accurately, and realizes the comprehensive integration and analysis of flight data, providing data support for the digital transformation of the airport. The present invention provides more comprehensive and accurate flight information, personalized data services, provides data sources for passenger service systems, and expands the increase of services in the aviation field. The present invention overcomes the isolation of data. In traditional flight data processing, data from different sources are often processed and analyzed in isolation, resulting in the inability to fully utilize the correlation and complementarity between data. The comprehensiveness and comprehensiveness of data are achieved through this technical solution. The present invention overcomes the inconsistency of data. There are differences in the collection method, processing method and data format of different data sources, which may lead to inconsistency between data, which will affect the accuracy and reliability of data. Through this solution, the data is cleaned, converted and calibrated and other pre-processing operations are performed to eliminate the inconsistency between data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure; Figure 1 It is a flow chart of a method for fusion processing of airport flight data based on timestamps provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a method for fusion processing of airport flight data based on timestamps provided by an embodiment of the present invention; Figure 3 is a schematic diagram of an airport flight data fusion processing system based on timestamps provided by an embodiment of the present invention; In the figure: 1. Database construction module; 2. Data preprocessing module; 3. Data synchronization and sorting module; 4. Data fusion module; 5. Data analysis and monitoring module. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0020] The innovation of the present invention is that the present invention realizes real-time and accurate fusion of flight data by integrating multi-source flight data, including secondary radar data, automatic dependent surveillance-broadcast (ADS-B) data, flight plan data, etc., and using timestamp as a key attribute. The introduction of timestamp solves the inconsistency of data in the time dimension, and builds a batch processing framework based on Spring Batch to realize fast and real-time processing of flight data. The processing of outliers and missing values ​​in the data is completed through the functions of data cleaning, verification and conversion. The use of visual display and multi-dimensional data reports provides users with intuitive and clear data views and data analysis results.

[0021] Embodiment 1, as Figure 1 As shown, the airport flight data fusion processing method based on timestamp provided by the embodiment of the present invention includes: S1, API interface documents of different data sources use secure communication protocols to establish connection channels, obtain data from different data sources, and tag the flight messages with data sources and timestamps and send them to the message queue to achieve real-time capture of data streams from different data sources; S2, using the sequential nature of the message queue and using the single consumer mode to store the collected messages in the database for parsing, extracting key data fields in the message including flight date, flight number, departure station, arrival station, scheduled departure time, and data source identifier as the unique key of the data, defining other data items in the message as structured data and storing them in a relational table for preprocessing operations; S3, synchronizes and sorts the parsed data using timestamps; S4, real-time data processing according to preset fusion rules; S5, stores the fused real-time data in the database, analyzes and monitors the fusion results, and identifies potential problems.

[0022] For example, Figure 2 This is the principle of the airport flight data fusion processing method based on timestamp provided by an embodiment of the present invention.

[0023] Exemplarily, in step S1 , capturing data streams from different data sources in real time includes collecting flight data from multiple data sources in real time or periodically, including timestamp information.

[0024] In step S2, the preprocessing operation includes: preliminary cleaning and verification of the data to ensure the integrity and correctness of the data.

[0025] (1) Remove duplicate data: Identify and delete duplicate data records by comparing the key fields that uniquely identify the data, such as flight date, flight number, departure airport, arrival airport, scheduled departure time, and data source.

[0026] (2) Handling missing values: Use strategies such as computational filling and flight data relevance (e.g., the arrival and departure parking positions of the same flight are consistent, and the landing time of the arrival flight is consistent with the landing time of the next stop of the previous flight) to handle or correct erroneous values.

[0027] (3) Correct erroneous data: Use data rules, regular expressions, etc. to identify and correct errors in the data, such as: check-in time is earlier than boarding time, boarding time is earlier than take-off time, and flight times at foreign airports are converted according to corresponding time zones.

[0028] (4) Data type conversion and formatting: Format the data, such as removing extra spaces, special characters, punctuation marks, etc. Convert the data to the correct data type, such as converting a string type date to a date type, converting a string type number to a number type, etc.

[0029] In step S3, the parsed data is synchronized and sorted using timestamps, including timestamp alignment. The order of messages from different data sources is controlled by using the first-in-first-out order of the message queue to ensure that all data is stored one by one in chronological order. With the help of the clock server, the uniqueness of the recording system time is ensured, and the data from different data sources are time-aligned to ensure that all data are compared and integrated within the same time frame.

[0030] In step S4, data fusion processing: on the basis of time alignment, define the weight of the data source to which each data item belongs. When the data items of the same flight have values ​​in different data sources at the same time, the value source is determined according to the set weight; for the same flight, when multiple identical flights appear within the set time window, randomly select a flight data, match the unique identification keyword with the flight, and perform logical deletion on the redundant flights. The relevant data items are assigned to the valid flight data according to the weight ratio. When the flight is updated again (including the same flight that has been logically deleted), the unique identification keyword of the original flight is replaced with the updated flight unique identification, and the synchronization is completed. Update flight-related data items to keep each flight with only one unique identifier to eliminate data redundancy, reduce errors, and generate comprehensive flight information, including basic flight information (flight date, flight number, departure station, destination station, aircraft type, aircraft number, flight mission, planned take-off time, expected take-off time, actual take-off time, planned landing time, expected landing time, actual landing time, flight status), resource usage information (check-in counter, boarding gate, baggage carousel, jet bridge, aircraft stand, terminal building, etc.), service guarantee information (cabin door opening and closing time, aircraft push-in and exit time, catering start and end time, aircraft taxi time, etc.), etc.

[0031] In step S5, the fusion results are analyzed and monitored. During the data fusion process, due to the different batches of fused data, a new data message will be generated in each polling cycle to resolve possible data conflicts (such as inconsistent descriptions of the same flight status by different data sources). At this time, it is necessary to define a fusion timestamp for each fused data, monitor the update frequency of the fusion batch, ensure that the latest data is obtained each time, clean up and archive outdated or no longer needed data, and reduce conflicts. Standardize the encoding of the same data items from different data sources, and unify inconsistent data description information when fusion is performed. Add query functions for fused data results, and manually intervene and make decisions for data with complex relationships.

[0032] After step S5, the results need to be output and displayed: the integrated flight data is output in the form of structured data tables, API interfaces, specific data formats (such as CSV, JSON, XML), etc., and displayed through a visual interface to monitor the changes in basic flight information, resource usage information (check-in counters, boarding gates, baggage carousels, jet bridges, aircraft stands, etc.), and service guarantee information (cabin door opening and closing time, aircraft entry and exit time, catering end time, aircraft taxiing time, etc.) in real time, and abnormal situations are discovered and handled in a timely manner.

[0033] Perform log recording and auditing: record key information in the data fusion process, such as data source, processing time, and fusion results, extract data fusion result monitoring indicators, use a visual interface to display real-time data fusion processing trends, configure monitoring alarm rules for failed fusion information, and push abnormal processing information in real time through instant messaging tools such as email and DingTalk.

[0034] Embodiment 2, as Figure 3 As shown, the airport flight data fusion processing system based on timestamp provided by the embodiment of the present invention includes: Database construction module 1 is used to capture data streams from different data sources in real time through message queues and build a database; The data preprocessing module 2 is used to parse the captured database, extract key data fields, and perform preprocessing operations.

[0035] The data synchronization and sorting module 3 is used to synchronize and sort the parsed data using timestamps to ensure that the data is processed in chronological order.

[0036] The data fusion module 4 is used to perform data processing in real time according to preset fusion rules to ensure the accuracy and consistency of the fusion results.

[0037] The data analysis and monitoring module 5 is used to store the fused real-time data into a database and perform data analysis and monitoring on the fusion results to facilitate the discovery of potential problems.

[0038] The technical solution provided by the present invention uses timestamp as a key attribute to fuse information from different data sources, enriches the data range, and reduces data processing delays; through precise alignment of timestamps, improves data accuracy and reduces data errors and inconsistencies; through flexible configuration of data fusion strategies, realizes personalized data processing and analysis; through real-time monitoring, multi-dimensional data analysis and alarm reminders, improves data quality and data reliability; uses visual display and multi-dimensional data reports to provide users with intuitive and clear data views and data analysis results.

[0039] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for fusion processing of airport flight data based on timestamp, characterized in that: The method includes: S1, API interface documents of different data sources use secure communication protocols to establish connection channels, obtain data from different data sources, and tag the flight messages with data sources and timestamps and send them to the message queue to achieve real-time capture of data streams from different data sources; S2, using the sequential nature of the message queue and using the single consumer mode to store the collected messages in the database for parsing, extracting key data fields in the message including flight date, flight number, departure station, arrival station, scheduled departure time, and data source identifier as the unique key of the data, defining other data items in the message as structured data and storing them in a relational table for preprocessing operations; S3, synchronizes and sorts the parsed data using timestamps; S4, real-time data processing according to preset fusion rules; S5, stores the fused real-time data in the database, analyzes and monitors the fusion results, and identifies potential problems.

2. The airport flight data fusion processing method based on timestamp according to claim 1 is characterized in that: In step S1 , data streams from different data sources are captured in real time, including: collecting flight data from multiple data sources in real time or periodically, including timestamp information.

3. The airport flight data fusion processing method based on timestamp according to claim 1 is characterized in that: In step S2, a preprocessing operation is performed, including: preliminary cleaning and verification of the data.

4. The airport flight data fusion processing method based on timestamp according to claim 1 is characterized in that: In step S3, the parsed data is synchronized and sorted using timestamps, including: timestamp alignment, based on timestamp information, time alignment of data from different data sources; wherein time alignment includes processing time differences and data delays.

5. The airport flight data fusion processing method based on timestamp according to claim 1 is characterized in that: In step S4, data processing is performed in real time according to preset fusion rules, including: processing the data using a fusion algorithm based on time alignment; the fusion algorithm includes data item weighting and key data update methods to eliminate data redundancy, reduce errors, and generate comprehensive flight information.

6. The airport flight data fusion processing method based on timestamp according to claim 1 is characterized in that: In step S5, data analysis and monitoring are performed on the fusion results, including: in the process of data fusion, the data conflicts are adjudicated by using the formulated conflict resolution strategy.

7. The airport flight data fusion processing method based on timestamp according to claim 6 is characterized in that: Utilize the established conflict resolution strategy to make decisions, including: making decisions based on data source reliability and time freshness factors.

8. The airport flight data fusion processing method based on timestamp according to claim 6 is characterized in that: After step S5, the results need to be output and displayed: the integrated flight data is output in the form of structured data tables, API interfaces, CSV, JSON, and XML data formats, and displayed through a visual interface to monitor changes in basic flight information, resource usage information, and service guarantee information in real time, and to promptly detect and handle abnormal situations.

9. The airport flight data fusion processing method based on timestamp according to claim 8 is characterized in that: After the results are output and displayed, log recording and auditing are performed to record the data source, processing time, and fusion result information during the data fusion process, extract data fusion result monitoring indicators, and use a visual interface to display the real-time data fusion processing trend. For fusion information that fails to be processed, configure monitoring alarm rules and push processing exception information in real time via email and DingTalk instant messaging.

10. A timestamp-based airport flight data fusion processing system, characterized in that: The system implements the airport flight data fusion processing method based on timestamps as described in any one of claims 1 to 9, and the system includes: The database construction module (1) captures data streams from different data sources in real time through message queues and builds a database; The data preprocessing module (2) is used to parse the captured database, extract key data fields, and perform preprocessing operations; The data synchronization and sorting module (3) uses timestamps to synchronize and sort the parsed data to ensure that the data is processed in chronological order; The data fusion module (4) is used to process data in real time according to preset fusion rules to ensure the accuracy and consistency of the fusion results; The data analysis and monitoring module (5) is used to store the fused real-time data in a database, analyze and monitor the fusion results, and discover potential problems.

Images