Civil aviation data acquisition method, system, equipment and medium

Abstract

The invention discloses a civil aviation data acquisition method, system and device and a medium, belongs to the technical field of big data acquisition, and aims to solve the technical problem of how to avoid irregularity of data quality and formats of different data sources and guarantee data quality and data security. The abnormal data layer is used for storing unanalyzed data when the data processed by the data processing layer is abnormal; storing each piece of most original data into a distributed file system, and finally mapping the most original data into a table to form an original data layer; unencrypted data are directly analyzed into objects which can be used by technicians, or ciphertext data are correspondingly analyzed according to civil aviation requirements and a requirement manual; an arrival time is added to each piece of data, and then the maximum time difference of the two adjacent pieces of data is obtained; and an incremental primary key is added for each piece of data, and whether each piece of data is lost or not is visually displayed.

Description

Technical Field

[0001] The present invention relates to the technical field of big data collection, and in particular to a civil aviation data collection method, system, equipment and medium. Background Art

[0002] The civil aviation industry is one of the world's busiest and most complex transportation industries, with thousands of flights flying around the world every day, involving the transportation of millions of passengers and cargo. With the rapid development of the aviation market and the advancement of technology, the reliance on data is increasing. At present, the amount of civil aviation data is extremely large and complex, which affects the integrity, originality, accuracy, timeliness, continuity and traceability of the data, affects operational efficiency, and thus affects customer experience.

[0003] Therefore, how to avoid the uneven data quality and format of different data sources, ensure data quality and data security, and provide the most basic data support for the subsequent civil aviation system to optimize service quality, improve dual efficiency, and ensure flight safety is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The technical task of the present invention is to provide a civil aviation data collection method, system, equipment and medium to solve the problem of how to avoid uneven data quality and format of different data sources, ensure data quality and data security, and provide the most basic data support for subsequent civil aviation systems to optimize service quality, improve dual effects, and ensure flight safety.

[0005] The technical task of the present invention is achieved in the following way: a method for collecting civil aviation data, which is specifically as follows:

[0006] Add an exception data layer on the data processing layer. The exception data layer is used to store unparsed data when an exception occurs in the data processing layer, so that developers can use it to check the cause of the exception and ensure data integrity.

[0007] Each original data is stored in a distributed file system and eventually mapped into a table to form an original data layer to ensure the originality of the data.

[0008] Directly parse the unencrypted data into objects that can be used by technical personnel or parse the ciphertext data accordingly according to civil aviation requirements and requirement manuals to ensure the accuracy of each piece of parsed data;

[0009] Add an arrival time to each piece of data to obtain the maximum time difference between two adjacent pieces of data;

[0010] Add an incremental primary key to each piece of data to intuitively show whether each piece of data is lost, thereby obtaining the data continuity of each system.

[0011] As a preferred method, an arrival time is added to each piece of data, and the maximum time difference between two adjacent pieces of data is obtained as follows:

[0012] During data collection and processing, the arrival time difference between two adjacent data is obtained, and then the maximum time difference between each two data is determined based on the continuity of each system data;

[0013] If the delay of system data exceeds the threshold, the corresponding information will be sent to the monitoring system to accurately reflect the data timeliness of each system.

[0014] As a preferred method, add an incremental primary key to each piece of data to intuitively show whether each piece of data is lost. The details are as follows:

[0015] Compare each piece of data with the incrementing primary key of the previous piece of data:

[0016] If the increment of the corresponding data gradually differs from the increment of the previous data by more than one, it is considered that data is lost and a data continuity alarm is sent to the monitoring system.

[0017] Preferably, the primary key is the MD value of the data provided by the civil aviation system after removing useless information.

[0018] Preferably, the method is to store the data in three layers with different data formats in each layer, and convert each original data into a unique primary key with a length of a set threshold through an encryption algorithm to ensure the traceability of the parsed data.

[0019] A civil aviation data collection system, the system comprising:

[0020] The data processing layer is used to store the collected and successfully parsed data;

[0021] The abnormal data layer is used to store special data with parsing errors and defective data to ensure data integrity;

[0022] The original data layer is used to store the original data without any processing;

[0023] The time addition layer is used to add an arrival time to each piece of data, thereby obtaining the maximum time difference between two adjacent pieces of data;

[0024] The primary key adding layer is used to add an incremental primary key to each piece of data, intuitively showing whether each piece of data is lost, and thus obtaining the data continuity of each system.

[0025] Preferably, the original data stored in the original data layer is as follows:

[0026] For ordinary plaintext data, the specific fields appearing in the plaintext are mapped into available objects;

[0027] For ciphertext that cannot be used directly, the ciphertext is parsed through low or high-level programming according to the regulations of the civil aviation.

[0028] Better yet, the primary key addition layer compares each piece of data with the incremented primary key of the previous piece of data:

[0029] If the increment of the corresponding data gradually differs from the increment of the previous data by more than one, it is considered that data is lost and a data continuity alarm is sent to the monitoring system.

[0030] An electronic device comprising: a memory and at least one processor;

[0031] Wherein, the memory stores computer-executable instructions;

[0032] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the civil aviation data collection method as described above.

[0033] A computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the above-mentioned civil aviation data collection method is implemented.

[0034] The civil aviation data collection method, system, device and medium of the present invention have the following advantages:

[0035] (I) The present invention ensures the data quality from different dimensions such as integrity, originality, accuracy, timeliness, continuity and traceability, improves operational efficiency, optimizes service quality, ensures flight safety and, at the same time, enhances customer experience; the details are as follows:

[0036] (1) Integrity: In addition to storing the successfully parsed data collected in the data processing layer, some special data or even defective data is added to the abnormal data layer when parsing errors occur to ensure data integrity;

[0037] (2) Originality: Since the parsing layer stores the original data by field, if the original data is not stored, there is a risk of losing the original data. Therefore, an original data layer is added on the basis of the parsing layer and the abnormal data layer. Each piece of data is stored in the original data layer without any processing.

[0038] (3) Accuracy: The collected raw data will be discussed on a case-by-case basis. For ordinary plaintext data, it can be mapped to usable objects based on the specific fields that appear in the plaintext. For ciphertext that cannot be used directly, the ciphertext will be parsed through low / high-level programming according to the regulations of the civil aviation;

[0039] (4) Timeliness: In order to intuitively display the timeliness of each system, an arrival time mark is added to each piece of data. The arrival time difference between two adjacent pieces of data is used to determine whether there is a data timeliness delay in this system. If there is a timeliness delay, a message is sent to the monitoring system, so that the timeliness of each system can never be intuitively seen.

[0040] (5) Continuity: Add an incremental primary key to each piece of data to record the data. Use the incremental primary key difference between two adjacent pieces of data to determine whether the system has lost data. If data is lost, a message needs to be sent to the monitoring system to alert the technicians. Ensure data continuity;

[0041] (6) Traceability: In some scenarios, it is necessary to find the original data corresponding to the parsed data, so each piece of original data is encrypted into a shorter unique value as the primary key, which facilitates data restoration and tracing for questions that arise during downstream data analysis;

[0042] (ii) The present invention provides a solution to ensure data quality and security based on massive and complex data, thereby providing the most basic data support for subsequent civil aviation systems to optimize service quality, enhance dual efficiency, and ensure flight safety;

[0043] (III) The present invention can easily control the internal data quality by controlling and monitoring the data integrity, originality, accuracy, timeliness, continuity and traceability, avoid the uneven data quality and format of different data sources, and ensure the rationality of problem handling through the originality and traceability of the data, further ensuring the reliability of civil aviation data and meeting the needs of current regulatory authorities for the potential value of aviation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention is further described below in conjunction with the accompanying drawings.

[0045] Attached Figure 1 This is a flowchart of the civil aviation data collection method. DETAILED DESCRIPTION

[0046] The civil aviation data collection method, system, device and medium of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Embodiment 1:

[0048] As attached Figure 1 As shown, this embodiment provides a civil aviation data collection method, the method is specifically as follows:

[0049] S1. Add an exception data layer on the data processing layer. The exception data layer is used to store unparsed data when an exception occurs in the data processing layer, so that developers can use it to check the cause of the exception and ensure the integrity of the data.

[0050] S2: Store each piece of original data in a distributed file system and finally map it into a table to form an original data layer to ensure the originality of the data.

[0051] S3. Directly parse the unencrypted data into objects that can be used by technical personnel or parse the ciphertext data accordingly according to civil aviation requirements and the requirements manual to ensure the accuracy of each piece of parsed data;

[0052] S4. Add an arrival time to each piece of data, and then obtain the maximum time difference between two adjacent pieces of data;

[0053] S5. Add an incremental primary key to each piece of data to intuitively show whether each piece of data is lost, thereby obtaining the data continuity of each system.

[0054] In step S4 of this embodiment, an arrival time is added to each piece of data, and then the maximum time difference between two adjacent pieces of data is obtained as follows:

[0055] During data collection and processing, the arrival time difference between two adjacent data is obtained, and then the maximum time difference between each two data is determined based on the continuity of each system data;

[0056] If the delay of system data exceeds the threshold, the corresponding information will be sent to the monitoring system to accurately reflect the data timeliness of each system.

[0057] In step S5 of this embodiment, an incremental primary key is added to each piece of data to intuitively show whether each piece of data is lost or not, as follows:

[0058] Compare each piece of data with the incrementing primary key of the previous piece of data:

[0059] If the increment of the corresponding data gradually differs from the increment of the previous data by more than one, it is considered that data is lost and a data continuity alarm is sent to the monitoring system.

[0060] The primary key in step S5 of this embodiment is the MD value of the data obtained from the civil aviation system after removing useless information.

[0061] In this embodiment, the method is to store data in three layers, and the data format of each layer is different, and each original data is converted into a unique primary key with a length of a set threshold through an encryption algorithm to ensure the traceability of the parsed data.

[0062] Embodiment 2:

[0063] This embodiment provides a civil aviation data collection system, which includes:

[0064] The data processing layer is used to store the collected and successfully parsed data;

[0065] The abnormal data layer is used to store special data with parsing errors and defective data to ensure data integrity;

[0066] The original data layer is used to store the original data without any processing;

[0067] The time addition layer is used to add an arrival time to each piece of data, thereby obtaining the maximum time difference between two adjacent pieces of data;

[0068] The primary key adding layer is used to add an incremental primary key to each piece of data, intuitively showing whether each piece of data is lost, and thus obtaining the data continuity of each system.

[0069] The original data stored in the original data layer in this embodiment is as follows:

[0070] For ordinary plaintext data, the specific fields appearing in the plaintext are mapped into available objects;

[0071] For ciphertext that cannot be used directly, the ciphertext is parsed through low or high-level programming according to the regulations of the civil aviation.

[0072] The primary key addition layer in this embodiment compares each piece of data with the incrementing primary key of the previous piece of data:

[0073] If the increment of the corresponding data gradually differs from the increment of the previous data by more than one, it is considered that data is lost and a data continuity alarm is sent to the monitoring system.

[0074] Embodiment 3:

[0075] This embodiment also provides an electronic device, including: a memory and at least one processor;

[0076] Wherein, the memory stores computer-executable instructions;

[0077] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the civil aviation data collection method described in any one of the present inventions.

[0078] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0079] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0080] Embodiment 4:

[0081] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the civil aviation data collection method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0082] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0083] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0084] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0085] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A civil aviation data collection method, characterized in that: The method is as follows: Add an exception data layer on the data processing layer. The exception data layer is used to store unparsed data when an exception occurs in the data processing layer, so that developers can use it to check the cause of the exception and ensure data integrity. Each original data is stored in a distributed file system and eventually mapped into a table to form an original data layer to ensure the originality of the data. Directly parse the unencrypted data into objects that can be used by technical personnel or parse the ciphertext data accordingly according to civil aviation requirements and requirement manuals to ensure the accuracy of each piece of parsed data; Add an arrival time to each piece of data to obtain the maximum time difference between two adjacent pieces of data; Add an incremental primary key to each piece of data to intuitively show whether each piece of data is lost, thereby obtaining the data continuity of each system.

2. The civil aviation data collection method according to claim 1, characterized in that: Add an arrival time to each piece of data, and then obtain the maximum time difference between two adjacent pieces of data as follows: During data collection and processing, the arrival time difference between two adjacent data is obtained, and then the maximum time difference between each two data is determined based on the continuity of each system data; If the delay of system data exceeds the threshold, the corresponding information will be sent to the monitoring system to accurately reflect the data timeliness of each system.

3. The civil aviation data collection method according to claim 1, characterized in that: Add an incremental primary key to each piece of data to visually show whether each piece of data is missing. The details are as follows: Compare each piece of data with the incrementing primary key of the previous piece of data: If the increment of the corresponding data gradually differs from the increment of the previous data by more than one, it is considered that data is lost and a data continuity alarm is sent to the monitoring system.

4. The civil aviation data collection method according to claim 1, characterized in that: The primary key is obtained by obtaining the MD value of the data provided by the civil aviation system after removing useless information.

5. The civil aviation data collection method according to any one of claims 1 to 4, characterized in that: This method is to store data in three layers, with different data formats in each layer, and convert each original data into a unique primary key with a length of a set threshold through an encryption algorithm to ensure the traceability of the parsed data.

6. A civil aviation data collection system, characterized in that: The system includes: The data processing layer is used to store the collected and successfully parsed data; The abnormal data layer is used to store special data with parsing errors and defective data to ensure data integrity; The original data layer is used to store the original data without any processing; The time addition layer is used to add an arrival time to each piece of data, thereby obtaining the maximum time difference between two adjacent pieces of data; The primary key adding layer is used to add an incremental primary key to each piece of data, intuitively showing whether each piece of data is lost, and thus obtaining the data continuity of each system.

7. The civil aviation data collection system according to claim 6, characterized in that: The original data stored in the original data layer is as follows: For ordinary plaintext data, the specific fields appearing in the plaintext are mapped into available objects; For ciphertext that cannot be used directly, the ciphertext is parsed through low or high-level programming according to the regulations of the civil aviation.

8. The civil aviation data collection system according to claim 6 or 7, characterized in that: The primary key addition layer compares each piece of data with the incremented primary key of the previous piece of data: If the increment of the corresponding data gradually differs from the increment of the previous data by more than one, it is considered that data is lost and a data continuity alarm is sent to the monitoring system.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the civil aviation data collection method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the civil aviation data collection method as described in any one of claims 1 to 5 is implemented.

Images