A parallel data analysis method for flight data

Abstract

The present invention belongs to the field of industrial design and research and development, and is specifically a parallel data parsing method for flight data. By performing parallel parsing of flight data file data packets, the efficiency of data parsing can be effectively improved, thereby improving the data utilization efficiency of engineering designers. By encoding and compressing time and parameter values based on the characteristics of flight data, the storage space of the parsed data can be reduced, saving storage costs. After the parsed flight data files are stored in a distributed columnar database at parameter granularity, an efficient parameter query service interface can be provided, enabling rapid query and display of parameter curves, thereby improving the data analysis efficiency of engineering designers. This method facilitates fault analysis and diagnosis by logging the parsing process. This method adopts a microservices architecture design with high cohesion and low coupling, and can be deployed in a containerized manner, making deployment convenient and highly scalable.

Description

Technical Field

[0001] The present invention belongs to the field of industrial design and research and development, and specifically relates to a parallel data parsing method for flight data. Background Art

[0002] Aircraft generate vast amounts of flight data during testing, flight trials, and equipment deployment. This data is of critical value in model development, certification and evaluation, maintenance, repair, and accident investigation. Flight data is stored in binary files and, upon transmission, is stored in the company's file storage system. Aircraft engineering professionals download and pre-process the files before using various data analysis software to review and analyze the data, a time-consuming process. With the significant advancement of aircraft information technology, flight data is also rapidly expanding, posing significant challenges to data processing, storage, analysis, and management.

[0003] In the internet sector, big data technology has been widely applied, achieving significant results in e-commerce, intelligent recommendations, logistics distribution, advertising and marketing, and other businesses. In the aviation industry, Boeing established an aviation data analysis laboratory in 2015 to collect massive amounts of data generated during flight. Based on a big data processing system and a flight test data analysis platform, the results of this data analysis guide design, manufacturing, and operations, effectively supporting the development and production of new aircraft and improving efficiency.

[0004] To improve the efficiency of data query and analysis, the flight data binary files need to be parsed and stored at parameter granularity, and stored through stand-alone parsing tools to traditional relational databases. This takes a long time to parse, the data space expands significantly after parsing, and the cost is high. Summary of the Invention

[0005] The purpose of the present invention is to solve the parameter-level data parsing and data storage problems of massive flight data binary files, so as to provide parameter granularity query services in seconds and improve the data utilization efficiency of engineering designers.

[0006] The technical solution of the present invention:

[0007] A parallel data parsing method for flight data includes the following steps:

[0008] After being transferred back to the NAS file storage system, the flight data file in step 1 is stored. The flight data file is a binary file consisting of a file header and tens of millions of data packets. The file header contains information such as the aircraft number, takeoff time, and decoding dictionary version number. Each data packet records multiple sensor signal parameter values for a specific aircraft subsystem at a specific moment in time and consists of a data packet header and a data packet body. The data packet header is of fixed length and has a data packet start identifier. It also contains the aircraft subsystem identifier, packet acquisition time, and data packet length information. The data packet body consists of multiple sensor signal parameter values.

[0009] Step 2: The decoding dictionary service component loads multiple versions of decoding rule files. The decoding rules define the relative start word, start bit, data type, and data length of each sensor signal parameter in the data packet body.

[0010] Step 3: After the server where the file parsing component is located is mounted on the NAS file storage system, the file parsing component includes a depacketizing module, a packet parsing module, a data compression module, and a storage module. The depacketizing module reads the flight data binary file and, upon detecting the start identifier of a data packet, extracts the data packet based on its length and passes it to the packet parsing module for processing.

[0011] In step 4, after receiving the data packet, the packet parsing module calls the decoding dictionary service component based on the decoding dictionary version number and the aircraft subsystem identifier of the data packet as input information to obtain the parsing rule information, specifically the relative start word, start bit, data type and data length information of all sensor signal parameters in this data packet.

[0012] In step 5, the packet parsing module reads the sensor signal parameters in the data packet into a byte array based on the relative start word, start bit, and data length according to the parsing rules. It then parses and converts the data according to the data type. This yields the value of each sensor signal parameter in the data packet and passes it to the data compression module for processing.

[0013] Step 6: The data compression processing module receives the sensor signal parameter values after parsing multiple data packets. For each different sensor signal, the parameter values at multiple moments are constructed into a time series value, encoded and compressed, and the compressed data is handed over to the storage module for processing.

[0014] In step 7, the storage module writes the compressed sensor signal data into the distributed column database.

[0015] Furthermore, it also includes parsing scheduling. Before step 2, multiple file parsing components can be deployed on multiple servers and managed through the parsing scheduling components. For multiple flight data files in the NAS file system, parsing can be scheduled simultaneously.

[0016] Furthermore, the method also includes the parallel unpacking method of step 4, in which the packet parsing module adopts a multi-threading method to parse multiple data packets in parallel at the same time.

[0017] Furthermore, the coding compression method of step 6 is also included, in which the collected time series values are compressed using delta-of-delta coding, and the sensor signal parameter values are compressed using XOR coding. Coding compression can reduce the storage space of the parsed data and save storage costs.

[0018] Furthermore, it also includes parsing process log records. The parsing log component records the start time, end time, parsing time, parsing file name, parsing file size, number of sensor signal values after parsing, and packet parsing exception information of the entire parsing process.

[0019] Furthermore, it also includes data query services. The data query service component provides a query interface with sensor signal parameter granularity. The aircraft number, take-off and landing information, the aircraft subsystem to which it belongs, and the sensor signal name are input. By querying the distributed column database, the time series value of a certain sensor signal under a certain flight take-off and landing process is returned. The column database has an index, which can achieve query interface return in seconds, which can improve the efficiency of data query display.

[0020] Furthermore, the decoding rule file in step 2 is stored in the form of a CSV file or a database table.

[0021] Furthermore, the data type of the sensor signal parameter in step 5 can be a discrete quantity or a continuous quantity, and the continuous quantity includes an int type and a float type.

[0022] Beneficial effects of the present invention:

[0023] (1) By analyzing the flight data file data packets in parallel, the efficiency of data analysis can be effectively improved, thereby improving the data utilization efficiency of engineering designers;

[0024] (2) According to the characteristics of flight data, by encoding and compressing the time value and parameter value, the storage space of the parsed data can be reduced, thus saving storage costs;

[0025] (3) After parsing the flight data files and storing them in a distributed column-based database at parameter granularity, it can provide an efficient parameter query service interface, enabling rapid query and display of parameter curves, thereby improving the data analysis efficiency of engineering designers;

[0026] (4) This method facilitates fault diagnosis by logging the parsing process;

[0027] (5) This method adopts a microservice architecture design with high cohesion and low coupling. It can be deployed in a containerized manner, which is convenient to deploy and has strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the structure diagram of the flight data binary file;

[0029] The flight data file consists of a file header and tens of millions of data packets. The file header contains the following information:

[0030] Aircraft number, takeoff time, decoding dictionary version number. Each data packet records multiple sensor signal parameter values of a certain aircraft subsystem at a certain moment, and consists of a data packet header and a data packet body. The data packet header is of fixed length and contains the data packet start identifier, the aircraft subsystem identifier, the packet acquisition time, and the packet body length information. The data packet body consists of multiple sensor signal parameter values;

[0031] Figure 2 This is the architecture diagram of the parallel data parsing component for flight data;

[0032] Figure 3 This is a diagram of the parallel parsing process of flight data. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0034] The parallel data parsing method for flight data includes the following steps:

[0035] After being transferred back to the NAS file storage system, the flight data file in step 1 is stored. The flight data file is a binary file consisting of a file header and tens of millions of data packets. The file header contains information such as the aircraft number, takeoff time, and decoding dictionary version number. Each data packet records multiple sensor signal parameter values for a specific aircraft subsystem at a specific moment in time and consists of a data packet header and a data packet body. The data packet header is of fixed length and has a data packet start identifier. It also contains the aircraft subsystem identifier, packet acquisition time, and data packet length information. The data packet body consists of multiple sensor signal parameter values.

[0036] Step 2: The decoding dictionary service component loads multiple versions of decoding rule files. The decoding rules define the relative start word, start bit, data type, and data length of each sensor signal parameter in the data packet body.

[0037] Step 3: After the server where the file parsing component is located is mounted on the NAS file storage system, the file parsing component includes a depacketizing module, a packet parsing module, a data compression module, and a storage module. The depacketizing module reads the flight data binary file and, upon detecting the start identifier of a data packet, extracts the data packet based on its length and passes it to the packet parsing module for processing.

[0038] In step 4, after receiving the data packet, the packet parsing module calls the decoding dictionary service component based on the decoding dictionary version number and the aircraft subsystem identifier of the data packet as input information to obtain the parsing rule information, specifically the relative start word, start bit, data type and data length information of all sensor signal parameters in this data packet.

[0039] In step 5, the packet parsing module reads the sensor signal parameters in the data packet into a byte array based on the relative start word, start bit, and data length according to the parsing rules. It then parses and converts the data according to the data type. This yields the value of each sensor signal parameter in the data packet and passes it to the data compression module for processing.

[0040] Step 6: The data compression processing module receives the sensor signal parameter values after parsing multiple data packets. For each different sensor signal, the parameter values at multiple moments are constructed into a time series value, encoded and compressed, and the compressed data is handed over to the storage module for processing.

[0041] In step 7, the storage module writes the compressed sensor signal data into the distributed column database.

[0042] Furthermore, it also includes parsing scheduling. Before step 2, multiple file parsing components can be deployed on multiple servers and managed through the parsing scheduling component. For multiple flight data files in the NAS file system, parsing can be scheduled at the same time to increase the parsing concurrency, thereby improving the overall parsing efficiency. It also includes the parallel unpacking method of step 4. The packet parsing module adopts a multi-threaded method to parse multiple data packets in parallel at the same time. It also includes the encoding compression method of step 6. The acquisition time series value adopts delta-of-delta encoding compression, and the sensor signal parameter value adopts XOR encoding compression. Through encoding compression, the storage space of the data after parsing can be reduced, saving storage costs. It also includes parsing process log recording. The parsing log component records the start time, end time, parsing time, parsing file name, parsing file size, number of sensor signal values after parsing, and packet parsing exception information of the entire parsing process. The parsing process log can be used to trace back the parsing process for parsing exception fault diagnosis. The data query service also includes a data query service component that provides a query interface with granularity for sensor signal parameters. Inputs include the aircraft number, takeoff and landing information, the aircraft subsystem to which it belongs, and the sensor signal name. By querying a distributed columnar database, the time series values of a sensor signal during a specific flight's takeoff and landing process are returned. The columnar database is indexed, enabling query interfaces to return values within seconds, improving data query and display efficiency. The decoding rule file in step 2 can be stored as a CSV file or in a database table. The data type of the sensor signal parameter in step 5 can be either discrete or continuous, with continuous values including int and float types.

[0043] Example 1

[0044] Figure 1 The structure of a flight data binary file is shown. Each flight data file consists of a file header and tens of millions of data packets. The file header contains information such as the aircraft number, takeoff time, and decoding dictionary version number. Each data packet records multiple sensor signal parameter values of a certain aircraft subsystem at a specific moment in time and consists of a data packet header and a data packet body. The data packet header is of fixed length and contains a data packet start identifier (for example, the hexadecimal number FEFE), an aircraft subsystem identifier, the packet acquisition time, and the data packet length information. The data packet body consists of multiple sensor signal parameter values. The relative position, length, and data type of each sensor signal in the data packet body are defined by the decoding rule file.

[0045] Figure 2 A two-dimensional table diagram showing the decoding rules is used for storage in a CSV file or relational database. Each row defines the relative start word, start bit, data length, and data type of a sensor signal parameter in the data packet body.

[0046] Figure 3The data parsing component architecture diagram is shown, including the parsing scheduling component, decoding dictionary component, data parsing component, parsing log component, and distributed database. Among them, the data parsing component includes a file unpacking module, a packet parsing module, a data compression module, and a storage module.

[0047] An embodiment of a parallel data parsing method for flight data is mainly performed in sequence by the following steps:

[0048] Step 1: After the flight is complete, the flight data files are transferred back to the enterprise NAS file storage. Deploy the Linux server system with the file parsing component and mount the NAS file storage.

[0049] Step 2 imports multiple versions of decoding dictionary rule CSV files into the MySQL relational database. The decoding dictionary service component connects to the MySQL database to load the decoding rule data and provides a parsing rule information query interface for the file parsing component. The input parameters of the query interface are the decoding dictionary version number and the aircraft subsystem identifier. The output is the data packet parsing two-dimensional table information under the corresponding conditions. Each row of the two-dimensional table defines the relative starting word, starting bit, data length and data type of a sensor signal parameter in the data packet.

[0050] In step 3, the file parsing component's unpacking module reads the flight data binary file from the NAS file storage and obtains the flight number, takeoff time, and decoding dictionary version number by reading the file header. It then begins scanning the file byte by byte. When it finds the packet start identifier (hexadecimal FEFE), it reads the packet header to obtain the aircraft subsystem identifier, packet acquisition time, and packet length. The packet is then intercepted based on the packet length and passed to the packet parsing module for processing.

[0051] In step 4, after receiving the data packet, the packet parsing module calls the query interface service of the decoding dictionary component based on the decoding dictionary version number in the flight data file header and the aircraft subsystem identifier in the data packet header as input information to obtain the parsing rule information, including the relative start word, start bit, data type and data length information of each sensor signal parameter.

[0052] In step 5, the packet parsing module parses the sensor signal parameters in the data packet one by one according to the parsing rules, converts them according to the data type, and temporarily stores them in a byte array. Different sensor signal parameters have different byte arrays. For a particular sensor signal parameter, when a certain number of temporarily stored values (1000) are reached, the time series value is passed to the data compression module for processing.

[0053] Step 6: After receiving the sensor signal sequence value, the data compression processing module performs encoding compression processing, wherein the signal parameter sequence is compressed using XOR encoding, and the compressed data is handed over to the storage module for processing. The XOR compression logic is as follows:

[0054] 1) The first value is not compressed

[0055] 2) If the XOR result with the previous value is 0 (i.e. the value is the same), only one bit is used for storage, and the value is 0

[0056] 3) If the XOR result is non-zero, the first bit of the control is stored as 1, and the following values are processed as follows

[0057] a) When the control bit is 0, the data block with meaningful bits is included in the previous data block

[0058] b) When the control bit is 1, use the next 5 bits to store the number of leading 0s, then use 6 bits to store the length of the non-zero bits in the middle of the XOR, and finally store the non-zero bits in the middle

[0059] Step 7: The storage module writes the compressed sensor signal data into the distributed column-based database HBASE.

[0060] The above description is merely a detailed description of specific embodiments of the present invention. Any unspecified portions are conventional techniques. However, the scope of the present invention is not limited thereto. Any changes or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed in the present invention are intended to be encompassed within the scope of the present invention. The scope of the present invention shall be determined by the scope of the claims.

Claims

1. A parallel data parsing method for flight data, characterized in that: The steps include: Step 1: After the flight data file is transmitted back, it is stored in the NAS file storage system. The flight data file is a binary file consisting of a file header and tens of millions of data packets. The file header contains information such as the aircraft number, takeoff time, and decoding dictionary version number. Each data packet records multiple sensor signal parameter values of a certain aircraft subsystem at a specific moment and consists of a data packet header and a data packet body. The data packet header is of fixed length and has a data packet start identifier. It also contains the aircraft subsystem identifier, packet collection time, and packet body length information. The data packet body consists of multiple sensor signal parameter values. Step 2: The decoding dictionary service component loads multiple versions of decoding rule files. The decoding rules define the relative start word, start bit, data type, and data length of each sensor signal parameter in the data packet body. Step 3: After the server where the file parsing component is located is mounted with the NAS file storage system, the file parsing component includes a depacketizing module, a packet parsing module, a data compression module, and a storage module. The depacketizing module reads the flight data binary file and, upon detecting the start identifier of a data packet, intercepts the data packet based on the packet length and passes the data packet to the packet parsing module for processing. Step 4: After receiving the data packet, the packet parsing module calls the decoding dictionary service component based on the decoding dictionary version number and the aircraft subsystem identifier of the data packet as input information to obtain parsing rule information, specifically the relative start word, start bit, data type and data length information of all sensor signal parameters in this data packet; Step 5: The packet parsing module reads the sensor signal parameters in the data packet into a byte array according to the relative start word, start bit and data length according to the parsing rule information, and performs parsing and conversion according to the data type; Thus, the parameter value of each sensor signal in the data packet is obtained and passed to the data compression module for processing; Step 6: The data compression processing module receives the sensor signal parameter values after parsing multiple data packets. For each different sensor signal, the parameter values at multiple moments are constructed into a time series value, encoded and compressed, and the compressed data is handed over to the storage module for processing; In step 7, the storage module writes the compressed sensor signal data into the distributed column database.

2. The parallel data parsing method for flight data according to claim 1, characterized in that: It also includes parsing scheduling. Before step 2, multiple file parsing components can be deployed on multiple servers and managed through the parsing scheduling components. For multiple flight data files in the NAS file system, parsing can be scheduled simultaneously.

3. The parallel data parsing method for flight data according to claim 1, characterized in that: It also includes the parallel unpacking method of step 4, where the packet parsing module adopts a multi-threaded method to parse multiple data packets in parallel at the same time.

4. The parallel data parsing method for flight data according to claim 1, characterized in that: It also includes the coding compression method of step 6, where the collected time series values are compressed using delta-of-delta coding, and the sensor signal parameter values are compressed using XOR coding. Coding compression can reduce the storage space of the parsed data and save storage costs.

5. The parallel data parsing method for flight data according to claim 1, characterized in that: It also includes parsing process log records. The parsing log component records the start time, end time, parsing time, parsing file name, parsing file size, number of sensor signal values after parsing, and packet parsing exception information of the entire parsing process.

6. The parallel data parsing method for flight data according to claim 1, characterized in that: It also includes data query services. The data query service component provides a query interface with sensor signal parameter granularity. By inputting the aircraft number, take-off and landing information, the aircraft subsystem to which it belongs, and the sensor signal name, it queries the distributed column-based database and returns the time series value of a certain sensor signal during a certain flight take-off and landing process. The column-based database has an index, which can achieve query interface return in seconds.

7. The parallel data parsing method for flight data according to claim 1, characterized in that: The decoding rule file in step 2 is stored in CSV file format or database table format.

8. The parallel data parsing method for flight data according to claim 1, characterized in that: The data type of the sensor signal parameter in step 5 is a discrete quantity or a continuous quantity. The continuous quantity includes the int type and the float type.

Images