Massive flight leg data processing method and system

Abstract

The invention provides a massive flight leg data processing method and system, and relates to the technical field of massive civil aircraft flight leg data processing, and the method comprises the steps: S1, data processing task splitting: carrying out the sub-task splitting of an analysis task according to the condition of analyzing massive civil aircraft flight leg data at a time, each subtask query data is positioned in a single partition folder of the distributed system, so that the task is prevented from acquiring data across servers. The method solves the problem that server requirements are more and more when mass civil aircraft flight segment data are analyzed at a time and time is longer and longer along with the increase of the data size, mass flight segment data analysis can be completed under limited server resources, meanwhile, mass flight segment data analysis is efficiently completed under the condition that the servers are sufficient, and the efficiency of mass flight segment data analysis is improved. Under the condition that only three or four server resources exist, the effect of analyzing data of one hundred thousand legs in one hour is achieved, and the analysis speed is higher along with the increase of the servers and the performance.

Description

Technical Field

[0001] The present invention relates to the technical field of massive civil aircraft flight segment data processing, and in particular to a massive flight segment data processing method and system. Background Art

[0002] With the continuous development of the civil aviation transportation industry and the gradual deepening of digital transformation, the application of machine learning-based status assessment and fault prediction modeling technology to analyze civil aircraft flight segment data and realize civil aircraft system status assessment and fault prediction analysis is becoming more and more widespread.

[0003] Through the study of the current civil aircraft flight segment data analysis system, it was found that as the number of aircraft in the airline fleet increases, the volume of segment data becomes larger and larger, and the total data volume has reached the TB level. With continuous accumulation, this amount will continue to increase to the PB level or even more. The time required to obtain all historical flight segment data for model analysis and calculation at one time is getting longer and longer, and the scale of resources is getting larger and larger. Currently, data analysis mostly takes several hours or even several days, which seriously limits the timeliness of civil aircraft system status assessment and fault prediction analysis. Summary of the invention

[0004] The present invention proposes a method and system for processing massive flight segment data, which solves the problem that a one-time analysis of massive civil aircraft flight segment data takes longer and longer as the amount of data increases, and more and more servers are required. The method can complete the analysis of massive flight segment data under limited server resources, and at the same time, when there are sufficient servers, the analysis of massive segment data can be completed efficiently. When there are only 3 or 4 server resources, the effect of analyzing 100,000 segment data in one hour can be achieved. Moreover, as the number of servers and performance increases, the analysis speed will be faster. Multi-fleet data is partitioned and stored according to reasonable dimensions, and data processing subtasks are split according to partition storage rules. The computing server flexibly claims tasks according to its own resources and provides a subtask detection mechanism. Through the above key points, the characteristics of massive, efficient, resource-controllable and highly fault-tolerant data processing can be achieved.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for processing massive flight segment data, comprising the following steps:

[0006] Step 1: Split the data processing task: Split the analysis task into subtasks based on the condition of analyzing a large amount of civil aircraft flight segment data at one time. Each subtask query data is located in a single partition folder of the distributed system to avoid tasks from obtaining data across servers.

[0007] Step 2: Automatically claim data processing subtasks: Put the data processing split subtasks into the redis cache database and establish a distributed lock mechanism. During the subtask processing, provide a subtask heartbeat sending mechanism and update the heartbeat records into the cache database;

[0008] Step 3: Automatic recovery process in case of error: Create a detection task in the cache database. The non-data processing server claims the detection task through a distributed lock to determine the heartbeat status of each subtask.

[0009] Furthermore, in step 1, the subtasks are divided according to fleet, aircraft and flight month.

[0010] Furthermore, in step 2, the server capable of analyzing the flight segment data may obtain the data processing subtask from the cache database according to its own computing resources to perform task processing.

[0011] Furthermore, in step 3, if the heartbeat of the subtask is abnormal, an attempt is made to connect to the computing server to detect the status of the subtask. If an abnormality still occurs, the subtask status is automatically cleared and re-entered into the cache database subtask queue for redistribution.

[0012] A massive flight segment data processing system, comprising:

[0013] Front-end display module: provides data processing task form submission, processing task list display, processing task log display, processing task subtask list display, subtask status, and heartbeat information display;

[0014] Service application module: used to deploy application backend services, provide data processing task reception, subtask splitting and storage, monitoring task execution, task status, logs, subtask heartbeat monitoring, and related information return functions;

[0015] Data algorithm processing module: used for data processing subtask segment data acquisition, algorithm logic calculation, subtask heartbeat / status / log update;

[0016] Cache database module: used to store and quickly retrieve frequently accessed data to improve application performance and responsiveness;

[0017] Distributed data storage module: responsible for storing fleet flight segment data. The data is partitioned and stored according to the fleet, aircraft, flight segment month and time dimensions.

[0018] The cache database module includes a subtask queue module, a subtask status module, a task status module and a task log module, and the subtask queue module, the subtask status module, the task status module and the task log module are all bidirectionally connected;

[0019] The output end bidirectional signal of the subtask queue module in the cache database module is connected to the input end of the distributed data storage module.

[0020] Furthermore, the front-end display module includes a data processing task submission module, a processing task list module, a processing task log viewing module, a subtask list module, a subtask status monitoring module and a subtask heartbeat monitoring module, and the data processing task submission module, the processing task list module, the processing task log viewing module, the subtask list module, the subtask status monitoring module and the subtask heartbeat monitoring module are all bidirectionally connected.

[0021] Furthermore, the service application module includes a task processing acquisition module, a subtask splitting module, a task monitoring execution module, a task status module, a task log module and an application server module, and the task processing acquisition module, the subtask splitting module, the task monitoring execution module, the task status module, the task log module and the application server module are all bidirectionally connected.

[0022] Furthermore, the data algorithm processing module includes a subtask data acquisition module, a subtask data processing module, a subtask heartbeat update module, a subtask status update module, a subtask log update module and a computing server module, and the subtask data acquisition module, subtask data processing module, subtask heartbeat update module, subtask status update module, subtask log update module and computing server module are all bidirectionally connected.

[0023] Furthermore, the output signal of the data processing task submitted in the front-end display module is connected to the input of the processing task acquisition module in the service application module, the output signal of the data processing task submitted in the front-end display module is connected to the input of the subtask data acquisition module in the data algorithm processing module, and the output signal of the data processing task submitted in the front-end display module is connected to the input of the subtask queue module in the cache database module.

[0024] Furthermore, the output-end bidirectional signal of the processing task acquisition module in the service application module is connected to the input-end of the distributed data storage module, the output-end signal of the processing task acquisition module in the service application module is connected to the input-end of the subtask data acquisition module in the data algorithm processing module, the output-end bidirectional signal of the subtask data acquisition module in the data algorithm processing module is connected to the input-end of the distributed data storage module, and the output-end signal of the subtask data acquisition module in the data algorithm processing module is connected to the input-end of the subtask queue module in the cache database module.

[0025] The present invention provides a method and system for processing massive flight segment data, which has the following beneficial effects:

[0026] (1) The method and system for processing massive flight segment data solves the problem that the one-time analysis of massive civil aircraft flight segment data takes longer and longer as the amount of data increases, and the server requirements increase. The massive flight segment data analysis can be completed with limited server resources. At the same time, when there are sufficient servers, the massive flight segment data analysis can be completed efficiently. When there are only 3 or 4 server resources, the effect of analyzing 100,000 flight segment data in one hour can be achieved. Moreover, as the number of servers and performance increases, the analysis speed will be faster.

[0027] (2) The method and system for processing massive flight segment data partitions and stores data from multiple fleets according to reasonable dimensions, splits data processing subtasks according to partition storage rules, and the computing server flexibly claims tasks according to its own resources and provides a subtask detection mechanism. Through the above key points, the characteristics of massive, efficient, resource-controllable, and highly fault-tolerant data processing can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a method and system for processing massive flight segment data according to the present invention;

[0029] Figure 2 The present invention is a general system diagram of a method and system for processing massive flight segment data;

[0030] Figure 3 A schematic diagram of a front-end display module of a method and system for processing massive flight segment data according to the present invention;

[0031] Figure 4 A schematic diagram of a service application module of a method and system for processing massive flight segment data according to the present invention;

[0032] Figure 5 A schematic diagram of a data algorithm processing module of a method and system for processing massive flight segment data according to the present invention;

[0033] Figure 6 A schematic diagram of a cache database module of a method and system for processing massive flight segment data according to the present invention;

[0034] Figure 7 The present invention provides a method and system for processing massive flight segment data and an overall flow chart of data processing;

[0035] Figure 8 A data processing log logic flow chart of a method and system for processing massive flight segment data according to the present invention;

[0036] Fig. 9 The present invention provides a method and system for processing massive flight segment data and a data processing subtask status monitoring and fault-tolerant logic flow chart.

[0037] In the figure: 1. Front-end display module; 2. Service application module; 3. Data algorithm processing module; 4. Cache database module; 5. Distributed data storage module. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0039] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0040] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0041] See also Figure 1-9 The present invention provides a technical solution: a method for processing massive flight segment data, comprising the following steps:

[0042] Step 1: Split the data processing task: Split the analysis task into subtasks based on the condition of analyzing a large amount of civil aircraft flight segment data at one time. Each subtask query data is located in a single partition folder of the distributed system to avoid tasks from obtaining data across servers.

[0043] Step 2: Automatically claim data processing subtasks: Put the data processing split subtasks into the redis cache database and establish a distributed lock mechanism. During the subtask processing, provide a subtask heartbeat sending mechanism and update the heartbeat records into the cache database;

[0044] Step 3: Automatic recovery process in case of error: Create a detection task in the cache database. The non-data processing server claims the detection task through a distributed lock to determine the heartbeat status of each subtask.

[0045] Specifically, in step one, the subtasks are divided according to fleet, aircraft and flight month.

[0046] Specifically, in step 2, the server with the flight segment data analysis function can obtain the data processing subtask from the cache database according to its own computing resources to perform task processing.

[0047] Specifically, in step S3, if the heartbeat of the subtask is abnormal, an attempt is made to connect to the computing server to detect the state of the subtask. If an abnormality still occurs, the subtask state is automatically cleared and put back into the cache database subtask queue for reallocation.

[0048] A massive flight segment data processing system, comprising:

[0049] Front-end display module 1: provides data processing task form submission, processing task list display, processing task log display, processing task subtask list display, subtask status, and heartbeat information display;

[0050] Service application module 2: used to deploy application backend services, provide data processing task reception, subtask splitting and storage, monitoring task execution, task status, logs, subtask heartbeat monitoring, and related information return functions;

[0051] Data algorithm processing module 3: used for data processing subtask segment data acquisition, algorithm logic calculation, subtask heartbeat / status / log update;

[0052] Cache database module 4: used to store and quickly retrieve frequently accessed data to improve the performance and response speed of the application;

[0053] Distributed data storage module 5: responsible for storing fleet flight segment data, the data is partitioned and stored according to the dimensions of fleet, aircraft, flight segment month and time;

[0054] The cache database module 4 includes a subtask queue module, a subtask state module, a task state module and a task log module, and the subtask queue module, the subtask state module, the task state module and the task log module are all bidirectionally connected;

[0055] The bidirectional signal of the output end of the subtask queue module in the cache database module 4 is connected to the input end of the distributed data storage module 5 .

[0056] Specifically, the front-end display module 1 includes a data processing task submission module, a processing task list module, a processing task log viewing module, a subtask list module, a subtask status monitoring module and a subtask heartbeat monitoring module. The data processing task submission module, the processing task list module, the processing task log viewing module, the subtask list module, the subtask status monitoring module and the subtask heartbeat monitoring module are all bidirectionally connected.

[0057] Specifically, the service application module 2 includes a task acquisition processing module, a subtask splitting module, a task execution monitoring module, a task status module, a task log module and an application server module. The task acquisition processing module, the subtask splitting module, the task execution monitoring module, the task status module, the task log module and the application server module are all bidirectionally connected.

[0058] Specifically, the data algorithm processing module 3 includes a subtask data acquisition module, a subtask data processing module, a subtask heartbeat update module, a subtask status update module, a subtask log update module and a computing server module. The subtask data acquisition module, the subtask data processing module, the subtask heartbeat update module, the subtask status update module, the subtask log update module and the computing server module are all bidirectionally connected.

[0059] Specifically, the output signal of the data processing task submitted in the front-end display module 1 is connected to the input of the processing task acquisition module in the service application module 2, the output signal of the data processing task submitted in the front-end display module 1 is connected to the input of the subtask data acquisition module in the data algorithm processing module 3, and the output signal of the data processing task submitted in the front-end display module 1 is connected to the input of the subtask queue module in the cache database module 4.

[0060] Specifically, the output-end bidirectional signal of the processing task acquisition module in the service application module 2 is connected to the input end of the distributed data storage module 5, the output-end signal of the processing task acquisition module in the service application module 2 is connected to the input end of the subtask data acquisition module in the data algorithm processing module 3, the output-end bidirectional signal of the subtask data acquisition module in the data algorithm processing module 3 is connected to the input end of the distributed data storage module 5, and the output-end signal of the subtask data acquisition module in the data algorithm processing module 3 is connected to the input end of the subtask queue module in the cache database module 4.

[0061] The entire system is built based on SpringBoot and VUE. The entire system business architecture includes a distributed data storage layer, a service application layer, a data algorithm processing layer, and a front-end display layer.

[0062] The distributed data storage layer is composed of a distributed database or file system, which is responsible for storing fleet aircraft segment data. The data is partitioned and stored in data blocks according to the dimensions of fleet, aircraft, flight segment month and time; the service application layer is composed of high-availability application servers, which are used to deploy application backend services and provide data processing task reception, subtask splitting storage, monitoring task execution, task status, logs, subtask heartbeat monitoring and related information return functions; the data algorithm processing layer is composed of multiple computing servers, which are used for data processing subtask segment data acquisition, algorithm logic calculation, subtask heartbeat / status / log update, etc.; the front-end display layer provides data processing task form submission, processing task list display, processing task log display, processing task subtask list display, subtask status, heartbeat and other information display.

[0063] The task of analyzing massive amounts of civil aircraft flight segment data at one time is first submitted to the service application layer through the front-end service. The service application layer substitutes the parameters such as the fleet, aircraft, and analysis segment time range passed in by the front-end into the data processing task splitting method for sub-task splitting, and submits the monitoring task to a cache database such as redis.

[0064] The data processing task splitting method is to split the fleet, aircraft, and flight segment month subtask arrays based on the fleet, aircraft, and analysis segment time range parameter data. If the parameter range does not include the entire month segment data, the segment information of the current month segment that meets the parameter range is added to the subtask information array. After the information is supplemented, the subtask information data is cached in the cache database queue such as redis. For example: the user submits the PRSOV average pressure regulation value characteristic value calculation task corresponding to the flight segment from January 10, 2020 to August 10, 2024 for aircraft 1 in fleet 1. According to the task splitting method, it will be split into 44 data processing subtasks corresponding to aircraft 1 in fleet 1 from January 2020 to August 2024. Among them, the January 2020 subtask needs to query the segment information from January 10 to 31 and put it into the subtask information, and the August 2024 subtask needs to query the segment information from August 1 to 10 and put it into the subtask information.

[0065] After the data processing subtask information data is cached in the cache database queue such as redis, the data algorithm processing layer server automatically claims the data processing task method according to the computing resources and claims the subtask for data processing.

[0066] The specific logic of the method for automatically claiming data processing tasks with computing resources is as follows: the algorithm server obtains the CPU / memory / network resources of this service, and obtains the subtask according to the distributed lock of the data processing subtask queue when the CPU / memory / network resources are all idle. According to the subtask fleet, aircraft, flight month and specific flight information, it enters the distributed data repository to obtain the partitioned flight data to the server for algorithm calculation, and at the same time adds the subtask status, processing server information, heartbeat and algorithm calculation log information to the cache database. During the algorithm calculation process, the subtask status information of the cache database such as redis is obtained to determine whether the subtask is still tracked and processed by this server. If the server information is not found to be this server, the data processing is interrupted. If the processing server is consistent with this server, the heartbeat information of the subtask is updated according to the heartbeat interval setting time. After the algorithm calculation is completed, the algorithm completion status and log are updated to the cache database. If the algorithm calculation is successful, the algorithm result is updated to the result database. For example, there are three existing computing servers that need to process 44 data processing subtasks corresponding to one aircraft in the fleet from January 2020 to August 2024. Initially, each of the three servers obtains subtasks according to the distributed lock serial acquisition for data processing. Assuming that after each server claims three subtasks for processing, the CPU / memory / network resources are full, then the task claiming is stopped. Currently, there are 9 subtasks being executed and the remaining 35 subtasks are waiting to be executed. After the server subtasks are executed and the resources are released, the other 35 subtasks are executed in sequence. Under the limited execution of the three computing servers, the 44 data processing subtasks can be executed in sequence according to the computing resources until they are completed. Assuming that there are 15 computing servers, each with 3 subtasks, the 44 data processing subtasks can be claimed by 15 servers at one time for concurrent computing, achieving efficient concurrent processing.

[0067] During the execution of all subtasks of the data processing task, the application server obtains the task monitoring task based on the distributed lock. The monitoring task automatically recovers the processing method according to the task calculation error, detects the heartbeat and status of all subtasks of the data processing task, and merges the overall task log into the cache database. If all subtasks are completed, the data processing task status and log are updated, and the cache database information corresponding to all related subtasks is cleared.

[0068] The specific logic of the automatic recovery method for task calculation errors is as follows: data processing task monitoring provides subtask status judgment logic. If it is monitored that the subtask is being executed, the difference between its heartbeat time and the current time is judged. Assuming that the time difference is greater than 3 times the defined heartbeat interval, an attempt is made to connect to the calculation server to detect the subtask status. If the connection fails, the subtask processing server is cleared, the status is set to waiting status, and the subtask is put back into the subtask allocation queue, waiting for the next task to be claimed and calculated.

[0069] The main technical features of the present invention are: it realizes the quasi-real-time and efficient merging of incremental data and the on-demand cooling and re-merging of existing data, supports differentiated merging strategies at the user or bucket level, and each merging strategy can set various parameters according to actual needs, thereby improving the merging efficiency and the relevance of the merged files, and improving the read and write performance and space utilization.

[0070] The above are only preferred embodiments of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A method for processing massive flight segment data, characterized in that: The following steps are involved: S1. Data processing task splitting: Based on the condition of analyzing a large amount of civil aircraft flight segment data at one time, the analysis task is split into subtasks. The query data of each subtask is located in a single partition folder of the distributed system to avoid the task of obtaining data across servers; S2. Automatically claim data processing subtasks: Put the data processing split subtasks into the redis cache database and establish a distributed lock mechanism. During the processing of the subtasks, a subtask heartbeat sending mechanism is provided to update the heartbeat records into the cache database; S3. Automatic error recovery processing: A detection task is established in the cache database. The non-data processing server claims the detection task through a distributed lock to determine the heartbeat status of each subtask.

2. A method for processing massive flight segment data according to claim 1, characterized in that: In step S1, the subtasks are divided according to fleet, aircraft and flight month.

3. A method for processing massive flight segment data according to claim 1, characterized in that: In step S2, the server with the function of analyzing the flight segment data may obtain the data processing subtask from the cache database according to its own computing resources to perform task processing.

4. The method for processing massive flight segment data according to claim 1, characterized in that: In step S3, if the heartbeat of the subtask is abnormal, an attempt is made to connect to the computing server to detect the state of the subtask. If an abnormality still occurs, the subtask state is automatically cleared and put back into the cache database subtask queue for reallocation.

5. A massive flight segment data processing system, characterized in that: include: Front-end display module (1): provides data processing task form submission, processing task list display, processing task log display, processing task subtask list display, subtask status, and heartbeat information display; Service application module (2): used to deploy application backend services, provide data processing task reception, subtask splitting and storage, monitoring task execution, task status, logs, subtask heartbeat monitoring, and related information return functions; Data algorithm processing module (3): used for data processing subtask segment data acquisition, algorithm logic calculation, and subtask heartbeat / status / log update; Cache database module (4): used to store and quickly retrieve frequently accessed data to improve application performance and responsiveness; Distributed data storage module (5): responsible for storing fleet aircraft standby flight segment data, the data is partitioned and stored in data blocks according to the dimensions of fleet, standby aircraft, flight segment month and time; The cache database module (4) comprises a subtask queue module, a subtask status module, a task status module and a task log module, and the subtask queue module, the subtask status module, the task status module and the task log module are all bidirectionally connected; The output end bidirectional signal of the subtask queue module in the cache database module (4) is connected to the input end of the distributed data storage module (5).

6. A massive flight segment data processing system according to claim 2, characterized in that: The front-end display module (1) comprises a data processing task submission module, a processing task list module, a processing task log viewing module, a subtask list module, a subtask status monitoring module and a subtask heartbeat monitoring module, and the data processing task submission module, the processing task list module, the processing task log viewing module, the subtask list module, the subtask status monitoring module and the subtask heartbeat monitoring module are all bidirectionally connected.

7. A massive flight segment data processing system according to claim 2, characterized in that: The service application module (2) comprises a processing task acquisition module, a subtask splitting module, a monitoring task execution module, a task status module, a task log module and an application server module, and the processing task acquisition module, the subtask splitting module, the monitoring task execution module, the task status module, the task log module and the application server module are all bidirectionally connected.

8. A massive flight segment data processing system according to claim 2, characterized in that: The data algorithm processing module (3) comprises a subtask data acquisition module, a subtask data processing module, a subtask heartbeat update module, a subtask status update module, a subtask log update module and a computing server module, and the subtask data acquisition module, the subtask data processing module, the subtask heartbeat update module, the subtask status update module, the subtask log update module and the computing server module are all bidirectionally connected.

9. A massive flight segment data processing system according to claim 2, characterized in that: The output terminal signal submitted by the data processing task in the front-end display module (1) is connected to the input terminal of the processing task acquisition module in the service application module (2), the output terminal signal submitted by the data processing task in the front-end display module (1) is connected to the input terminal of the subtask data acquisition module in the data algorithm processing module (3), and the output terminal signal submitted by the data processing task in the front-end display module (1) is connected to the input terminal of the subtask queue module in the cache database module (4).

10. A massive flight segment data processing system according to claim 2, characterized in that: The output bidirectional signal of the processing task acquisition module in the service application module (2) is connected to the input of the distributed data storage module (5); the output signal of the processing task acquisition module in the service application module (2) is connected to the input of the subtask data acquisition module in the data algorithm processing module (3); the output bidirectional signal of the subtask data acquisition module in the data algorithm processing module (3) is connected to the input of the distributed data storage module (5); the output signal of the subtask data acquisition module in the data algorithm processing module (3) is connected to the input of the subtask queue module in the cache database module (4).

Images