Data processing system and method for air freight, medium and equipment

Abstract

The invention provides a data processing system and method for air freight, a medium and equipment, and the system comprises a basic data setting module which is used for setting basic data; the rule setting module is used for setting a data processing rule; the data acquisition module is used for synchronously acquiring source data from a data source, and the source data comprises air freight scheduled flight data and scheduled transportation data in an assessment period; the data processing module is used for pre-processing the planned flight execution flight data and the planned transportation data thereof, and performing tagging processing on the pre-processed data according to the basic data to obtain planned operation data of the assessment cycle; and the assessment index generation module is used for processing the planned operation data according to the data processing rule to obtain an assessment index of each route segment of the assessment cycle, and the assessment indexes correspond to the route segments, the moments and the associated business department. The system can flexibly set rules, and is high in data processing efficiency and high in processing result accuracy.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing technology for air cargo transportation. Background Art

[0002] In air cargo management, traditional performance evaluation methods often rely on manual statistics and calculations. These methods involve obtaining relevant source data, typically tabular data, from data sources. Multiple intermediate tables are then manually created, and corresponding data is extracted and matched according to established rules. This process is complex, multi-step, time-consuming, and inefficient. Furthermore, it can lead to inaccurate results and lacks flexibility, making it unable to meet the demands of fluctuating market conditions, environments, and regulatory changes that frequently impact evaluation results.

[0003] Therefore, the existing method of manually processing air cargo data has technical problems such as low efficiency, low accuracy of results, and lack of flexibility. Summary of the Invention

[0004] In order to at least partially solve the above-mentioned existing technical problems, the purpose of the present invention is to provide a data processing system, method, medium and device for air cargo transportation.

[0005] According to one aspect of the present invention, a data processing system for air cargo is provided, wherein the data processing system comprises:

[0006] A basic data setting module is used to set basic data, wherein the basic data at least includes: flight data and the business department associated with the flight, wherein the flight data includes: flight number, aircraft type, route segment and time;

[0007] Rule setting module, used to set data processing rules;

[0008] A data acquisition module is used to synchronously acquire source data from a data source, wherein the source data includes: air cargo planned flight data and planned transportation data within the assessment period;

[0009] a data processing module, configured to pre-process the planned flight data and the planned transportation data, and label the pre-processed data based on the basic data to obtain the planned operation data for the assessment period;

[0010] The assessment index generation module is used to process the planned operating data according to the data processing rules to obtain the assessment index of each route segment in the assessment period, wherein the assessment index corresponds to the route segment and time, and the associated business department.

[0011] Optionally, the data processing rules include:

[0012] Cardinality generation rules;

[0013] Coefficient generation rules;

[0014] Indicator generation rules.

[0015] Optionally, the pre-processing of the planned flight data and the planned transportation data includes:

[0016] The planned flight data and planned transportation data are deduplicated and formatted in a standardized manner.

[0017] Optionally, the step of labeling the pre-processed data based on the basic data to obtain the planned operating data for the assessment period includes:

[0018] According to the flight number in the preprocessed data, the aircraft type, route segment and time, and business department corresponding to the flight number in the basic data are used as labels, and the preprocessed data is labeled to obtain the planned operating data for the assessment period.

[0019] Optionally, the processing of the planned operating data according to the data processing rules to obtain the assessment indicators for each route segment in the assessment period includes:

[0020] Processing the planned operating data according to the cardinality generation rule to obtain cardinality data for each flight in the assessment period;

[0021] Processing the planned operating data according to the coefficient generation rule to obtain coefficient data for each flight in the assessment period;

[0022] According to the indicator generation rule, the base data and the coefficient data are processed to obtain the assessment indicator of each route segment in the assessment period.

[0023] Optionally, the source data further includes: flight data and operational data completed within the assessment period, wherein the data processing system further includes:

[0024] A performance evaluation module is used to obtain the completed indicators for each route segment in the evaluation period based on the evaluation indicators of each route segment in the evaluation period, the flight data completed in the evaluation period, and the completed operation data;

[0025] The indicator completion rate of each assessment object is determined based on the assessment indicators of each route segment in the assessment period and the completed indicators of each route segment in the assessment period, wherein the assessment objects include: routes or sales departments.

[0026] Optionally, the basic data further includes: a person in charge associated with the business department, wherein the assessment object also includes the person in charge.

[0027] Optionally, the performance appraisal module is further configured to:

[0028] The performance of the assessment subject is determined based on the assessment subject's indicator completion rate and preset performance rules.

[0029] Optionally, the data processing system further includes:

[0030] Data query and analysis module, used for data query and analysis.

[0031] Optionally, the data processing system further includes:

[0032] The appeal and review module is used for the assessment subjects to submit appeals and to review the appeals.

[0033] According to another aspect of the present invention, a data processing method for air cargo is provided, wherein the data processing method comprises:

[0034] Set up basic data and data processing rules;

[0035] Synchronously obtain source data from a data source, wherein the source data includes: planned air cargo flight data and planned transportation data within the assessment period;

[0036] Preprocessing the planned flight data and the planned transportation data, and labeling the preprocessed data based on the basic data to obtain the planned operating data for the assessment period;

[0037] The planned operating data is processed according to the data processing rules to obtain the assessment indicators of each route segment in the assessment period.

[0038] Optionally, the source data further includes: flight data and operating data completed within the assessment period, wherein the data processing method further includes:

[0039] Based on the assessment indicators of each route segment in the assessment period, the flight data and the completed operating data within the assessment period, the completed indicators of each route segment in the assessment period are obtained, and based on the assessment indicators of each route segment in the assessment period and the completed indicators of each route segment in the assessment period, the indicator completion rate of each assessment object is determined, wherein the assessment object includes: route or business department.

[0040] Optionally, the data processing method further includes:

[0041] The performance of the assessment subject is determined based on the assessment subject's indicator completion rate and preset performance rules.

[0042] Compared with the prior art, the present invention provides a data processing system, method, medium and equipment for air cargo, the data processing system comprising: a basic data setting module for setting basic data, wherein the basic data at least includes: flight data and a business department associated with the flight, wherein the flight data includes: flight number, aircraft type, route segment and time; a rule setting module for setting data processing rules; a data acquisition module for synchronously acquiring source data from a data source, wherein the source data includes: planned air cargo flight data and planned transportation data within an assessment period; a data processing module for preprocessing the planned flight data and its planned transportation data, and labeling the preprocessed data according to the basic data to obtain planned operating data for the assessment period; an assessment indicator generation module for processing the planned operating data according to the data processing rules to obtain assessment indicators for each route segment of the assessment period, wherein the assessment indicators correspond to the route segment and time, and the associated business department. The present invention can synchronously acquire source data from a data source, combine basic data with pre-set data processing rules, and label the source data to obtain assessment indicators for the assessment cycle. This allows for automated generation of assessment indicators based on air cargo data for performance appraisals. Data processing rules can be flexibly set and take effect in real time, resulting in high data processing efficiency and highly accurate processing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0044] Figure 1 A schematic diagram of a data processing system for air cargo transportation according to one aspect of the present invention is shown;

[0045] Figure 2 A schematic flow chart of a data processing method for air cargo transportation according to another aspect of the present invention is shown;

[0046] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below with reference to the accompanying drawings.

[0048] In a typical configuration of each embodiment of the present invention, the execution subject of the method, each trusted party of the system and / or each module of the device includes one or more processors (CPU), input / output interface, network interface and memory.

[0049] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0050] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0051] In order to further illustrate the technical means adopted by the present invention and the effects achieved, the technical solutions of the present invention are clearly and completely described below in conjunction with the accompanying drawings and various embodiments.

[0052] Figure 1 A schematic diagram of a data processing system for air cargo transportation according to one aspect of the present invention is shown, wherein the data processing system of one embodiment includes:

[0053] The basic data setting module 110 is used to set basic data, wherein the basic data includes at least flight data and the business department associated with the flight, wherein the flight data includes the flight number, aircraft type, route segment and time;

[0054] A rule setting module 120 is used to set data processing rules;

[0055] The data acquisition module 130 is used to synchronously acquire source data from a data source, wherein the source data includes: air cargo planned flight data and planned transportation data within the assessment period;

[0056] The data processing module 140 is configured to pre-process the planned flight data and the planned transportation data, and label the pre-processed data based on the basic data to obtain the planned operation data for the assessment period;

[0057] The assessment index generation module 150 is used to process the planned operation data according to the data processing rules to obtain the assessment index of each route segment in the assessment period, wherein the assessment index corresponds to the route segment and time, and the associated business department.

[0058] In this embodiment, one or more modules of the data processing system 100 may be deployed in a device 100, which is a computer device and / or cloud installed with a software and hardware environment for related data processing. The computer device includes but is not limited to a personal computer, a laptop computer, an industrial computer, a network host, a single network server, or a set of multiple network servers. The cloud is composed of a large number of computers or network servers based on cloud computing, wherein cloud computing is a type of distributed computing, a virtual supercomputer composed of a group of loosely coupled computer sets. Here, the computer device and / or cloud are only examples. Other existing or future devices and / or resource sharing platforms that are applicable to this application should also be included in the scope of protection of this application and are incorporated herein by reference.

[0059] In this embodiment, the data processing system 100 includes: a basic data setting module 110 , a rule setting module 120 , a data acquisition module 130 , a data processing module 140 and an assessment indicator generation module 150 .

[0060] In this embodiment, the basic data setting module 110 is used to set relevant basic data, wherein the basic data includes at least: flight data for cargo transportation and business department information associated with the flight, wherein the flight data may include: flight number, aircraft type operating the flight, route corresponding to the flight, and corresponding route segments and times that constitute the route. Among them, one flight number usually corresponds to one route, and one route usually includes a departure point, several stopover points and a destination. The departure point to the first stopover point, the previous stopover point to the next stopover point, and the last stopover point to the destination are considered to be corresponding route segments, and each route segment has a departure time and an arrival time. Among them, combined with the business operation model, one business department can usually be responsible for the operation of a flight or route. The basic data setting module 110 can be used to set aircraft model information for cargo flights, including major aircraft categories (e.g., Boeing B787, Airbus A320, etc.), sub-type numbers (e.g., B787-9, A320-CEO, etc.), and / or aircraft attributes (e.g., wide-body, narrow-body). The module can also set the flight number and aircraft model, the route corresponding to the flight, and the default and / or specific arrival and / or departure times for the route segments. The basic data setting module 110 can also be used to set charter flights (chartered routes and boarding routes require the corresponding charter flight indicators and revenue to be deducted when generating assessment indicators), no-cargo and mail stations (no-cargo and mail stations are not included in the assessment indicators for the corresponding route segments), and micro-stations. The basic data setting module 110 can also be used to set designated coefficients for the relevant route segments within the assessment period, by default and / or in specific circumstances, to match changes in operating data caused by increases in the CPI (Consumer Price Index) and / or for routes and segments for which no correlation coefficients can be generated.

[0061] Continuing with this embodiment, data processing rules are set through the rule setting module 120, including at least: rules for generating the base, coefficient, and index of the route segments corresponding to the flights in the basic data, and may also include: assessment rules for setting the appeal time limit, performance payment ratio and approval process; and exemption rules for assessment, for setting assessment objects that are exempted from assessment.

[0062] Optionally, the data processing rules include:

[0063] Cardinality generation rules;

[0064] Coefficient generation rules;

[0065] Indicator generation rules.

[0066] In this optional embodiment, the rule setting module 120 can be used to set a base number generation rule to clarify the base number used for the assessment of the route segment under default and / or specified conditions such as different flight types (for example, international flights, domestic flights), different aircraft models, and different categories of route segments (for example, newly added route segments). For example, under default conditions, the base number corresponding to the route segment can be set to the planned total revenue / planned number of flights for the route segment during the assessment period. Under specified conditions (for example, the route segment has no planned flights during the assessment period), the base number can be specified. Coefficient generation rules can be set to clarify the coefficients used for the assessment of the route segment under default and / or specified conditions such as different aircraft models and different base numbers. For example, under default conditions, the coefficient corresponding to the route segment can be set to the base number of the previous assessment period / the base number of the current assessment period for the route segment. Indicator generation rules can be set to clarify the assessment indicators of route segments under default and / or specified conditions such as different aircraft models. For example, the assessment indicator corresponding to the route flight can be set as the base x coefficient of the route segment, and the total assessment indicator corresponding to the route flight can be set as the assessment indicator of the route segment x the number of planned flights.

[0067] Continuing with this embodiment, the data acquisition module 130 can synchronously acquire source data from a data source, wherein the source data includes: planned air cargo flight data and planned transportation data within the assessment period.

[0068] The data processing system 100 is connected to the data source through a network, and can synchronously obtain source data from the data source through the data acquisition module 130 according to the API interface supported by both parties. The source data synchronously obtained from the data source includes at least: air cargo planned flight data within the assessment period (for example, the assessment period can be a natural month or a quarter) and planned transportation data corresponding to the route and segment of the flight.

[0069] Continuing with this embodiment, the data processing module 140 can pre-process the air cargo planned flight data within the assessment period and the planned transportation data corresponding to the flight routes and segments in the source data obtained by the data acquisition module 130, and then label the pre-processed data based on the flight data set by the basic data setting module 110, the business department associated with the flight, and other basic data to obtain the planned operating data within the assessment period. The data can be classified and summarized according to the labels for use in assessment, query, analysis, display, etc.

[0070] Optionally, the pre-processing of the planned flight data and the planned transportation data includes:

[0071] The planned flight data and planned transportation data are deduplicated and formatted in a standardized manner.

[0072] In this optional embodiment, the data processing module 140 can be used to deduplicate and standardize the air cargo planned flight data within the assessment period and the planned transportation data corresponding to the flight routes and segments in the source data obtained by the data acquisition module 130 to ensure data integrity and consistency.

[0073] Optionally, the step of labeling the pre-processed data based on the basic data to obtain the planned operating data for the assessment period includes:

[0074] According to the flight number in the preprocessed data, the aircraft type, route segment and time, and business department corresponding to the flight number in the basic data are used as labels, and the preprocessed data is labeled to obtain the planned operating data for the assessment period.

[0075] In this optional embodiment, through the data processing module 140, based on the flight data set by the basic data setting module 110, the basic data such as the business department associated with the flight, and the flight number in the pre-processed data, the aircraft type, route segment and time, and business department corresponding to the flight number in the basic data are used as labels, and the pre-processed data are labeled to obtain the planned operating data for the assessment period.

[0076] Continuing with this embodiment, through the assessment indicator generation module 150, the planned operating data obtained through the data processing module 140 can be processed according to the data processing rules set by the rule setting module 120 to obtain the assessment indicators of each route segment in the assessment period, wherein the assessment indicators correspond to the route segment and time, and the associated business department.

[0077] Optionally, the processing of the planned operating data according to the data processing rules to obtain the assessment indicators for each route segment in the assessment period includes:

[0078] Processing the planned operating data according to the cardinality generation rule to obtain cardinality data for each flight in the assessment period;

[0079] Processing the planned operating data according to the coefficient generation rule to obtain coefficient data for each flight in the assessment period;

[0080] According to the indicator generation rule, the base data and the coefficient data are processed to obtain the assessment indicator of each route segment in the assessment period.

[0081] In this optional embodiment, the assessment index generation module 150 can process the planned operating data obtained by the data processing module 140 according to the base generation rules set by the rule setting module 120 to obtain the base data of each flight in the assessment period; the planned operating data obtained by the data processing module 140 can be processed according to the coefficient generation rules set by the rule setting module 120 to obtain the coefficient data of each flight in the assessment period; and then the obtained base data and coefficient data can be processed according to the index generation rules set by the rule setting module 120 to obtain the assessment index of each route segment in the assessment period. As an example, the base data of a certain route segment in the current assessment period = the planned total revenue of the route segment in the current assessment period / the number of flights planned to operate the route segment. The coefficient data of a certain route segment in the current assessment period = the base data of the route segment in the previous assessment period / the base data of the route segment in the current assessment period. The assessment index of a certain route segment in the current assessment period = the base data of a certain route segment in the current assessment period x the coefficient data of a certain route segment in the current assessment period; the total assessment index of a certain route segment in the current assessment period = the assessment index of a certain route segment in the current assessment period x the number of planned flights.

[0082] The data processing system 100 in the above-described embodiment and / or optional embodiment synchronously acquires source data from a data source, pre-processes the source data, and then labels it based on the underlying data to obtain planned operating data. This planned operating data is then processed according to data processing rules to obtain assessment indicators for each route segment during the assessment period. The entire data processing process requires no human intervention, and data processing rules can be flexibly set and take effect in real time, resulting in high data processing efficiency and highly accurate processing results.

[0083] Optionally, the source data further includes: flight data and operational data completed within the assessment period, wherein the data processing system 100 further includes:

[0084] The performance evaluation module 160 is configured to obtain a completed indicator for each route segment during the evaluation period based on the evaluation indicator for each route segment during the evaluation period, the flight data completed during the evaluation period, and the completed operation data;

[0085] The indicator completion rate of each assessment object is determined based on the assessment indicators of each route segment in the assessment period and the completed indicators of each route segment in the assessment period, wherein the assessment objects include: routes or sales departments.

[0086] The data acquisition module 130 can also synchronously obtain the data of completed flights and completed operating data within the assessment period from the data source. In this optional embodiment, the data processing system 100 also includes: a performance assessment module 160. Among them, the performance assessment module 160 can obtain the completed indicators of each route segment within the assessment period based on the assessment indicators of each route segment of the assessment period obtained by the assessment indicator generation module 150, the data of completed flights and completed operating data within the assessment period obtained by the data acquisition module 130, and then determine the indicator completion rate of each assessment object based on the assessment indicators of each route segment of the assessment period obtained by the assessment indicator generation module 150 and the completed indicators of each route segment within the assessment period. Among them, the assessment indicators of each route segment within the assessment period correspond to the route segment and time, and the associated business department. In combination with actual application scenarios, routes or business departments can be used as assessment objects. As an example, the indicator completion rate of the assessment object during the assessment period = the sum of the completed indicators of all routes and segments related to the assessment object during the assessment period / the sum of the assessment indicators of all routes and segments related to the assessment object during the assessment period.

[0087] Optionally, the basic data further includes: a person in charge associated with the business department, wherein the assessment object also includes the person in charge.

[0088] In some application scenarios, the assessment target also includes the head of the sales department. In this optional embodiment, the basic data setting module 110 can also be used to set the head of the sales department associated with the flight. In combination with the actual application scenario, the performance assessment module 160 can be used to determine the target completion rate of the head of the sales department.

[0089] Optionally, the performance appraisal module 160 is further configured to:

[0090] The performance of the assessment subject is determined based on the assessment subject's indicator completion rate and preset performance rules.

[0091] In this optional embodiment, the performance evaluation module 160 can also determine the performance of the evaluation subject based on the evaluation subject's indicator completion rate and preset performance rules. In an exemplary embodiment, if the evaluation subject is a sales department, its performance is determined in stages based on the indicator completion rate. For example, if the indicator completion rate is 60%, the performance corresponding to the evaluation indicator is determined at 50%; if the indicator completion rate is 90%, the performance corresponding to the evaluation indicator is determined at 80%. At the end of the evaluation period, the performance is determined and settled based on the final actual indicator completion rate.

[0092] Optionally, the data processing system 100 further includes:

[0093] The data query and analysis module 170 is used for data query and analysis.

[0094] In this optional embodiment, the data processing system 100 can also query and analyze various types of data through the data query and analysis module 170, including but not limited to: querying source data, conditional query indicators (e.g., overall assessment indicators, completed indicators, indicator completion rates, and sales department assessment indicators, completed indicators, and indicator completion rates), querying performance results, conditional querying of chartered cabin and board information, querying flight data and operating data, querying basic data, querying data processing rules, etc., as well as operating trend analysis, source data analysis, etc. The data query and analysis module 170 can also visualize the query and analysis results.

[0095] Optionally, the data processing system 100 further includes:

[0096] The appeal and review module 170 is used for the assessment subject to submit an appeal and to review the appeal.

[0097] In this optional embodiment, combined with actual application scenarios, there may be data included in the base calculation that does not conform to the actual situation. The assessment subject can view his or her performance through the appeal and review module 170 during the validity period of the appeal. If there is any objection, he or she can submit an appeal (which may include the route segment involved in the appeal, the appeal type, the appeal reason, etc.). After the review and confirmation, the performance can be updated by adjusting the assessment indicators and / or completed indicators. For example, when determining the completed indicators of the route segment, the relevant data of the chartered cabin and the chartered board should be deducted, and the relevant data of the micro-stations should be considered; when determining the assessment indicators of the route segment, the relevant data of the non-cargo and mail stations should be deducted, etc. The appeal record, appeal progress, etc. can also be queried through the appeal and review module 170.

[0098] Through the data processing system 100 in the above optional embodiment, source data is synchronously obtained from the data source, and after pre-processing the source data, labeling processing is performed based on the basic data to obtain planned operating data. Then, according to the data processing rules, the planned operating data is processed to obtain the assessment indicators of each route segment in the assessment period. The actual flight data and actual transportation data that have occurred during the assessment period can be synchronously obtained from the data source. After processing, the actual operating data within the assessment period is obtained, and then the completed indicators are determined. Based on the assessment indicators and the completed indicators, the indicator completion rate of the assessment object is determined, and the performance of the assessment object is determined. Relevant data can also be queried and analyzed, and the relevant data can be visualized. The assessment object can also make a statement on the performance that is controversial and the statement is reviewed. In the entire data processing process, human participation is reduced as much as possible, and data processing rules can be flexibly set and take effect in real time. The data processing efficiency is high and the processing results are highly accurate.

[0099] As an example, the data acquisition module 130 of the data processing system 100 can use the xx-job distributed task scheduling framework to call the httpAPI by executing the timed sharding task, and obtain the source data from the data source synchronous sharding, which can ensure that the data obtained from the data source is not lost or duplicated (for example, through the retry strategy, it can automatically retry after the timed sharding task fails, and obtain the timestamp deduplication through the transaction ID, etc.). The data processing module 140 can use the Apache HttpClient integrated with the xx-job distributed task scheduling framework, and receive the source data obtained by the data acquisition module 130 through the connection pool and retry strategy, and perform pre-processing such as cleaning and format conversion on the source data. Then, combined with the basic data set by the basic data setting module 110 and stored in ClickHouse, the pre-processed data is labeled to obtain the planned operating data within the assessment period. Through the assessment indicator generation module 150, according to the data processing rules set by the rule setting module 120 and stored in Redis, the planned operating data within the assessment period is processed to obtain the assessment indicators of each route segment of the assessment period. The query and analysis module 160 can adopt the Spring Boot architecture in combination with visualization components to display the query and analysis content in a visual multi-dimensional dashboard / chart format. Among them, the Redis cache + ClickHouse storage method is adopted, and the data is written to ClickHouse based on the atomic granularity (for example, the Buffer table engine is used to temporarily store data, and then asynchronously written to the main table, and the ReplacingMergeTree engine is used to process duplicate data) to meet data integrity and consistency, as well as high throughput (ClickHouse storage) and low latency (Redis cache). Among them, in order to reduce or avoid the Too many parts error that may be caused by ClickHouse high-speed writing, corresponding measures can be taken (for example, enabling async_insert=1 to asynchronously submit write requests, using the Buffer table as a write buffer layer, etc.). In order to avoid the risk of Redis downtime or concentrated cache expiration causing a surge in instantaneous database pressure, which may cause an avalanche, multi-level caching (for example, local Caffeine + distributed Redis), caching null values, and using random expiration times (for example, basic data never expires, and the expiration time of dynamic data fluctuates around 10%). You can also take measures such as task result verification (for example, verifying the amount of sharded data obtained through ClickHouse's count()) and compensation tasks (for example, regularly starting full data verification tasks) to ensure the consistency of xx-job distributed tasks.To ensure data security, ClickHouse can enable RBAC (Role-Based Access Control) to limit user access rights, and Redis can enable SSL (Secure Sockets Layer) transmission + ACL (Access Control List) permission control.

[0100] In addition, using Redis cache + ClickHouse storage allows for rapid synchronization after changes to data processing rules (for example, data processing rules are stored in Redis, data is read in real time, and adjustments to data processing rules such as cardinality, coefficients, and / or indicator generation rules take effect in real time). Acquired source data can also be backed up to object storage (for example, OSS / S3) for disaster recovery.

[0101] As another example, Apache Doris or StarRocks can be used to replace ClickHouse to replace the data storage solution; Apache Ignite or Dragonfly can be used to replace Redis to replace the data caching solution; SeaTunnel can be used to replace Apache HttpClient, and Apache DolphinScheduler can be used to replace xx-job to replace the data synchronization and task scheduling solution.

[0102] Another example is to use a hybrid data storage architecture solution of ClickHouse + Apache Doris to reduce data storage costs (in actual cases, the hybrid architecture solution can reduce storage costs by 35% compared to the ClickHouse storage solution), where hot data is stored in ClickHouse and warm data is stored in Apache Doris.

[0103] Figure 2 A data processing method for air cargo transportation according to another aspect of the present invention is shown, wherein the data processing method in one embodiment includes:

[0104] S201 sets basic data and data processing rules;

[0105] S202 synchronously obtains source data from a data source, wherein the source data includes: air cargo planned flight data and planned transportation data within the assessment period;

[0106] S203 pre-processes the planned flight data and the planned transportation data, and labels the pre-processed data based on the basic data to obtain the planned operation data for the assessment period;

[0107] S204 processes the planned operation data according to the data processing rules to obtain the assessment index of each route segment in the assessment period.

[0108] In this embodiment, in step S201, basic data can be set through the basic data setting module 110 of the data processing system 100, and data processing rules can be set through the rule setting module 120 of the data processing system 100. The basic data at least includes: flight data for cargo transportation and business department information associated with the flight, wherein the flight data may include: flight number, aircraft type operating the flight, route corresponding to the flight, and corresponding route segments and time constituting the route. The data processing rules at least include: rules for generating the base number, coefficient, and index of the route segments corresponding to the flights in the basic data, and may also include: assessment rules for setting the appeal time limit, performance payment ratio, and approval process; and exemption rules for setting assessment objects exempted from assessment.

[0109] The data processing system 100 is connected to the data source via a network. Continuing with this embodiment, in step S202, source data can be synchronously acquired from the data source via the data acquisition module 130 of the data processing system 100 according to an API interface supported by both parties. The source data synchronously acquired from the data source includes at least: planned air cargo flight data within an assessment period (e.g., a calendar month or quarterly assessment period) and planned transportation data corresponding to the flight routes and segments.

[0110] Continuing with this embodiment, in step S203, the data processing module 140 of the data processing system 100 can be used to pre-process the air cargo planned flight data within the assessment period and the planned transportation data corresponding to the flight routes and segments in the obtained source data, and then label the pre-processed data based on basic data such as flight data and business departments associated with the flights to obtain planned operating data within the assessment period. The data can be classified and summarized according to the labels for use in assessment, query, analysis, display, etc.

[0111] Continuing with this embodiment, in step S204, the assessment indicator generation module 150 of the data processing system 100 can process the obtained planned operating data according to the set data processing rules to obtain the assessment indicators of each route segment in the assessment period.

[0112] Optionally, the source data further includes: flight data and operating data completed within the assessment period, wherein the data processing method further includes:

[0113] S205 obtains the completed indicators of each route segment in the assessment period based on the assessment indicators of each route segment in the assessment period, the flight data and the completed operating data in the assessment period, and determines the indicator completion rate of each assessment object based on the assessment indicators of each route segment in the assessment period and the completed indicators of each route segment in the assessment period, wherein the assessment objects include: routes or business departments.

[0114] The data acquisition module 130 of the data processing system 100 can also synchronously obtain the data of completed flights and completed operating data within the assessment period from the data source. In this optional embodiment, in step S205, the performance evaluation module 160 of the data processing system 100 can obtain the completed indicators of each route segment within the assessment period based on the obtained assessment indicators of each route segment within the assessment period, and then determine the indicator completion rate of each assessment object based on the assessment indicators of each route segment within the assessment period and the completed indicators of each route segment within the assessment period obtained by the assessment indicator generation module 150, wherein the assessment indicators of each route segment within the assessment period correspond to the route segment and time, and the associated business department. In combination with the actual application scenario, the route or business department can be used as the assessment object.

[0115] Optionally, the data processing method further includes:

[0116] S206 determines the performance of the assessment object according to the indicator completion rate of the assessment object and preset performance rules.

[0117] In this optional embodiment, in step S206, the performance evaluation module 170 of the data processing system 100 can also determine the performance of the evaluation object based on the indicator completion rate of the evaluation object and preset performance rules.

[0118] In an optional embodiment, the data processing method further includes: during the validity period of the appeal, the audit subject can view his or her own performance through the appeal and audit module 170 of the data processing system 100. If there is any objection, the audit subject can make an appeal (which may include the route segment involved in the appeal, the type of appeal, the reason for the appeal, etc.). After the audit is confirmed to be correct, the performance can be updated by adjusting the assessment indicators and / or completed indicators.

[0119] One or more method steps of the method embodiments and / or optional embodiments of the present application may be implemented through corresponding modules of the data processing system 100. The implementation entities not mentioned are the same as the relevant modules in the aforementioned system embodiments and / or optional embodiments and will not be repeated here.

[0120] According to yet another aspect of the present invention, a computer-readable medium is provided. The computer-readable medium stores computer-readable instructions. The computer-readable instructions can be executed by a processor to implement part or all of the above method.

[0121] It should be noted that the various method embodiments and / or optional embodiments of the present invention may be implemented in part or in whole in software and / or a combination of software and hardware. The software program involved in the present invention may be executed by a processor to implement some or all of the steps or functions of the various embodiments and / or optional embodiments described above. Similarly, the software program of the present invention (including related data structures) may be stored in a computer-readable recording medium.

[0122] In addition, a portion of the present invention may be implemented as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide part or all of the method and / or technical solution according to the present invention through the operation of the computer. The program instructions for invoking the method of the present invention may be stored in a fixed or removable recording medium, and / or transmitted via a data stream in a broadcast or other signal-carrying medium, and / or stored in the working memory of a computer device that operates according to the program instructions.

[0123] According to another aspect of the present invention, a data processing device for air cargo is provided, wherein the device comprises: a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute part or all of the methods and / or technical solutions of the aforementioned embodiments and / or optional embodiments.

[0124] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments and / or optional embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalents of the claims be included in the present invention. Any figure marks in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software and / or hardware. Words such as first, second, etc. are used to indicate names and do not indicate any particular order.

Claims

1. A data processing system for air cargo, characterized in that: The data processing system comprises: A basic data setting module is used to set basic data, wherein the basic data at least includes: flight data and the business department associated with the flight, wherein the flight data includes: flight number, aircraft type, route segment and time; Rule setting module, used to set data processing rules; A data acquisition module is used to synchronously acquire source data from a data source, wherein the source data includes: air cargo planned flight data and planned transportation data within the assessment period; a data processing module, configured to pre-process the planned flight data and the planned transportation data, and label the pre-processed data based on the basic data to obtain the planned operation data for the assessment period; The assessment index generation module is used to process the planned operating data according to the data processing rules to obtain the assessment index of each route segment in the assessment period, wherein the assessment index corresponds to the route segment and time, and the associated business department.

2. The data processing system according to claim 1, wherein: The data processing rules include: Cardinality generation rules; Coefficient generation rules; Indicator generation rules.

3. The data processing system according to claim 2, wherein: The preprocessing of the planned flight data and the planned transportation data includes: The planned flight data and planned transportation data are deduplicated and formatted in a standardized manner.

4. The data processing system according to claim 2, wherein: The pre-processed data is labeled based on the basic data to obtain the planned operating data for the assessment period, including: According to the flight number in the preprocessed data, the aircraft type, route segment and time, and business department corresponding to the flight number in the basic data are used as labels, and the preprocessed data is labeled to obtain the planned operating data for the assessment period.

5. The data processing system according to claim 2, wherein: The planned operating data is processed according to the data processing rules to obtain the assessment indicators for each route segment in the assessment period, including: Processing the planned operating data according to the cardinality generation rule to obtain cardinality data for each flight in the assessment period; Processing the planned operating data according to the coefficient generation rule to obtain coefficient data for each flight in the assessment period; According to the indicator generation rule, the base data and the coefficient data are processed to obtain the assessment indicator of each route segment in the assessment period.

6. The data processing system according to claim 1, wherein: The source data also includes: flight data and operational data completed within the assessment period, wherein the data processing system also includes: A performance evaluation module is used to obtain the completed indicators for each route segment in the evaluation period based on the evaluation indicators of each route segment in the evaluation period, the flight data completed in the evaluation period, and the completed operation data; The indicator completion rate of each assessment object is determined based on the assessment indicators of each route segment in the assessment period and the completed indicators of each route segment in the assessment period, wherein the assessment objects include: routes or sales departments.

7. The data processing system according to claim 6, wherein: The basic data also includes: the person in charge associated with the business department, wherein the assessment object also includes the person in charge.

8. The data processing system according to claim 6, wherein: The performance appraisal module is also used to: The performance of the assessment subject is determined based on the assessment subject's indicator completion rate and preset performance rules.

9. The data processing system according to any one of claims 1 to 8, characterized in that: The data processing system further includes: Data query and analysis module, used for data query and analysis.

10. The data processing system according to claim 9, wherein: The data processing system further includes: The appeal and review module is used for the assessment subjects to submit appeals and to review the appeals.

11. A data processing method for air cargo, characterized in that: The data processing method includes: Set up basic data and data processing rules; Synchronously obtain source data from a data source, wherein the source data includes: planned air cargo flight data and planned transportation data within the assessment period; Preprocessing the planned flight data and the planned transportation data, and labeling the preprocessed data based on the basic data to obtain the planned operating data for the assessment period; The planned operating data is processed according to the data processing rules to obtain the assessment indicators of each route segment in the assessment period.

12. The data processing method according to claim 11, wherein the source data further comprises: The data of completed flights and completed operations during the assessment period are characterized in that the data processing method further includes: Based on the assessment indicators of each route segment in the assessment period, the flight data and the completed operating data within the assessment period, the completed indicators of each route segment in the assessment period are obtained, and based on the assessment indicators of each route segment in the assessment period and the completed indicators of each route segment in the assessment period, the indicator completion rate of each assessment object is determined, wherein the assessment object includes: route or business department.

13. The data processing method according to claim 12, characterized in that: The data processing method further includes: The performance of the assessment subject is determined based on the assessment subject's indicator completion rate and preset performance rules.

14. A computer-readable medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions are executed by a processor to implement part or all of the method according to any one of claims 11 to 13.

15. A data processing device for air cargo, characterized in that: The device comprises: one or more processors; and A memory storing computer-readable instructions, which, when executed, cause the processor to perform part or all of the operations of the method according to any one of claims 11 to 13.

Images