Method, device and electronic equipment for rebuilding international freight database

Abstract

The application discloses a reconstruction method and device of an international freight database and electronic equipment. The method is applied to the field of data reconstruction. The method comprises the following steps: performing data analysis operations on data in N tasks in parallel through N processes to generate structured data of each task in the N tasks; grouping the structured data of each task based on a business primary key to obtain M groups corresponding to the N tasks; performing data reconstruction operations on data in the M groups corresponding to the N tasks in parallel through M sub-processes to generate M file databases corresponding to the N tasks; and merging the M file databases corresponding to the N tasks according to a partition order, and determining the reconstructed international freight database according to the merged file databases. Through the application, the problem that the database reconstruction efficiency is low due to large data volume and the data must be processed in time sequence in the process of reconstructing the international freight database by an airline in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of data reconstruction, and more specifically, to a method, apparatus, and electronic device for reconstructing an international freight rate database. Background Technology

[0002] Traditional methods for building and maintaining international civil aviation fare databases primarily rely on a serial processing mechanism. This involves loading raw data batch by batch onto a data processing server, then parsing and storing it in a dedicated file database for international civil aviation fare calculations. This process follows the chronological order of data release, ensuring accuracy and timeliness. However, with the continuous expansion of data volume, especially when processing fares and rules spanning up to two years, database reconstruction becomes extremely time-consuming, typically requiring about a week to complete. This involves processing nearly 20,000 incremental update packages, including decompression, parsing, editing, and updating indexes. This lengthy reconstruction cycle under this serial processing model not only affects system response speed but also limits the efficiency of data recovery and off-site reconstruction, becoming a critical problem that the industry urgently needs to address.

[0003] Existing solutions, while attempting to introduce parallel processing in some stages, such as the data decompression process, suffer from limited overall performance improvement due to the lack of effective data grouping and editing strategies, especially the failure to balance the processing load of different data types. The processing time for some large data types (such as integer fare) is too long, which seriously slows down the overall process.

[0004] There is currently no effective solution to the problem of low database reconstruction efficiency for airlines during the reconstruction of international fare databases, which is caused by the large amount of data and the requirement to process data in chronological order. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, and electronic device for reconstructing an international fare database, in order to solve the problem of low database reconstruction efficiency caused by the large amount of data and the requirement to process data in chronological order during the reconstruction of international fare databases by airlines in related technologies.

[0006] To achieve the above objectives, according to one aspect of this application, a method for reconstructing an international freight rate database is provided. The method includes: performing data parsing operations on data in N tasks in parallel using N processes to generate structured data for each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in the international freight rate database to be reconstructed, and N is a positive integer; grouping the structured data of each of the N tasks based on a business primary key to obtain M groups corresponding to the N tasks, where M is a positive integer; performing data reconstruction operations on the data in the M groups corresponding to the N tasks in parallel using M sub-processes to generate M groups of file databases corresponding to the N tasks; merging the M groups of file databases corresponding to the N tasks according to partitioning order, and determining the reconstructed international freight rate database based on the merged file databases.

[0007] Furthermore, before performing data parsing operations on the data in N tasks in parallel through N processes, the method further includes: reading the file directory corresponding to the international freight rate database to be reconstructed, obtaining the full data and the incremental data from the file directory; determining the data processing task corresponding to the full data based on the full data; determining the modification time of each file contained in the incremental data, determining N-1 data processing tasks corresponding to the incremental data for each file contained in the incremental data according to the modification time; and determining the N tasks based on the data processing task corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0008] Furthermore, by using N processes in parallel to perform data parsing operations on the data in N tasks to generate structured data for each of the N tasks, the process includes: creating L threads for the data processing task corresponding to the full data in the N tasks, where L is a positive integer; using the L threads in parallel to decompress the L compressed files in the data processing task corresponding to the full data to obtain L readable data; and using the L threads in parallel to parse the L readable data into structured data based on a predefined data structure to generate the structured data for the data processing task corresponding to the full data.

[0009] Furthermore, by using N processes to perform data parsing operations on the data in N tasks in parallel, structured data for each of the N tasks is generated. This includes: creating N-1 processes for the N-1 data processing tasks corresponding to the incremental data in the N tasks; using the N-1 processes to decompress and parse the compressed files corresponding to the N-1 data processing tasks corresponding to the incremental data in parallel, and after decompressing and parsing the compressed file corresponding to the Xth data processing task corresponding to the incremental data, decompressing and parsing the compressed file corresponding to the (X+2)th data processing task corresponding to the incremental data, where X is a positive integer greater than 0 and less than N-2; after decompressing and parsing the compressed file corresponding to the (N-3)th data processing task corresponding to the incremental data, decompressing and parsing the compressed files corresponding to the (N-2)th and (N-1)th data processing tasks corresponding to the incremental data, to obtain the structured data for the N-1 data processing tasks corresponding to the incremental data.

[0010] Further, the structured data of each of the N tasks is grouped based on the business primary key to obtain M groups corresponding to the N tasks. This includes: calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in the N tasks; grouping the structured data of the data processing task corresponding to the full data according to the hash value of each data record in the full data to obtain M first groups; calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; and if the hash value of the business primary key of any target group in the M first groups is the same as the hash value of any target data record in the structured data of the data processing task corresponding to the incremental data, the target data in the incremental data is assigned to the target group in the M first groups to obtain the M groups corresponding to the N tasks.

[0011] Furthermore, by using M subprocesses in parallel to perform data reconstruction operations on the data in the M groups corresponding to the N tasks, a file database of M groups corresponding to the N tasks is generated. This includes: creating M subprocesses for the data in the M groups, and using the M subprocesses to perform data processing operations and index update operations on the structured data of the data processing tasks corresponding to the full data in the N tasks, wherein the data processing operations are used to reconstruct the international freight rate database; after detecting that the data processing operations and index update operations on the full data have been completed, using the M subprocesses in parallel to perform data reconstruction operations on the structured data of the N-1 data processing tasks corresponding to the incremental data in the N tasks. The data processing and index update operations are performed. After the data processing and index update operations are completed on the structured data corresponding to the Y-th data processing task corresponding to the incremental data, the data processing and index update operations are performed on the structured data corresponding to the (Y+1)-th data processing task corresponding to the incremental data, where Y is a positive integer greater than 0 and less than N-1. After the data processing and index update operations are completed on the structured data corresponding to the (N-2)-th data processing task corresponding to the incremental data, the data processing and index update operations are performed on the structured data corresponding to the (N-1)-th data processing task corresponding to the incremental data, to obtain the M-group file database.

[0012] Further, the M groups of file databases corresponding to the N tasks are merged according to the partition order, and the reconstructed international freight rate database is determined based on the merged file database. This includes: determining the partition information of the partition where each file database in the M groups of file databases is located based on the current storage space, wherein the partition information includes at least: partition size and partition order; merging the M groups of file databases in different partitions according to the partition order and the business primary key to obtain the merged file database; updating the source index information of the M groups of file databases before merging based on the partition size, partition order, and business primary key to obtain target index information; and determining the reconstructed international freight rate database based on the target index information and the merged file database.

[0013] To achieve the above objectives, according to another aspect of this application, an apparatus for reconstructing an international freight rate database is provided. The apparatus includes: a parsing unit, configured to perform data parsing operations on data in N tasks in parallel through N processes, generating structured data for each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in the international freight rate database to be reconstructed, and N is a positive integer; a grouping unit, configured to group the structured data of each of the N tasks based on a business primary key, obtaining M groups corresponding to the N tasks, where M is a positive integer; a reconstruction unit, configured to perform data reconstruction operations on the data in the M groups corresponding to the N tasks in parallel through M sub-processes, generating M groups of file databases corresponding to the N tasks; and a merging unit, configured to merge the M groups of file databases corresponding to the N tasks according to partition order, and determine the reconstructed international freight rate database based on the merged file databases.

[0014] Furthermore, the apparatus further includes: an acquisition unit, configured to read the file directory corresponding to the international freight rate database to be reconstructed, and acquire the full data and the incremental data from the file directory before performing data parsing operations on the data in N tasks in parallel through N processes; a first determination unit, configured to determine the data processing task corresponding to the full data based on the full data; a second determination unit, configured to determine the modification time of each file contained in the incremental data, and determine N-1 data processing tasks corresponding to the incremental data according to the modification time; and a third determination unit, configured to determine the N tasks based on the data processing task corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0015] Further, the parsing unit includes: a first creation subunit, used to create a first process for the data processing task corresponding to the full data in the N tasks; a decompression subunit, used to decompress multiple compressed files in the data processing task corresponding to the full data in parallel using multiple threads in the first process to obtain multiple readable data; and a first parsing subunit, used to parse the multiple readable data into structured data in parallel using the multiple threads based on a predefined data structure to generate structured data for the data processing task corresponding to the full data.

[0016] Further, the parsing unit includes: a second creation subunit, used to create N-1 processes for the N-1 data processing tasks corresponding to the incremental data in the N tasks; a second parsing subunit, used to decompress and parse the compressed files corresponding to the N-1 data processing tasks corresponding to the incremental data in parallel using the N-1 processes, and after the decompression and parsing of the compressed file corresponding to the Xth data processing task corresponding to the incremental data is completed, to decompress and parse the compressed file corresponding to the X+2th data processing task corresponding to the incremental data, where X is a positive integer greater than 0 and less than N-2; and a third parsing subunit, used to decompress and parse the compressed files corresponding to the N-2th and N-1th data processing tasks corresponding to the incremental data after the decompression and parsing of the compressed file corresponding to the N-3th data processing task corresponding to the incremental data is completed, to obtain the structured data of the N-1 data processing tasks corresponding to the incremental data.

[0017] Further, the grouping unit includes: a first calculation subunit, used to calculate the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in the N tasks; a grouping subunit, used to group the structured data of the data processing task corresponding to the full data according to the hash value corresponding to each data record in the full data, to obtain M first groups; a second calculation subunit, used to calculate the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; and a partitioning subunit, used to partition the target data in the incremental data into the target group in the M first groups if the hash value of the business primary key corresponding to any target group in the M first groups is the same as the hash value corresponding to any target data record in the structured data of the data processing task corresponding to the incremental data, to obtain the M groups corresponding to the N tasks.

[0018] Further, the reconstruction unit includes: a third creation subunit, configured to create M subprocesses for the data in the M groups, and use the M subprocesses to perform data processing operations and index update operations on the structured data of the data processing tasks corresponding to the full data in the N tasks, wherein the data processing operations are used to reconstruct the international freight rate database; and a first processing subunit, configured to, after detecting the completion of the data processing operations and index update operations on the full data, use the M subprocesses in parallel to perform data processing operations and index update operations on the structured data of the N-1 data processing tasks corresponding to the incremental data in the N tasks, and... After performing data processing and index update operations on the structured data corresponding to the Yth data processing task corresponding to the incremental data, the second processing subunit performs data processing and index update operations on the structured data corresponding to the (Y+1)th data processing task corresponding to the incremental data, where Y is a positive integer greater than 0 and less than N-1; the second processing subunit is used to perform data processing and index update operations on the structured data corresponding to the (N-2)th data processing task corresponding to the incremental data after performing data processing and index update operations on the structured data corresponding to the (N-1)th data processing task corresponding to the incremental data, to obtain the M sets of file databases.

[0019] Further, the merging unit includes: a first determining subunit, used to determine the partition information of the partition where each file database in the M group file database is located based on the current storage space, wherein the partition information includes at least: partition size and partition order; a merging subunit, used to merge the M group file databases in different partitions according to the partition order and the business primary key to obtain the merged file database; an updating subunit, used to update the source index information of the M group file databases before merging based on the partition size, the partition order and the business primary key to obtain target index information; and a second determining subunit, used to determine the reconstructed international freight rate database based on the target index information and the merged file database.

[0020] To achieve the above objectives, according to one aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method for reconstructing the international freight rate database as described in any of the above claims, and the computer program, when executed by a processor, implements the steps of the method for reconstructing the international freight rate database as described in various embodiments of this application.

[0021] To achieve the above objectives, according to one aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including stored computer instructions, wherein, when the computer instructions are executed by a processor, the method for reconstructing the international freight rate database described in any one of the above claims is implemented.

[0022] To achieve the above objectives, according to one aspect of this application, an electronic device is provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method for reconstructing the international freight rate database described in any of the above claims.

[0023] This application employs the following steps: N processes perform parallel data parsing operations on the data in N tasks to generate structured data for each of the N tasks. The N tasks include data processing tasks corresponding to the full and incremental data in the international fare database to be reconstructed, where N is a positive integer. The structured data for each of the N tasks is grouped based on the business primary key, resulting in M ​​groups corresponding to the N tasks, where M is a positive integer. M sub-processes perform parallel data reconstruction operations on the data in the M groups corresponding to the N tasks, generating M sets of file databases corresponding to the N tasks. The M sets of file databases corresponding to the N tasks are merged according to partition order, and the reconstructed international fare database is determined based on the merged file database. This solves the problem in related technologies where airlines face low database reconstruction efficiency due to large data volumes and the need to process data in chronological order during the reconstruction of international fare databases.

[0024] By implementing concurrent data reconstruction and partition merging strategies, all data processing tasks can be carried out synchronously, significantly improving the efficiency of structured data generation and significantly shortening the reconstruction time of the international freight rate database, thus achieving the technical effect of accelerating the data processing flow. Simultaneously, intelligent partition merging of each task is achieved through multi-processing, making the data integration process both efficient and accurate. This enables rapid recovery of international freight rate data and consistent reconstruction of cross-regional databases, overcoming the inefficiency bottleneck of traditional serial processing methods. Especially when processing large-scale datasets, it fully utilizes the multi-core processing capabilities of the server, avoiding waste of system resources while ensuring data integrity and orderly storage according to its publication time, further enhancing the flexibility of data storage space maintenance and the reliability of data services. Attached Figure Description

[0025] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0026] Figure 1 This is a flowchart of the method for reconstructing the international freight rate database according to Embodiment 1 of this application;

[0027] Figure 2 This is a schematic diagram illustrating the optional data to be reconstructed task division provided in Embodiment 1 of this application;

[0028] Figure 3 This is a flowchart illustrating the optional data processing task corresponding to processing all data according to Embodiment 1 of this application;

[0029] Figure 4 This is a flowchart illustrating the optional data processing task corresponding to incremental data provided in Embodiment 1 of this application;

[0030] Figure 5 This is a flowchart illustrating the optional merging of file databases corresponding to different data processing tasks according to Embodiment 1 of this application;

[0031] Figure 6 This is a schematic diagram of the process for reconstructing optional international freight rate data according to Embodiment 1 of this application;

[0032] Figure 7 This is a schematic diagram of an international freight rate database reconstruction device according to Embodiment 2 of this application;

[0033] Figure 8 This is a schematic diagram of an electronic device for reconstructing an international freight rate database according to Embodiment 5 of this application. Detailed Implementation

[0034] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the user information (including but not limited to user device information, user personal information, collected data, used data, generated data, processed data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, collected information, used information, generated information, processed information, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, and necessary confidentiality measures have been taken. These measures do not violate public order and good morals, and corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0036] It should be noted that this application provides users with a corresponding entry point for choosing to agree to or reject the automated decision-making results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] Example 1

[0040] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of the method for reconstructing the international freight rate database according to Embodiment 1 of this application, as follows: Figure 1 As shown, the method includes the following steps:

[0041] Step S101: Data parsing operations are performed on the data in N tasks in parallel by N processes to generate structured data for each of the N tasks. The N tasks include data processing tasks corresponding to the full data and incremental data in the international freight rate database to be reconstructed, where N is a positive integer.

[0042] In the international civil aviation transportation industry, fare data is a crucial component. Fare data is typically released regularly in two forms: full data (all adds files) containing all fare information at a specific point in time, while update data reflects fare changes since the last release.

[0043] In this first embodiment, to improve database reconstruction efficiency, multiple processes can be used to process the full data and incremental data in parallel. For example, N processes (N is a positive integer) are started, corresponding to N independent data processing tasks, including full data processing of the international freight rate database and multiple incremental data processing tasks. Each process is responsible for one task, performing data parsing operations, decompressing the original compressed file and parsing it into structured data records, generating the structured data for its respective task.

[0044] It is important to note that this process strictly follows the data processing order to ensure that incremental data is processed one by one in the order of publication time after the full data processing is completed, in order to maintain the integrity and accuracy of the data.

[0045] Step S102: Group the structured data of each of the N tasks based on the business primary key to obtain M groups corresponding to the N tasks, where M is a positive integer.

[0046] In this first embodiment, a grouping strategy based on business primary keys is implemented for the structured data of each data processing task. This grouping strategy involves processing the business primary key in each data record, and then performing M-way grouping on the structured data in each task, where M is a positive integer representing the number of groups. The grouped data ensures that records with the same business primary key are grouped together, facilitating subsequent editing operations.

[0047] It's important to note that this grouping method considers not only data type but also the differences in data volume across different types, optimizing processing speed and ensuring that all processes complete tasks at a similar rate, thereby improving overall processing efficiency. Each task generates M groups, independent of other tasks, providing conditions for parallel execution of index editing and updating. Through the above data grouping steps, a uniform distribution of data records is achieved, avoiding resource imbalances and processing delays caused by a single process handling too much data.

[0048] Step S103: Data reconstruction operations are performed in parallel on the data in the M groups corresponding to the N tasks by M subprocesses to generate the M group file databases corresponding to the N tasks.

[0049] In this first embodiment, after data grouping, M subprocesses are launched to perform data reconstruction operations in parallel for M groups corresponding to N tasks. Each subprocess focuses on its assigned group, independently performing data editing and index updates, ensuring that the processing is non-blocking and efficient.

[0050] By executing in parallel, subprocesses can quickly process the grouped data and generate the target data, i.e., the reconstructed data. This method fully utilizes the capabilities of multi-core processors, reduces processing time, and maintains the sequentiality of data processing, avoiding database consistency issues caused by improper concurrent operations. Ultimately, after the data reconstruction operations of N tasks are completed, M sets of file databases are generated.

[0051] Step S104: Merge the M groups of file databases corresponding to N tasks according to the partition order, and determine the reconstructed international freight rate database based on the merged file database.

[0052] In this first embodiment, the data merging step is performed according to the preset partitioning order of the current storage space. This process involves integrating M groups of file databases corresponding to N tasks. Ultimately, the merged target data constitutes the reconstructed international freight rate database, maintaining data integrity and correct sorting according to data release time, achieving efficient database reconstruction, and thus meeting the needs for rapid recovery and off-site reconstruction of international freight rate data.

[0053] It's important to note that after completing the data reconstruction operations for N tasks, M file databases will be generated. Each file database contains independent data, and there is no duplicate data between any two file databases. Therefore, the data record merging can be completed simply by merging the files, without needing to traverse every single data entry, thus improving database creation speed. Furthermore, to fully utilize the server's multi-core hardware capabilities, the data merging process can be performed concurrently, with each data type processed as an independent unit.

[0054] In summary, the international fare database reconstruction method provided in Embodiment 1 of this application involves parallel data parsing operations on data from N tasks using N processes to generate structured data for each of the N tasks. The N tasks comprise data processing tasks corresponding to the full and incremental data in the international fare database to be reconstructed, where N is a positive integer. The structured data for each of the N tasks is grouped based on the business primary key, resulting in M ​​groups corresponding to the N tasks, where M is a positive integer. M sub-processes then perform parallel data reconstruction operations on the data in the M groups corresponding to the N tasks, generating M sets of file databases corresponding to the N tasks. Finally, the M sets of file databases corresponding to the N tasks are merged according to partition order, and the reconstructed international fare database is determined based on the merged file database. This method solves the problem in related technologies where airlines face low database reconstruction efficiency due to the large data volume and the requirement to process data in chronological order during the reconstruction of international fare databases.

[0055] By implementing concurrent data reconstruction and partition merging strategies, all data processing tasks can proceed synchronously, significantly improving the efficiency of structured data generation and substantially shortening the reconstruction time of the international freight rate database, thus accelerating the data processing workflow. Simultaneously, intelligent partition merging of each task through thread pool technology ensures both high efficiency and accuracy in data integration, enabling rapid recovery of international freight rate data and consistent reconstruction of cross-regional databases. This overcomes the inefficiency bottleneck of traditional serial processing methods. Especially when processing large-scale datasets, it fully utilizes the multi-core processing capabilities of the server, avoiding waste of system resources while ensuring data integrity and orderly storage according to its publication time, further enhancing the flexibility of data storage space maintenance and the reliability of data services.

[0056] Optionally, in the method for reconstructing the international freight rate database provided in Embodiment 1 of this application, before performing data parsing operations on the data in N tasks in parallel through N processes, the method further includes: reading the file directory corresponding to the international freight rate database to be reconstructed, and obtaining full data and incremental data from the file directory; determining the data processing task corresponding to the full data based on the full data; determining the modification time of each file contained in the incremental data, and determining N-1 data processing tasks corresponding to the incremental data for each file contained in the incremental data according to the modification time; and determining N tasks based on the data processing tasks corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0057] In this first embodiment, in order to rebuild the international freight rate database, it is necessary to scan the file directory related to the reconstruction of the international freight rate database in the storage area, accurately identify the full data files and incremental data files in the directory, and ensure the complete acquisition of all data to be reconstructed.

[0058] Then, based on the information from the full dataset, the first data processing task is generated as the starting point for rebuilding the database.

[0059] Secondly, by analyzing the last modification timestamp of each incremental data file, the incremental files are associated with N-1 subsequent data processing tasks in chronological order. This ensures the continuity and temporal order of data updates, where N and N-1 are both positive integers. For example, all incremental data is divided into multiple jobs based on a certain quantity (e.g., dividing 1000 hours of incremental data into one job).

[0060] Finally, the first full data processing task and N-1 incremental data processing tasks are integrated to form N ordered, independently executable data processing jobs. For example, Figure 2 This is a schematic diagram of the optional data to be reconstructed task provided in Embodiment 1 of this application. Figure 2 Job0 in this context refers to the data processing task corresponding to the full dataset mentioned above. Figure 2 Job1 to JobN in the above context refer to the N-1 data processing tasks corresponding to the incremental data mentioned above.

[0061] Through the above steps, the technical effect of accurately planning the data processing task flow is achieved, ensuring the systematic and efficient nature of data reconstruction work, and providing a solid foundation for subsequent parallel processing is achieved.

[0062] Optionally, in the method for reconstructing the international freight rate database provided in Embodiment 1 of this application, data parsing operations are performed on the data in N tasks in parallel by N processes to generate structured data for each of the N tasks. This includes: creating a first process for the data processing task corresponding to the full data in the N tasks; using multiple threads in the first process to decompress multiple compressed files in the data processing task corresponding to the full data in parallel to obtain multiple readable data; and using multiple threads in parallel to parse the multiple readable data into structured data based on a predefined data structure to generate the structured data for the data processing task corresponding to the full data.

[0063] In this first embodiment, based on the characteristics of the full data processing task, multi-threading technology can be employed. L threads are determined and created based on the first process, where the value of L is based on the server's processing power to ensure the rational allocation and utilization of computing resources. Then, multiple threads start simultaneously, each responsible for decompressing a compressed file. This parallel processing method significantly accelerates the decompression process of the full data, quickly generating multiple readable data files, preparing the data source for subsequent parsing. Next, these readable data files are parsed simultaneously using multiple threads, transforming unstructured text information into a predefined data structure, namely the reconstructed international freight rate data structure. Finally, through parallel operations by multiple threads, the structured datasets generated by the multiple threads are integrated to form the complete data required for the full data processing task, providing a foundation for subsequent data editing and index updates.

[0064] Through the above steps, the technical effect of accelerating the decompression and parsing of full data through multi-threaded parallel processing is achieved. This not only shortens data preparation time but also ensures data accuracy and processing efficiency, laying a solid data foundation for the subsequent database reconstruction process. This strategy fully utilizes the capabilities of modern multi-core processors, effectively balancing processing speed and resource consumption, and its advantages are particularly evident when dealing with large-scale datasets.

[0065] Figure 3 This is a flowchart illustrating an optional data processing task corresponding to processing all data, provided in Embodiment 1 of this application. For example... Figure 3 As shown, after the main process (i.e., the first process mentioned above) starts, it reads the directory structure of all full and incremental data, sorts it by release time, and then divides it into Jobs. Since the full data volume is large, with dozens of compressed files, it can be divided into a separate Job0. A thread pool is created, with each thread responsible for decompressing one file and parsing it as a text file. After all parsing work is completed, a large amount of structured data is generated.

[0066] Optionally, in the method for reconstructing the international freight rate database provided in Embodiment 1 of this application, data parsing operations are performed on the data in N tasks in parallel by N processes to generate structured data for each of the N tasks. This includes: creating N-1 processes for N-1 data processing tasks corresponding to incremental data in the N tasks; using N-1 processes in parallel to decompress and parse the compressed files corresponding to the N-1 data processing tasks corresponding to incremental data, and after the decompression and parsing of the compressed file corresponding to the Xth data processing task corresponding to incremental data is completed, decompressing and parsing the compressed file corresponding to the X+2th data processing task corresponding to incremental data is performed, where X is a positive integer greater than 0 and less than N-2; after the decompression and parsing of the compressed file corresponding to the N-3th data processing task corresponding to incremental data is completed, decompressing and parsing the compressed files corresponding to the N-2th and N-1th data processing tasks corresponding to incremental data is performed to obtain the structured data of the N-1 data processing tasks corresponding to incremental data.

[0067] In this first embodiment, based on N-1 tasks involving incremental data processing, a corresponding number of independent processes are created to ensure that each incremental data task has its own dedicated processing unit, namely the aforementioned N-1 processes, providing structural support for parallel processing.

[0068] Then, these N-1 threads begin processing their respective assigned compressed files in parallel, performing decompression and parsing operations to generate structured data. It's important to note that this parallel processing follows a specific dependency: the decompression and parsing process for the X+2 data processing task only begins after the compressed file corresponding to the Xth data processing task has been decompressed and parsed, where X is a positive integer greater than 0 and less than N-2. By controlling the execution order between threads, resource conflicts and data corruption caused by concurrent operations can be avoided, ensuring the accuracy and consistency of data processing. Here, N, N-1, N-2, N-3, X, and X+2 are all positive integers.

[0069] Finally, the (N-1)th and (N-2)th data processing tasks in the incremental data can be executed simultaneously. Subsequently, the structured data for all N-1 incremental data processing tasks is generated one by one, and the processing of each task is carried out in the order of data release time, providing a data foundation for the subsequent data grouping and data reconstruction stages.

[0070] In one alternative embodiment, Figure 4 This is a flowchart illustrating the optional data processing task corresponding to incremental data processing according to Embodiment 1 of this application. Figure 4As shown, the incremental data processing flow is similar to the full data processing flow, except that the execution of each Job needs to wait for the processing status of the preceding Job. This is because all Jobs do not start simultaneously; starting them all would consume all CPU and memory resources, rendering the system unavailable. Furthermore, during the data reconstruction phase, the business logic requires that the preceding Job's data editing be completed; otherwise, data errors will occur.

[0071] Specifically, during the data decompression and parsing phase, if JobZ is currently executing, it will observe whether the preceding Job, i.e., JobZ-2, has completed. If not, it will sleep and wait. Therefore, although multiple processes are created simultaneously, due to the execution dependencies of the processes (such as...), Figure 4 The execution of the JobZ process depends on the completion of the Z-2 process. Most processes are in a sleep state and do not consume CPU resources. To further explain, although they are waiting, CPU resources are not idle and can be used to handle the data reconstruction process of JobZ-1 or JobZ-2 (such as the data processing operations and index update operations described below).

[0072] Through the above steps, the technical effect of fully utilizing multi-threaded parallel processing capabilities while ensuring the timeliness of data processing was achieved, significantly improving the processing efficiency of incremental data and creating favorable conditions for the efficient reconstruction of the international civil aviation fare database. This strategy not only optimized the processing flow and reduced waiting time, but also avoided potential bottlenecks when processing large-scale datasets, thereby accelerating the database update speed, enhancing the recovery capability of international fare data, and improving the feasibility of off-site database reconstruction.

[0073] Optionally, in the international freight rate database reconstruction method provided in Embodiment 1 of this application, the structured data of each of the N tasks is grouped based on the business primary key to obtain M groups corresponding to the N tasks. This includes: calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in the N tasks; grouping the structured data of the data processing task corresponding to the full data according to the hash value of each data record in the full data to obtain M first groups; calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; and, if the hash value of the business primary key of any target group in the M first groups is the same as the hash value of any target data record in the structured data of the data processing task corresponding to the incremental data, the target data in the incremental data is assigned to the target group in the M first groups to obtain M groups corresponding to the N tasks.

[0074] In this first embodiment, to evenly group the data, the hash value corresponding to the business primary key of each record in the full dataset can be read and calculated, ensuring that the same business primary key produces a consistent hash value. The business primary key can be one or more fields from a data table in the international freight rate database, and is not specifically limited in this first embodiment.

[0075] Then, based on the generated first hash value, the structured data of the entire dataset is divided into M first groups, where M is the pre-defined total number of groups. This number can be determined by the data types contained in the international freight rate database, or by the amount of data in the international freight rate database; no specific limitation is made in this embodiment. It is important to note that due to the distributed nature of hash values, this grouping process ensures that data is evenly distributed across the groups, with each process handling roughly the same amount of data. This prevents any one process from experiencing a significantly longer processing time than others. This even distribution strategy reduces the overall system processing latency.

[0076] In existing technologies, the general grouping strategy is based on data type, with the same data type processed in one process. However, the drawback is that the amount of data for each data type varies greatly, with some data types (intallfare) accounting for more than 30% of the total. As a result, other processes have already finished processing while this process is still incomplete, leading to low overall processing efficiency for data reconstruction.

[0077] Finally, for the processing of incremental data, a hash function is also applied to calculate the second hash value of the business primary key for each record. Based on these values, the structured data in each data processing task within the incremental data is allocated to the aforementioned M first groups. Combining the results of grouping the full data and incremental data, M groups corresponding to the N tasks are established.

[0078] In one optional embodiment, for structured data of type P02 in a certain task, M is set to 3. The data is grouped according to the business primary key fields A, B, and C of the P02 data type, and a hash value is calculated. Then, the remainder is taken with respect to the total number of groups. The calculation results are shown in Table 1. The hash calculation results can represent the partition numbers of different data, as shown in Table 1. The three data entries in Table 1 are split into partition 2, partition 0, and partition 1. Each partition subprocess independently reconstructs the data.

[0079] Table 1

[0080]

[0081] The above steps achieve the technical effect of efficiently grouping structured data, which facilitates subsequent parallel data editing and index update operations. Because the data is rationally distributed across multiple groups, each group can be processed independently, reducing resource contention during processing and ensuring the correctness and integrity of the data. Furthermore, the hash-based grouping method based on business primary keys can dynamically adapt to the processing needs of different data types, thus improving the reconstruction efficiency of high-performance databases.

[0082] Optionally, in the international freight rate database reconstruction method provided in Embodiment 1 of this application, M subprocesses are used in parallel to perform data reconstruction operations on the data in M ​​groups corresponding to N tasks, generating M group file databases corresponding to N tasks. This includes: creating M subprocesses for the data in the M groups; using the M subprocesses to perform data processing operations and index update operations on the structured data of the data processing tasks corresponding to the full data in the N tasks; wherein the data processing operations are used to reconstruct the international freight rate database; after detecting the completion of the data processing operations and index update operations on the full data, the M subprocesses are used in parallel to perform data reconstruction operations on the N-1 data processing tasks corresponding to the incremental data in the N tasks. The system performs data processing and index update operations on the structured data of each incremental data processing task. After completing the data processing and index update operations on the structured data corresponding to the Y-th data processing task, it performs data processing and index update operations on the structured data corresponding to the (Y+1)-th data processing task, where Y is a positive integer greater than 0 and less than N-1. After completing the data processing and index update operations on the structured data corresponding to the (N-2)-th data processing task, it performs data processing and index update operations on the structured data corresponding to the (N-1)-th data processing task, resulting in M ​​sets of file databases.

[0083] In this first embodiment, for the aforementioned M groups, M independent child processes are launched. Each child process is assigned data processing operations and index update operations for a specific group of data. It's important to note that the `fork` method can be used to create a child process for each task (including the data processing task corresponding to the full data and the data processing task corresponding to the incremental data, i.e., the aforementioned Job). This allows each child process to inherit the memory information of the main process, including task information for the N tasks (i.e., information for all Jobs), and the task sequence number assigned to the current child process (i.e., the Job number). Since each child process inherits the memory data of its parent process, it can read the structured data processed by the parent process. In other words, each child process obtains all the decompressed and parsed data without memory copying, using only one copy of physical memory globally, thus saving CPU and memory resources.

[0084] Then, each subprocess performs data processing operations on its assigned group of data, including but not limited to data editing, rule application, and comparison and updating with existing database records. For example, data processing operations include, but are not limited to, searching for data with the same primary key in the text database based on the data's business primary key; if not found, adding the record directly; otherwise, deleting or modifying it. This parallel processing phase significantly improves data processing speed and reduces the time consumed by a single process processing large volumes of data. After the data processing operations are completed, M subprocesses continue to perform index update operations, establishing or updating the index positions of data records to ensure database query performance.

[0085] Specifically, Figure 4 This is a flowchart illustrating the optional data processing task corresponding to incremental data processing according to Embodiment 1 of this application. Figure 4 As shown, during the execution phases of data processing and index update operations, if JobZ is currently executing, it will observe whether the preceding Job, i.e., JobZ-1, has completed. If not, it will sleep and wait. Therefore, although multiple processes are created simultaneously, due to the execution dependencies of the processes (such as...), Figure 4 The execution of the JobZ child process depends on the completion of the JobZ-1 child process.

[0086] Finally, after data processing and index update operations, the data from the M groups are used to obtain the file database corresponding to the M groups. This database is then merged into the reconstructed international civil aviation fare database, thus completing the reconstruction of the international civil aviation fare database.

[0087] Through the above steps, the technical effect of accelerating the reconstruction of the international civil aviation fare database by using multi-process parallel processing and index updates is achieved. This strategy not only improves data processing efficiency but also simplifies the formation of the final dataset through merging operations, avoiding redundant data processing and storage, and enhancing the performance of the entire international civil aviation fare database reconstruction process. The parallel sub-process processing and index update operations fully utilize the computing power of multi-core processors, providing a more efficient method for database maintenance and updates.

[0088] Optionally, in the method for reconstructing the international freight rate database provided in Embodiment 1 of this application, the M groups of file databases corresponding to N tasks are merged according to the partition order, and the reconstructed international freight rate database is determined based on the merged file database. This includes: determining the partition information of the partition where each file database in the M groups of file databases is located based on the current storage space, wherein the partition information includes at least: partition size and partition order; merging the M groups of file databases in different partitions according to the partition order and the business primary key to obtain the merged file database; updating the source index information of the M groups of file databases before merging according to the partition size, partition order and business primary key to obtain the target index information; and determining the reconstructed international freight rate database based on the target index information and the merged file database.

[0089] In this first embodiment, Figure 5 This is a flowchart illustrating an optional process for merging file databases corresponding to different data processing tasks, provided in Embodiment 1 of this application. To merge and rebuild the data and save storage space, it is necessary to assess the characteristics of the current storage space, including storage capacity and structural layout, and determine the file database corresponding to each group (e.g., ...). Figure 5 The detailed information of the partitions in the storage space of file database group 1 or file database group M, namely the partition information mentioned above, which at least covers the size of the partitions and their order in the storage architecture, to ensure that the physical framework for data merging operations can be constructed.

[0090] Then, based on the determined partitioning order, the business primary key is used as the merging identifier to sequentially join data records belonging to the same business primary key in different partitions (e.g., ...). Figure 5 The operation of merging records in the database achieves logical integration of data, forming a merged file database, which provides a data foundation for building a complete database.

[0091] Secondly, to adapt to the change in storage location after data merging, it is necessary to update the source index information of the file database corresponding to each group, and adjust the index offset information (such as...) based on the partition size, partition order, and business primary key. Figure 5 By merging the indexes in the database, the target index information is obtained, ensuring that the new data location is accurately recorded to ensure database query performance.

[0092] Finally, after integrating the target index information with the merged file database, the final state of the reconstructed international civil aviation fare database was determined, thus completing the update and reconstruction of the international civil aviation fare database.

[0093] Through the above steps, the technical effects of optimizing data storage and indexing, and efficiently rebuilding the international civil aviation fare database were achieved. This strategy not only ensured the accuracy of data merging but also ensured high-efficiency database query performance by dynamically adjusting index information, providing a fast and reliable processing flow for data storage space maintenance and data recovery. Simultaneously, the partition merging method customized based on storage space characteristics made full use of storage resources, reduced the time consumed by data migration and copying, and significantly improved the overall efficiency of database reconstruction.

[0094] In one optional embodiment, taking the P02 data type as an example, Table 2 shows the partition information of the data processing task corresponding to the P02 data type before merging. Before merging, there are three file databases, namely P02 data files in partition 0, partition 1, and partition 2. Each file is actually a memory mirror of an array of P02 data structures. Therefore, by directly merging these three files in partition order (i.e., partition number order), the final file of the complete P02 data type can be obtained.

[0095] The index file can be a red-black binary tree data structure containing two parts: key and value. The key is the primary key string for the data business, and the value is an array of data record offsets (relative to the first character of the data file).

[0096] Table 2

[0097]

[0098] As shown in Table 2, the file size of partition 2 is 40,000 bytes, and a certain primary key (001EK LLCDPTN1) contains three data records with offsets of 0, 40, and 200 respectively (assuming each record is 40 bytes long). After merging the data from these three partitions in the order of partitions 0, 1, and 2, the offsets of their indexes can be shown in Table 3.

[0099] Table 3

[0100]

[0101] As can be seen from Table 3, for the data in the original partition 2, the data in partitions 0 and 1 were inserted in front, and all the data were shifted by 44000 + 48000 = 92000 bytes. Therefore, the value part of its index was uniformly increased by an offset of 92000 bytes.

[0102] Optionally, in this first embodiment, Figure 6 This is a schematic diagram illustrating the reconstruction process of optional international freight rate data provided in Embodiment 1 of this application. Figure 6As shown, the main process reads and sorts the full and incremental data according to their publication time, dividing them into multiple Jobs. The thread pool within each Job is used for parallel decompression and parsing of the data. For data processing tasks corresponding to full data, multiple threads are created using thread pool technology during the data decompression and parsing phases to perform the data decompression and parsing operations. For data processing tasks corresponding to incremental data, during the data decompression and parsing phase, if Job Z is currently being executed, it will observe whether the preceding Job, i.e., Job Z-2, has completed; if not, it will sleep and wait (e.g., ...). Figure 4 The execution of the JobZ process depends on the completion of the Z-2 process. To further clarify, although it is waiting, CPU resources are not idle and can be used to handle the data reconstruction process of JobZ-1 or JobZ-2 (such as the data processing and index update operations described below).

[0103] Then, the data is grouped according to the hash value of the business primary key. Multiple subprocesses are created within each job, and each subprocess independently performs data editing and index updates, resulting in a file database (e.g., M groups) corresponding to each group. Figure 6 (e.g., the first group of file databases, the Mth group of file databases, etc.). Specifically, for data processing tasks corresponding to the full dataset, during the execution phases of data processing and index update operations, M sub-processes perform data processing and index update operations on the structured data in the M groups in parallel. For data processing tasks corresponding to incremental data, during the execution phases of data processing and index update operations, if JobZ is currently being executed, it will observe whether the preceding Job, i.e., JobZ-1, has completed; if not, it will sleep and wait. Therefore, although multiple processes are created simultaneously, due to the execution dependencies of the processes (e.g., ... Figure 4 The execution of the JobZ child process depends on the completion of the JobZ-1 child process.

[0104] Finally, the file databases processed by all subprocesses (i.e., the file databases corresponding to the M groups mentioned above) are merged in partition order, and the offsets of data records in the index files are adjusted to obtain the reconstructed international freight rate database.

[0105] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0106] Example 2

[0107] Embodiment 2 of this application also provides an apparatus for reconstructing an international freight rate database. It should be noted that the apparatus for reconstructing an international freight rate database in Embodiment 2 can be used to execute the method for reconstructing an international freight rate database provided in Embodiment 1 of this application. The apparatus for reconstructing an international freight rate database provided in Embodiment 2 of this application will be described below.

[0108] Figure 7 This is a schematic diagram of an apparatus for reconstructing an international freight rate database according to Embodiment 2 of this application. Figure 7 As shown, the device includes: a parsing unit 701, a grouping unit 702, a reconstruction unit 703, and a merging unit 704.

[0109] Specifically, the parsing unit 701 is used to perform data parsing operations on the data in N tasks in parallel through N processes, and generate structured data for each of the N tasks. The N tasks include data processing tasks corresponding to the full data and incremental data in the international freight rate database to be reconstructed, and N is a positive integer.

[0110] Grouping unit 702 is used to group the structured data of each of the N tasks based on the business primary key, so as to obtain M groups corresponding to the N tasks, where M is a positive integer.

[0111] The reconstruction unit 703 is used to perform data reconstruction operations on the data in the M groups corresponding to the N tasks in parallel through M subprocesses, and generate the M group file databases corresponding to the N tasks.

[0112] Merging unit 704 is used to merge the M groups of file databases corresponding to N tasks according to the partition order, and to determine the reconstructed international freight rate database based on the merged file database.

[0113] The international fare database reconstruction apparatus provided in Embodiment 2 of this application, through a parsing unit 701, performs data parsing operations on the data in N tasks in parallel through N processes, generating structured data for each of the N tasks. The N tasks include data processing tasks corresponding to the full and incremental data in the international fare database to be reconstructed, where N is a positive integer. A grouping unit 702 groups the structured data of each of the N tasks based on the business primary key, obtaining M groups corresponding to the N tasks, where M is a positive integer. A reconstruction unit 703 performs data reconstruction operations on the data in the M groups corresponding to the N tasks in parallel through M sub-processes, generating M groups of file databases corresponding to the N tasks. A merging unit 704 merges the M groups of file databases corresponding to the N tasks according to the partition order, and determines the reconstructed international fare database based on the merged file database. This solves the problem in related technologies where airlines face low database reconstruction efficiency due to the large data volume and the need to process data in chronological order during the reconstruction of international fare databases.

[0114] By implementing concurrent data reconstruction and partition merging strategies, all data processing tasks can proceed synchronously, significantly improving the efficiency of structured data generation and substantially shortening the reconstruction time of the international freight rate database, thus accelerating the data processing workflow. Simultaneously, intelligent partition merging of each task through thread pool technology ensures both high efficiency and accuracy in data integration, enabling rapid recovery of international freight rate data and consistent reconstruction of cross-regional databases. This overcomes the inefficiency bottleneck of traditional serial processing methods. Especially when processing large-scale datasets, it fully utilizes the multi-core processing capabilities of the server, avoiding waste of system resources while ensuring data integrity and orderly storage according to its publication time, further enhancing the flexibility of data storage space maintenance and the reliability of data services.

[0115] Optionally, in the international freight rate database reconstruction apparatus provided in Embodiment 2 of this application, the apparatus further includes: an acquisition unit, configured to read the file directory corresponding to the international freight rate database to be reconstructed and acquire full data and incremental data from the file directory before performing data parsing operations on the data in N tasks in parallel through N processes; a first determination unit, configured to determine the data processing task corresponding to the full data based on the full data; a second determination unit, configured to determine the modification time of each file contained in the incremental data and determine N-1 data processing tasks corresponding to the incremental data according to the modification time; and a third determination unit, configured to determine N tasks based on the data processing tasks corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0116] Optionally, in the international freight rate database reconstruction apparatus provided in Embodiment 2 of this application, the above-mentioned parsing unit 701 includes: a first creation subunit, used to create a first process for the data processing task corresponding to the full data in N tasks; a decompression subunit, used to decompress multiple compressed files in the data processing task corresponding to the full data in parallel using multiple threads in the first process to obtain multiple readable data; and a first parsing subunit, used to parse the multiple readable data into structured data in parallel using multiple threads based on a predefined data structure to generate the structured data of the data processing task corresponding to the full data.

[0117] Optionally, in the international freight rate database reconstruction apparatus provided in Embodiment 2 of this application, the above-mentioned parsing unit 701 includes: a second creation subunit, used to create N-1 processes for N-1 data processing tasks corresponding to incremental data in N tasks; a second parsing subunit, used to decompress and parse the compressed files corresponding to the N-1 data processing tasks corresponding to incremental data in parallel using N-1 processes, and after the decompression and parsing of the compressed file corresponding to the Xth data processing task corresponding to incremental data is completed, to decompress and parse the compressed file corresponding to the X+2th data processing task corresponding to incremental data, where X is a positive integer greater than 0 and less than N-2; and a third parsing subunit, used to decompress and parse the compressed files corresponding to the N-2th and N-1th data processing tasks corresponding to incremental data after the decompression and parsing of the compressed file corresponding to the N-3th data processing task corresponding to incremental data is completed, to obtain the structured data of the N-1 data processing tasks corresponding to incremental data.

[0118] Optionally, in the international freight rate database reconstruction apparatus provided in Embodiment 2 of this application, the grouping unit 702 includes: a first calculation subunit, used to calculate the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in N tasks; a grouping subunit, used to group the structured data of the data processing task corresponding to the full data according to the hash value corresponding to each data record in the full data, to obtain M first groups; a second calculation subunit, used to calculate the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in N tasks; and a partitioning subunit, used to partition the target data in the incremental data into the target group in the M first groups when the hash value of the business primary key corresponding to any target group in the M first groups is the same as the hash value corresponding to any target data record in the structured data of the data processing task corresponding to the incremental data, to obtain M groups corresponding to N tasks.

[0119] Optionally, in the international freight rate database reconstruction apparatus provided in Embodiment 2 of this application, the reconstruction unit 703 includes: a third creation subunit, configured to create M subprocesses for data in M ​​groups, and use the M subprocesses to perform data processing operations and index update operations on the structured data of the data processing tasks corresponding to the full data in N tasks, wherein the data processing operations are used to reconstruct the international freight rate database; and a first processing subunit, configured to, after detecting that the data processing operations and index update operations on the full data have been completed, use the M subprocesses in parallel to perform data processing operations and index update operations on the structured data of the N-1 data processing tasks corresponding to the incremental data in the N tasks. The first processing unit performs an index update operation, and after completing the data processing and index update operations on the structured data corresponding to the Y-th data processing task corresponding to the incremental data, it performs data processing and index update operations on the structured data corresponding to the (Y+1)-th data processing task corresponding to the incremental data, where Y is a positive integer greater than 0 and less than N-1; the second processing subunit is used to perform data processing and index update operations on the structured data corresponding to the (N-2)-th data processing task corresponding to the incremental data, and then perform data processing and index update operations on the structured data corresponding to the (N-1)-th data processing task corresponding to the incremental data, to obtain M sets of file databases.

[0120] Optionally, in the international freight rate database reconstruction apparatus provided in Embodiment 2 of this application, the aforementioned merging unit 704 includes: a first determining subunit, used to determine the partition information of the partition where each file database in the M group file database is located based on the current storage space, wherein the partition information includes at least: partition size and partition order; a merging subunit, used to merge the M group file databases in different partitions according to the partition order and based on the business primary key to obtain the merged file database; an updating subunit, used to update the source index information of the M group file databases before merging based on the partition size, partition order and business primary key to obtain target index information; and a second determining subunit, used to determine the reconstructed international freight rate database based on the target index information and the merged file database.

[0121] The reconstruction device for the international freight rate database includes a processor and a memory. The parsing unit 701, grouping unit 702, reconstruction unit 703, and merging unit 704 mentioned above are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0122] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the efficiency of rebuilding the international freight rate database.

[0123] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0124] Embodiment 3 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a method for reconstructing an international freight rate database.

[0125] Embodiment 4 of the present invention provides a processor for running a program, wherein the program executes a method for reconstructing an international freight rate database during runtime.

[0126] like Figure 8 As shown, Embodiment 5 of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: Data parsing operations are performed on the data in N tasks in parallel by N processes to generate structured data for each of the N tasks, wherein the N tasks include data processing tasks corresponding to the full data and incremental data in the international freight rate database to be reconstructed, and N is a positive integer; The structured data of each of the N tasks is grouped based on the business primary key to obtain M groups corresponding to the N tasks, where M is a positive integer; Data reconstruction operations are performed on the data in the M groups corresponding to the N tasks in parallel by M sub-processes to generate M groups of file databases corresponding to the N tasks; The M groups of file databases corresponding to the N tasks are merged according to the partitioning order, and the reconstructed international freight rate database is determined based on the merged file database.

[0127] When the processor executes the program, it also performs the following steps: Before performing data parsing operations on the data in N tasks in parallel through N processes, the above method also includes: reading the file directory corresponding to the international freight rate database to be reconstructed, and obtaining full data and incremental data from the file directory; determining the data processing task corresponding to the full data based on the full data; determining the modification time of each file contained in the incremental data, and determining N-1 data processing tasks corresponding to the incremental data for each file contained in the incremental data according to the modification time; and determining N tasks based on the data processing tasks corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0128] When the processor executes the program, it also performs the following steps: It uses N processes to perform data parsing operations on the data in N tasks in parallel, generating structured data for each of the N tasks. This includes: creating a first process for the data processing task corresponding to the full data in the N tasks; using multiple threads in the first process to decompress multiple compressed files in the data processing task corresponding to the full data in parallel, obtaining multiple readable data; and using multiple threads in parallel to parse the multiple readable data into structured data based on a predefined data structure, generating the structured data for the data processing task corresponding to the full data.

[0129] When the processor executes the program, it also performs the following steps: Data parsing operations are performed on the data in N tasks in parallel by N processes to generate structured data for each of the N tasks. This includes: creating N-1 processes for the N-1 data processing tasks corresponding to the incremental data in the N tasks; using the N-1 processes in parallel to decompress and parse the compressed files corresponding to the N-1 data processing tasks corresponding to the incremental data, and after decompressing and parsing the compressed file corresponding to the Xth data processing task corresponding to the incremental data, decompressing and parsing the compressed file corresponding to the X+2th data processing task corresponding to the incremental data, where X is a positive integer greater than 0 and less than N-2; after decompressing and parsing the compressed file corresponding to the N-3th data processing task corresponding to the incremental data, decompressing and parsing the compressed files corresponding to the N-2th and N-1th data processing tasks corresponding to the incremental data to obtain the structured data for the N-1 data processing tasks corresponding to the incremental data.

[0130] When the processor executes the program, it also performs the following steps: grouping the structured data of each of the N tasks based on the business primary key to obtain M groups corresponding to the N tasks, including: calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in the N tasks; grouping the structured data of the data processing task corresponding to the full data according to the hash value of each data record in the full data to obtain M first groups; calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; if the hash value of the business primary key of any target group in the M first groups is the same as the hash value of any target data record in the structured data of the data processing task corresponding to the incremental data, the target data in the incremental data is assigned to the target group in the M first groups to obtain M groups corresponding to the N tasks.

[0131] The processor also performs the following steps when executing the program: It uses M subprocesses in parallel to perform data reconstruction operations on the data in M ​​groups corresponding to N tasks, generating a file database of M groups corresponding to N tasks. This includes: creating M subprocesses for the data in the M groups, and using these M subprocesses to perform data processing and index update operations on the structured data of the data processing tasks corresponding to the full data in the N tasks. The data processing operations are used to reconstruct the international freight rate database. After detecting the completion of the data processing and index update operations on the full data, it uses M subprocesses in parallel to process the structured data of the N-1 data processing tasks corresponding to the incremental data in the N tasks. Perform data processing and index update operations. After completing the data processing and index update operations on the structured data corresponding to the Y-th data processing task corresponding to the incremental data, perform data processing and index update operations on the structured data corresponding to the (Y+1)-th data processing task corresponding to the incremental data, where Y is a positive integer greater than 0 and less than N-1. After completing the data processing and index update operations on the structured data corresponding to the (N-2)-th data processing task corresponding to the incremental data, perform data processing and index update operations on the structured data corresponding to the (N-1)-th data processing task corresponding to the incremental data, resulting in M ​​sets of file databases.

[0132] When the processor executes the program, it also performs the following steps: merging the M groups of file databases corresponding to N tasks according to the partition order, and determining the reconstructed international freight rate database based on the merged file database, including: determining the partition information of each file database in the M groups of file databases based on the current storage space, wherein the partition information includes at least: partition size and partition order; merging the M groups of file databases in different partitions according to the partition order and the business primary key to obtain the merged file database; updating the source index information of the M groups of file databases before merging according to the partition size, partition order and business primary key to obtain the target index information; and determining the reconstructed international freight rate database based on the target index information and the merged file database.

[0133] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0134] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following steps: performing data parsing operations on data in N tasks in parallel through N processes to generate structured data for each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in the international freight rate database to be reconstructed, and N is a positive integer; grouping the structured data of each of the N tasks based on the business primary key to obtain M groups corresponding to the N tasks, wherein M is a positive integer; performing data reconstruction operations on the data in the M groups corresponding to the N tasks in parallel through M sub-processes to generate M groups of file databases corresponding to the N tasks; merging the M groups of file databases corresponding to the N tasks according to the partitioning order, and determining the reconstructed international freight rate database based on the merged file database.

[0135] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: Before performing data parsing operations on the data in N tasks in parallel through N processes, the above method further includes: reading the file directory corresponding to the international freight rate database to be reconstructed, obtaining full data and incremental data from the file directory; determining the data processing task corresponding to the full data based on the full data; determining the modification time of each file contained in the incremental data, determining N-1 data processing tasks corresponding to the incremental data for each file contained in the incremental data according to the modification time; and determining N tasks based on the data processing tasks corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0136] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: performing data parsing operations on the data in N tasks in parallel through N processes to generate structured data for each of the N tasks, including: creating a first process for the data processing task corresponding to the full data in the N tasks; using multiple threads in the first process to decompress multiple compressed files in the data processing task corresponding to the full data in parallel to obtain multiple readable data; and using multiple threads in parallel to parse the multiple readable data into structured data based on a predefined data structure to generate the structured data for the data processing task corresponding to the full data.

[0137] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: Data parsing operations are performed on the data in N tasks in parallel by N processes to generate structured data for each of the N tasks, including: creating N-1 processes for the N-1 data processing tasks corresponding to the incremental data in the N tasks; using the N-1 processes in parallel to decompress and parse the compressed files corresponding to the N-1 data processing tasks corresponding to the incremental data, and after the decompression and parsing of the compressed file corresponding to the Xth data processing task corresponding to the incremental data is completed, decompressing and parsing the compressed file corresponding to the X+2th data processing task corresponding to the incremental data is performed, where X is a positive integer greater than 0 and less than N-2; after the decompression and parsing of the compressed file corresponding to the N-3th data processing task corresponding to the incremental data is completed, decompressing and parsing the compressed files corresponding to the N-2th and N-1th data processing tasks corresponding to the incremental data is performed to obtain the structured data for the N-1 data processing tasks corresponding to the incremental data.

[0138] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: grouping the structured data of each of the N tasks based on the business primary key to obtain M groups corresponding to the N tasks, including: calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in the N tasks; grouping the structured data of the data processing task corresponding to the full data according to the hash value of each data record in the full data to obtain M first groups; calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; if the hash value of the business primary key of any target group in the M first groups is the same as the hash value of any target data record in the structured data of the data processing task corresponding to the incremental data, the target data in the incremental data is assigned to the target group in the M first groups to obtain M groups corresponding to the N tasks.

[0139] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: Data reconstruction operations are performed in parallel on data in M ​​groups corresponding to N tasks using M subprocesses to generate a file database of M groups corresponding to N tasks. This includes: creating M subprocesses for the data in the M groups; using the M subprocesses to perform data processing and index update operations on the structured data of the data processing tasks corresponding to the full data in the N tasks; wherein the data processing operations are used to reconstruct the international freight rate database; after detecting the completion of the data processing and index update operations on the full data, using the M subprocesses in parallel to process N-1 data groups corresponding to the incremental data in the N tasks. The task performs data processing and index update operations on the structured data. After completing the data processing and index update operations on the structured data corresponding to the Y-th data processing task for incremental data, the task performs data processing and index update operations on the (Y+1)-th data processing task for incremental data, where Y is a positive integer greater than 0 and less than N-1. After completing the data processing and index update operations on the structured data corresponding to the (N-2)-th data processing task for incremental data, the task performs data processing and index update operations on the structured data corresponding to the (N-1)-th data processing task for incremental data, resulting in M ​​sets of file databases.

[0140] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: merging M groups of file databases corresponding to N tasks according to partition order, and determining the reconstructed international freight rate database based on the merged file database, including: determining the partition information of each file database in the M groups of file databases based on the current storage space, wherein the partition information includes at least: partition size and partition order; merging the M groups of file databases in different partitions according to the partition order and based on the business primary key to obtain the merged file database; updating the source index information of the M groups of file databases before merging based on the partition size, partition order, and business primary key to obtain the target index information; and determining the reconstructed international freight rate database based on the target index information and the merged file database.

[0141] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0145] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0146] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for reconstructing an international freight rate database, characterized in that, include: Data parsing operations are performed on the data in N tasks in parallel by N processes to generate structured data for each of the N tasks. The N tasks include data processing tasks corresponding to the full data and incremental data in the international freight rate database to be reconstructed, where N is a positive integer. The process includes: creating a first process for the data processing task corresponding to the full data; using multiple threads in the first process to decompress multiple compressed files in the data processing task corresponding to the full data in parallel, obtaining multiple readable data; using the multiple threads in parallel to parse the multiple readable data into structured data based on a predefined data structure, generating structured data for the data processing task corresponding to the full data; creating N-1 processes for the N-1 data processing tasks corresponding to the incremental data in parallel; and using the N-1 processes to parse the data corresponding to the incremental data in parallel. The compressed files corresponding to the N-1 data processing tasks are decompressed and parsed. After the compressed file corresponding to the X-th data processing task of the incremental data is decompressed and parsed, the compressed file corresponding to the (X+2)-th data processing task of the incremental data is decompressed and parsed, where X is a positive integer greater than 0 and less than N-2. After the compressed file corresponding to the N-3-th data processing task of the incremental data is decompressed and parsed, the compressed files corresponding to the N-2-th and N-1-th data processing tasks of the incremental data are decompressed and parsed to obtain the structured data of the N-1 data processing tasks of the incremental data. The data parsing process strictly follows the data processing order to ensure that after the full data processing is completed, the incremental data is processed one by one according to the release time order to maintain the integrity and accuracy of the data. The structured data of each of the N tasks is grouped based on the hash value of the business primary key to obtain M groups corresponding to the N tasks, where M is a positive integer; The M subprocesses perform data reconstruction operations on the data in the M groups corresponding to the N tasks in parallel to generate the M group file databases corresponding to the N tasks; The M groups of file databases corresponding to the N tasks are merged according to the partition order, and the reconstructed international freight rate database is determined based on the merged file database.

2. The method according to claim 1, characterized in that, Before performing data parsing operations on the data from N tasks in parallel through N processes, the method further includes: Read the file directory corresponding to the international freight rate database to be reconstructed, and obtain the full data and the incremental data from the file directory; The data processing task corresponding to the full data is determined based on the full data. Determine the modification time of each file contained in the incremental data, and determine each file contained in the incremental data as N-1 data processing tasks corresponding to the incremental data according to the modification time; The N tasks are determined based on the data processing tasks corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

3. The method according to claim 1, characterized in that, Based on the hash value of the business primary key, the structured data of each of the N tasks is grouped to obtain M groups corresponding to the N tasks, including: Calculate the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in the N tasks; Based on the hash value corresponding to each data record in the full data, the structured data of the data processing task corresponding to the full data is grouped to obtain M first groups; Calculate the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; If the hash value of the business primary key corresponding to any target group in the M first groups is the same as the hash value of any target data record in the structured data of the data processing task corresponding to the incremental data, the target data in the incremental data is divided into the target groups in the M first groups to obtain the M groups corresponding to the N tasks.

4. The method according to claim 1, characterized in that, Data reconstruction operations are performed in parallel by M subprocesses on the data in the M groups corresponding to the N tasks to generate a file database of M groups corresponding to the N tasks, including: For the data in the M groups, M sub-processes are created. The M sub-processes are used to perform data processing operations and index update operations on the structured data of the data processing tasks corresponding to the full data in the N tasks. The data processing operations are used to rebuild the international freight rate database. After the data processing and index update operations on the full data are completed, the M subprocesses are used in parallel to perform data processing and index update operations on the structured data of the N-1 data processing tasks corresponding to the incremental data in the N tasks. After the data processing and index update operations on the structured data corresponding to the Y-th data processing task corresponding to the incremental data are completed, the data processing and index update operations on the structured data corresponding to the Y+1-th data processing task corresponding to the incremental data are performed, where Y is a positive integer greater than 0 and less than N-1. After performing data processing and index update operations on the structured data corresponding to the (N-2)th data processing task corresponding to the incremental data, perform data processing and index update operations on the structured data corresponding to the (N-1)th data processing task corresponding to the incremental data to obtain the M sets of file databases.

5. The method according to claim 1, characterized in that, The M groups of file databases corresponding to the N tasks are merged according to the partition order. Based on the merged file database, the reconstructed international freight rate database is determined, including: Based on the current storage space, determine the partition information of the partition where each file database in the M group of file databases is located, wherein the partition information includes at least: partition size and partition order; According to the partitioning order, the M groups of file databases in different partitions are merged based on the business primary key to obtain the merged file database; The source index information of the M groups of file databases before merging is updated based on the partition size, the partition order, and the business primary key to obtain the target index information; The reconstructed international freight rate database is determined based on the target index information and the merged file database.

6. A device for reconstructing an international freight rate database, characterized in that, include: The parsing unit is used to perform data parsing operations on the data in N tasks in parallel through N processes, and generate structured data for each of the N tasks. The N tasks include data processing tasks corresponding to the full data and incremental data in the international freight rate database to be reconstructed, and N is a positive integer. A grouping unit is used to group the structured data of each of the N tasks based on the hash value of the business primary key, so as to obtain M groups corresponding to the N tasks, where M is a positive integer; The reconstruction unit is used to perform data reconstruction operations on the data in the M groups corresponding to the N tasks in parallel through M sub-processes, and generate the M group file databases corresponding to the N tasks; The merging unit is used to merge the M groups of file databases corresponding to the N tasks according to the partitioning order, and to determine the reconstructed international freight rate database based on the merged file database. The parsing unit includes: a first creation subunit, used to create a first process for the data processing task corresponding to the full data in the N tasks; a decompression subunit, used to decompress multiple compressed files in the data processing task corresponding to the full data in parallel using multiple threads in the first process to obtain multiple readable data; and a first parsing subunit, used to parse the multiple readable data into structured data in parallel using the multiple threads based on a predefined data structure to generate the structured data of the data processing task corresponding to the full data. The parsing unit includes: a second creation subunit, used to create N-1 processes for the N-1 data processing tasks corresponding to the incremental data in the N tasks; a second parsing subunit, used to decompress and parse the compressed files corresponding to the N-1 data processing tasks corresponding to the incremental data in parallel using the N-1 processes, and after the decompression and parsing of the compressed file corresponding to the Xth data processing task corresponding to the incremental data is completed, to decompress and parse the compressed file corresponding to the X+2th data processing task corresponding to the incremental data, where X is a positive integer greater than 0 and less than N-2; and a third parsing subunit, used to decompress and parse the compressed files corresponding to the N-3th data processing task corresponding to the incremental data and the compressed files corresponding to the N-2th and N-1th data processing tasks corresponding to the incremental data after the decompression and parsing of the compressed file corresponding to the incremental data is completed, to obtain the structured data of the N-1 data processing tasks corresponding to the incremental data. The data parsing process strictly follows the data processing order to ensure that incremental data is processed one by one in the order of publication time after the full data is processed, so as to maintain the integrity and accuracy of the data.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes stored computer instructions, wherein, when the computer instructions are executed by a processor, the method for reconstructing the international freight rate database as described in any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method for reconstructing the international freight rate database as described in any one of claims 1 to 5.

Images