Reconstruction method and device of international freight rate database and electronic equipment

Abstract

The invention discloses a reconstruction method and device for an international freight rate database and electronic equipment, the method is applied to the field of data reconstruction, and the method comprises the following steps: carrying out data analysis operation on data in N tasks in parallel through N processes, and generating structured data of each task in the N tasks; grouping the structured data of each task based on the service primary key to obtain M groups corresponding to the N tasks; performing data reconstruction operation on the data in the M groups corresponding to the N tasks in parallel through the M sub-processes to generate M groups of file databases corresponding to the N tasks; and merging the M groups of file databases corresponding to the N tasks according to the partition sequence, and determining a reconstructed international freight rate database according to the merged file database. Through the method and the device, the problem of relatively low database reconstruction efficiency caused by large data volume and need to process data according to a time sequence in the process of reconstructing the international freight rate database by an airline company in related technologies is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data reconstruction, and in particular, to a method and device for reconstructing an international airfare database and an electronic device. BACKGROUND

[0002] The traditional method of constructing and maintaining an international civil aviation airfare database mainly relies on a serial processing mechanism, that is, raw data is loaded into a data processing server in batches, and then parsed and stored in a file database specially used for international civil aviation airfare calculation. This processing procedure follows the time sequence of data release, ensuring the accuracy and timeliness of the data. However, as the data size continues to expand, especially when dealing with airfares and rules that have been in effect for two years, the reconstruction of the database becomes extremely time-consuming, usually taking about a week to complete. The processing of nearly 20,000 incremental update packages includes steps such as decompression, parsing, editing, and updating the index. The long reconstruction period under this serial processing mode not only affects the response speed of the system, but also limits the efficiency of data recovery and off-site reconstruction, becoming a key problem that needs to be solved in the industry.

[0003] Existing solutions, although they attempt to introduce parallel processing in some aspects, such as the data decompression process, lack effective data grouping and editing strategies, especially the inability to balance the processing load of different data types, resulting in limited overall performance improvement. The processing time of some large data types (such as intallfare) is too long, seriously slowing down the overall process.

[0004] In view of the problem in the related art that the large amount of data and the necessity of processing the data in time sequence during the reconstruction of the international airfare database by the airline company results in low efficiency of the database reconstruction, no effective solution has been proposed so far. SUMMARY

[0005] The main purpose of the present application is to provide a method and device for reconstructing an international airfare database and an electronic device, to solve the problem in the related art that the large amount of data and the necessity of processing the data in time sequence during the reconstruction of the international airfare database by the airline company results in low efficiency of the database reconstruction.

[0006] In order to achieve the above object, according to one aspect of the present application, a method for reconstructing an international freight database is provided, the method comprising: performing data parsing operations on data in N tasks in parallel through N processes to generate structured data of each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in the international freight database to be reconstructed, N is a positive integer; grouping the structured data of each of the N tasks based on a business primary key to obtain M groups corresponding to the N tasks, wherein M is a positive integer; performing data reconstruction operations on data in the M groups corresponding to the N tasks in parallel through M sub-processes to generate M file databases corresponding to the N tasks; merging the M file databases corresponding to the N tasks according to a partition order to determine the reconstructed international freight database according to the merged file databases.

[0007] Further, before performing the data parsing operations on the data in the N tasks in parallel through the N processes, the method further comprises: reading a file directory corresponding to the international freight database to be reconstructed, and obtaining the full data and the incremental data from the file directory; determining a data processing task corresponding to the full data according to the full data; determining modification times of each file included in the incremental data, and determining N-1 data processing tasks corresponding to the incremental data according to the modification times of each file included in the incremental data; determining the N tasks according to the data processing task corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0008] Further, the performing the data parsing operations on the data in the N tasks in parallel through the N processes to generate the structured data of each of the N tasks comprises: for the data processing task corresponding to the full data in the N tasks, creating L threads, wherein L is a positive integer; decompressing L compressed files in the data processing task corresponding to the full data in parallel using the L threads to obtain L readable data; and parsing the L readable data into structured data based on a predefined data structure using the L threads to generate the structured data of the data processing task corresponding to the full data.

[0009] Further, the data parsing operation is performed on the data in the N tasks in parallel by N processes to generate structured data of each of the N tasks, including: creating N-1 processes for the N-1 data processing tasks corresponding to the incremental data in the N tasks; performing decompression and parsing on the compressed files corresponding to the N-1 data processing tasks of the incremental data in parallel by the N-1 processes, and after the decompression and parsing of the compressed file corresponding to the Xth data processing task of the incremental data are completed, the decompression and parsing of the compressed file corresponding to the X+2th data processing task of the incremental data are performed, where X is a positive integer greater than 0 and less than N-2; after the decompression and parsing of the compressed file corresponding to the N-3th data processing task of the incremental data are completed, the decompression and parsing of the compressed file corresponding to the N-2th data processing task and the compressed file corresponding to the N-1th data processing task of the incremental data are performed to obtain the structured data of the N-1 data processing tasks of the incremental data.

[0010] Further, the structured data of each of the N tasks is grouped based on the business primary key to obtain M groups corresponding to the N tasks, including: calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in the N tasks; grouping the structured data of the data processing task corresponding to the full data according to the hash value corresponding to each data record in the full data to obtain M first groups; calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; in the case that the hash value of the business primary key corresponding to any target group in the M first groups is the same as the hash value corresponding to any target data record in the structured data of the data processing task corresponding to the incremental data, the target data in the incremental data is divided into the target group in the M first groups to obtain the M groups corresponding to the N tasks.

[0011] Further, the data reconstruction operation is performed on the data in the M groups corresponding to the N tasks in parallel by M sub-processes to generate M groups of file databases corresponding to the N tasks, including: creating M sub-processes for the data in the M groups, and performing data processing operation and index updating operation on the structured data of the data processing tasks corresponding to the full data in the N tasks by using the M sub-processes, wherein the data processing operation is used to reconstruct the international freight rate database; after detecting that the data processing operation and the index updating operation on the full data are completed, performing data processing operation and index updating operation on the structured data of N-1 data processing tasks corresponding to the incremental data in the N tasks in parallel by using the M sub-processes, and after the data processing operation and the index updating operation on the structured data of the Yth data processing task corresponding to the incremental data are completed, performing data processing operation and index updating operation on the structured data of the Y+1th data processing task corresponding to the incremental data, wherein Y is a positive integer greater than 0 and less than N-1; after the data processing operation and the index updating operation on the structured data of the N-2th data processing task corresponding to the incremental data are completed, performing data processing operation and index updating operation on the structured data of the N-1th data processing task corresponding to the incremental data, to obtain the M groups of file databases.

[0012] Further, the M groups of file databases corresponding to the N tasks are merged according to the partition order, and the reconstructed international freight rate database is determined according to the merged file database, including: determining the partition information of each file database in the M groups of file databases in the current storage space, wherein the partition information at least includes: partition size, the partition order; merging the M groups of file databases in different partitions according to the business primary key according to the partition order, to obtain the merged file database; updating the source index information of the M groups of file databases before merging according to the partition size, the partition order and the business primary key, to obtain target index information; determining the reconstructed international freight rate database according to the target index information and the merged file database.

[0013] To achieve the above object, according to another aspect of the present application, there is provided a reconstruction device of an international freight rate database, comprising: an analysis unit configured to perform a data analysis operation on data in N tasks in parallel through N processes to generate structured data of each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in the international freight rate database to be reconstructed, and N is a positive integer; a grouping unit configured to group the structured data of each of the N tasks based on a business primary key to obtain M groups corresponding to the N tasks, wherein M is a positive integer; a reconstruction unit configured to perform a data reconstruction operation on data in the M groups corresponding to the N tasks in parallel through M sub-processes to generate M file databases corresponding to the N tasks; and a merging unit configured to merge the M file databases corresponding to the N tasks in a partition order to determine a reconstructed international freight rate database according to the merged file databases.

[0014] Further, the device further comprises: an acquisition unit configured to, before performing the data analysis operation on data in the N tasks in parallel through the N processes, read a file directory corresponding to the international freight rate database to be reconstructed, and acquire the full data and the incremental data from the file directory; a first determination unit configured to determine data processing tasks corresponding to the full data according to the full data; a second determination unit configured to determine modification times of each file included in the incremental data, and determine N-1 data processing tasks corresponding to the incremental data according to the modification times of each file included in the incremental data; and a third determination unit configured to determine the N tasks according to the data processing tasks corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0015] Further, the analysis unit comprises: a first creation sub-unit configured to create a first process for the data processing tasks corresponding to the full data in the N tasks; a decompression sub-unit configured to perform decompression on a plurality of compressed files in the data processing tasks corresponding to the full data in parallel through a plurality of threads in the first process to obtain a plurality of readable data; and a first analysis sub-unit configured to analyze the plurality of readable data into structured data in parallel through the plurality of threads based on a predefined data structure to generate structured data of the data processing tasks corresponding to the full data.

[0016] Further, the parsing unit comprises: a second creating sub-unit, configured to create N-1 processes for the N-1 data processing tasks corresponding to the incremental data in the N tasks; a second parsing sub-unit, configured to perform decompression and parsing on the compressed files corresponding to the N-1 data processing tasks corresponding to the incremental data in parallel using the N-1 processes, and after the decompression and parsing on the compressed file corresponding to the Xth data processing task corresponding to the incremental data are completed, perform decompression and parsing on the compressed file corresponding to the X+2th data processing task corresponding to the incremental data, wherein X is a positive integer greater than 0 and smaller than N-2; and a third parsing sub-unit, configured to perform decompression and parsing on the compressed file corresponding to the N-3th data processing task corresponding to the incremental data after the execution of the decompression and parsing on the compressed file is completed, and perform decompression and parsing on the compressed file corresponding to the N-2th data processing task and the compressed file corresponding to the N-1th data processing task, to obtain the structured data of the N-1 data processing tasks corresponding to the incremental data.

[0017] Further, the grouping unit comprises: a first calculating sub-unit, configured to calculate the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in the N tasks; a grouping sub-unit, configured to group the structured data of the data processing task corresponding to the full data according to the hash value corresponding to each data record in the full data, to obtain M first groups; a second calculating sub-unit, configured to calculate the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; and a dividing sub-unit, configured to, in the case that the hash value of the business primary key corresponding to any target group in the M first groups is the same as the hash value corresponding to any target data record in the structured data of the data processing task corresponding to the incremental data, divide the target data in the incremental data into the target group in the M first groups, to obtain the M groups corresponding to the N tasks.

[0018] Further, the reconstruction unit comprises: a third creation sub-unit, configured to create M sub-processes for data in the M groups, and perform data processing operation and index update operation on structured data of data processing tasks corresponding to the full-amount data in the N tasks by using the M sub-processes, wherein the data processing operation is used for reconstructing the international freight rate database; a first processing sub-unit, configured to perform data processing operation and index update operation on structured data of N-1 data processing tasks corresponding to the incremental data in the N tasks by using the M sub-processes in parallel after detecting that the data processing operation and the index update operation on the full-amount data are performed, and perform data processing operation and index update operation on structured data of a Y+1th data processing task corresponding to the incremental data after performing data processing operation and index update operation on structured data of a Yth data processing task corresponding to the incremental data, wherein Y is a positive integer greater than 0 and less than N-1; and a second processing sub-unit, configured to perform data processing operation and index update operation on structured data of an N-1th data processing task corresponding to the incremental data after performing data processing operation and index update operation on structured data of an N-2th data processing task corresponding to the incremental data, to obtain the M groups of file databases.

[0019] Further, the merging unit comprises: a first determination sub-unit, configured to determine partition information of a partition where each file database in the M groups of file databases is located according to a current storage space, wherein the partition information at least comprises partition size and partition order; a merging sub-unit, configured to merge the M groups of file databases in different partitions according to the business primary key in the partition order, to obtain the merged file database; an update sub-unit, configured to update source index information of the M groups of file databases before merging according to the partition size, the partition order and the business primary key, to obtain target index information; and a second determination sub-unit, configured to determine the reconstructed international freight rate database according to the target index information and the merged file database.

[0020] In order to achieve the above-mentioned purpose, according to an aspect of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the international freight rate database reconstruction method of any one of the above-mentioned aspects, and the computer program, when executed by the processor, implements the steps of the international freight rate database reconstruction method in the embodiments of the present application.

[0021] In order to achieve the above object, according to one aspect of the present application, a computer readable storage medium is provided, which comprises stored computer instructions, wherein the computer instructions, when executed by a processor, implement any one of the above-mentioned international freight rate database reconstruction methods.

[0022] In order to achieve the above object, according to one aspect of the present application, an electronic device is provided, comprising one or more processors and a memory, the memory being configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement any one of the above-mentioned international freight rate database reconstruction methods.

[0023] By the present application, the following steps are adopted: performing data parsing operations on data in N tasks in parallel by N processes to generate structured data of each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in an international freight rate database to be reconstructed, and N is a positive integer; grouping the structured data of each of the N tasks based on a business primary key to obtain M groups corresponding to the N tasks, wherein M is a positive integer; performing data reconstruction operations on data in the M groups corresponding to the N tasks in parallel by M sub-processes to generate M sets of file databases corresponding to the N tasks; and merging the M sets of file databases corresponding to the N tasks in accordance with a partition order to determine a reconstructed international freight rate database according to the merged file databases, thereby solving the problem in the related art that the database reconstruction efficiency is low due to large data volume and the necessity of processing data in time sequence during the reconstruction of the international freight rate database by the airline company.

[0024] By implementing the concurrent data reconstruction and partition merging strategy, all data processing tasks can be synchronously promoted, the generation efficiency of structured data is significantly improved, the reconstruction time of the international freight rate database can be significantly shortened, and the technical effect of accelerating the data processing flow is achieved. At the same time, through the multi-process, intelligent partition merging of each task can be realized, so that the data integration process is efficient and accurate, the rapid recovery of international freight rate data and the consistent reconstruction of cross-regional databases are achieved, the low-efficiency bottleneck of the traditional serial processing mode is overcome, especially when processing large-scale data sets, the multi-core processing capability of the server can be fully utilized, the waste of system resources is avoided, the data integrity and the ordered storage according to the release time are ensured, and the flexibility of data storage space maintenance and the reliability of data service are further improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and the illustrative embodiments of the present application and their description serve the purpose of explaining the present application. In the drawings:

[0026] Figure 1 is a flowchart of a method for reconstructing an international freight database according to Embodiment One of the present application;

[0027] Figure 2 is a schematic diagram of optional data division tasks to be reconstructed according to Embodiment One of the present application;

[0028] Figure 3 is a flowchart of a method for reconstructing an international freight database according to Embodiment One of the present application;

[0029] Figure 4 is a flowchart of a method for reconstructing an international freight database according to Embodiment One of the present application;

[0030] Figure 5 is a flowchart of a method for reconstructing an international freight database according to Embodiment One of the present application;

[0031] Figure 6 is a flowchart of a method for reconstructing an international freight database according to Embodiment One of the present application;

[0032] Figure 7 is a schematic diagram of a device for reconstructing an international freight database according to Embodiment Two of the present application;

[0033] Figure 8 is a schematic diagram of an electronic device for reconstructing an international freight database according to Embodiment Five of the present application. DETAILED DESCRIPTION

[0034] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0035] It should be noted that the user information (including but not limited to user equipment information, user personal information, collected data, used data, generated data, processed data, etc.) and data (including but not limited to data for analysis, stored data, displayed data, collected information, used information, generated information, processed information, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards of relevant countries and regions, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal. For example, interfaces are provided between the system and related users or institutions, and before obtaining related information, the interface needs to send an acquisition request to the aforementioned user or institution, and after receiving the consent information feedback from the aforementioned user or institution, the related information is acquired.

[0036] It should be noted that the present application provides a corresponding operation portal for the user to choose to agree or refuse the automatic decision result; if the user chooses to refuse, the expert decision process is entered.

[0037] In order to enable personnel in the art to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0038] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0039] Embodiment one

[0040] The present application will be described below in conjunction with preferred implementation steps, Figure 1 is a flowchart of the method for reconstructing the international freight database according to the embodiment one of the present application, as Figure 1 shown, the method comprises the following steps:

[0041] In step S101, data parsing operations are performed in parallel on data in N tasks by N processes to generate structured data for each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in the international airfare database to be reconstructed, and N is a positive integer.

[0042] In the international civil aviation transportation industry, airfare data is a crucial component. Airfare data is usually published in full and incremental forms on a regular basis, wherein the full data (all adds files) contains all airfare information at a specific time point, and the incremental data (update files) reflects airfare changes since the last publication.

[0043] In this embodiment, to improve the efficiency of database reconstruction, multiple processes can be used to process full data and incremental data in parallel. For example, N processes (N is a positive integer) are started to correspond to N independent data processing tasks, including full data processing and multiple incremental data processing of the international airfare database. Each process is responsible for a task, performs data parsing operations, decompresses and parses the original compressed file into structured data records, and generates structured data for each task.

[0044] It should be noted that this process strictly follows the data processing order to ensure that the full data processing is completed, and the incremental data is processed one by one in the order of publication time to maintain the integrity and accuracy of the data.

[0045] In step S102, the structured data of each of the N tasks is grouped based on the business primary key to obtain M groups corresponding to the N tasks, wherein M is a positive integer.

[0046] In this embodiment, for the structured data of each data processing task, a grouping strategy based on the business primary key is implemented. This grouping strategy involves processing the business primary key in each data record, and then grouping the structured data in each task into M groups, where M is a positive integer representing the number of groups. The grouped data ensures that records with the same business primary key are concentrated in the same group, facilitating subsequent editing operations.

[0047] It should be noted that this grouping method not only depends on the data type, but also considers the differences in the amount of data, optimizing the processing speed and ensuring that all processes can complete the task at a similar rate, thereby improving the overall processing efficiency. The M groups generated by each task are independent of other tasks, providing conditions for parallel execution of editing and updating indexes. Through the above data grouping steps, uniform distribution of data records is achieved, avoiding resource imbalance and processing delays caused by a single process processing too much data.

[0048] Step S103, performing data reconstruction operation on the data in the M groups corresponding to the N tasks in parallel through M sub-processes to generate M groups of file databases corresponding to the N tasks.

[0049] In the first embodiment, after data grouping, M sub-processes are started to perform data reconstruction operation in parallel for the M groups corresponding to the N tasks. Each sub-process focuses on the assigned group and independently performs data editing and index updating, ensuring that the processing process is non-blocking and efficient.

[0050] Through parallel execution, the sub-process can quickly complete the processing of the grouped data and generate the target data, i.e., the reconstructed data. This method fully utilizes the capabilities of multi-core processors, reduces processing time, maintains the order of data processing, and avoids database consistency problems caused by improper concurrent operations. Finally, after the data reconstruction operation of the N tasks is completed, M groups of file databases are generated.

[0051] Step S104, merging the M groups of file databases corresponding to the N tasks according to the partition order to determine the reconstructed international freight rate database according to the merged file database.

[0052] In the first embodiment, the data merging step is performed according to the partition order preset by the current storage space. This process involves integrating the M groups of file databases corresponding to the N tasks. Finally, the target data after merging constitutes the reconstructed international freight rate database, maintaining the integrity of the data and the correct ordering according to the data release time, achieving efficient reconstruction of the database, and thus meeting the needs of rapid recovery and off-site reconstruction of international freight rate data.

[0053] It should be noted that after completing the data reconstruction operation of the N tasks, M groups of file databases are generated, and the data of each file database is independent, and there is no duplicate data between any two file databases. Therefore, only file merging is needed to complete the data record merging, without traversing each data, thus achieving the effect of improving the speed of database creation. Further, in order to fully utilize the multi-core hardware capabilities of the server, the data merging process can be performed in parallel, and the granularity of concurrency is an independent processing unit for each data type.

[0054] To sum up, the international freight rate database reconstruction method provided by the embodiment one of the application, through N processes, data in N tasks are operated in parallel for data analysis, structured data of each task in N tasks is generated, wherein, N tasks include data processing tasks corresponding to full data and incremental data in the international freight rate database to be reconstructed, N is a positive integer; based on the business primary key, the structured data of each task in N tasks is grouped to obtain M groups corresponding to N tasks, wherein M is a positive integer; through M sub-processes, data in M groups corresponding to N tasks is operated in parallel for data reconstruction to generate M file databases corresponding to N tasks; according to the partition order, M file databases corresponding to N tasks are merged, and the international freight rate database after reconstruction is determined according to the merged file database, solving the problem that in the related art, due to large data volume and the necessity of processing data in time sequence, the database reconstruction efficiency is low during the reconstruction of the international freight rate database by the airline company.

[0055] By implementing the concurrent data reconstruction and partition merging strategy, all data processing tasks can be synchronized and advanced, the generation efficiency of structured data is significantly improved, the reconstruction time of the international freight rate database can be significantly shortened, and the technical effect of accelerating the data processing process is achieved. At the same time, the intelligent partition merging of each task realized by the thread pool technology makes the data integration process efficient and accurate, realizes the rapid recovery of international freight rate data and the consistent reconstruction of cross-regional databases, overcomes the low efficiency bottleneck of the traditional serial processing mode, especially when processing large-scale data sets, the multi-core processing capability of the server can be fully utilized, the waste of system resources is avoided, the integrity of the data and the ordered storage according to the release time are ensured, and the flexibility of data storage space maintenance and the reliability of data service are further improved.

[0056] Optionally, in the international freight rate database reconstruction method provided by the embodiment one of the application, before the data in N tasks is operated in parallel for data analysis by N processes, the above method further includes: reading a file directory corresponding to the international freight rate database to be reconstructed, obtaining full data and incremental data from the file directory; determining a data processing task corresponding to the full data according to the full data; determining the modification time of each file included in the incremental data, and determining N-1 data processing tasks corresponding to the incremental data for each file included in the incremental data according to the modification time; determining N tasks according to the data processing task corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0057] In the embodiment one, in order to reconstruct the international freight rate database, it is necessary to scan the file directory related to the reconstruction of the international freight rate database in the storage area, accurately identify the full data file and the incremental data file in the directory, and ensure the complete acquisition of all data to be reconstructed.

[0058] Then, based on the information of the full data, a first data processing task is generated as a starting point of the database reconstruction.

[0059] Secondly, by analyzing the last modification time stamp of each incremental data file, the incremental files are associated to N-1 subsequent data processing tasks in chronological order, which ensures the continuity and timing of data updates, where N and N-1 are positive integers. For example, all incremental data is divided into multiple Jobs by a certain number of values (e.g., 1000 hours of incremental data is divided into one Job).

[0060] Finally, the first full data processing task and N-1 incremental data processing tasks are integrated to form N ordered and independently executable data processing jobs. For example, Figure 2 is a schematic diagram of dividing tasks of data to be reconstructed according to an embodiment of the present application. Figure 2 Job0 in is the data processing task corresponding to the full data described above. Figure 2 Job1 to JobN in is the N-1 data processing tasks corresponding to the incremental data described above.

[0061] Through the above steps, the data processing task flow is accurately planned, the systematicness and efficiency of data reconstruction work are guaranteed, and a solid foundation is provided for subsequent parallel processing.

[0062] Optionally, in the method for reconstructing the international freight database provided in the first embodiment of the present application, the data in the N tasks is subjected to data parsing operations in parallel by N processes to generate structured data for each of the N tasks, including: creating a first process for the data processing task corresponding to the full data in the N tasks; using multiple threads in the first process to decompress multiple compressed files in the data processing task corresponding to the full data in parallel to obtain multiple readable data; and using multiple threads to parse the multiple readable data into structured data based on a predefined data structure to generate structured data for the data processing task corresponding to the full data.

[0063] In the first embodiment, according to the characteristics of the full data processing task, multi-threading technology can be used to determine and create L threads based on the first process, that is, the above-mentioned multiple threads, wherein the value of L is based on the processing capacity of the server to ensure reasonable allocation and utilization of computing resources. Then, multiple threads are started simultaneously, each responsible for decompression of a compressed file. This parallel processing significantly speeds up the decompression process of full data, quickly generating multiple readable data files, and preparing data sources for subsequent parsing work. Secondly, multiple threads are used to simultaneously parse these readable data files, converting unstructured text information into predefined data structures, that is, the data structure of the reconstructed international freight rate data. Finally, through multiple thread parallel jobs, the structured data sets generated by multiple threads are integrated to form the complete data required by the full data processing task, providing a basis for subsequent data editing and index updating.

[0064] Through the above steps, the technical effect of accelerating full data decompression and parsing through multi-thread parallel processing is achieved, which not only shortens the data preparation time, but also ensures the accuracy of the data and the efficiency of the processing, laying a solid data foundation for the subsequent database reconstruction process. This strategy fully utilizes the capabilities of modern multi-core processors, effectively balances processing speed and resource consumption, and is particularly advantageous when dealing with large-scale data sets.

[0065] Figure 3 is an optional flowchart of processing data processing tasks corresponding to full data according to the first embodiment of the present application. As shown in Figure 3 After the main process (i.e. the first process mentioned above) is started, the directory structure of all full data and incremental data is read, sorted by publication time, and then divided into Jobs. Since the amount of full data is large, the number of compressed files can reach dozens, so the full data can be divided into a Job0. A thread pool is created, and each thread is responsible for decompressing a file and performing text file parsing. After all parsing is complete, a large amount of structured data is generated.

[0066] Optionally, in the method for reconstructing the international freight database provided in Embodiment One of the present application, the data in the N tasks is subjected to data parsing operation in parallel through N processes to generate the structured data of each of the N tasks, including: creating N-1 processes for the N-1 data processing tasks corresponding to the incremental data; decompressing and parsing the compressed files corresponding to the N-1 data processing tasks corresponding to the incremental data in parallel through the N-1 processes, and after the decompression and parsing of the compressed file corresponding to the Xth data processing task corresponding to the incremental data are completed, decompressing and parsing the compressed file corresponding to the X+2th data processing task corresponding to the incremental data, wherein X is a positive integer greater than 0 and less than N-2; after the decompression and parsing of the compressed file corresponding to the N-3th data processing task corresponding to the incremental data are completed, decompressing and parsing the compressed file corresponding to the N-2th data processing task corresponding to the incremental data and the compressed file corresponding to the N-1th data processing task to obtain the structured data of the N-1 data processing tasks corresponding to the incremental data.

[0067] In Embodiment One, based on the N-1 tasks involving incremental data processing, a corresponding number of independent processes are created to ensure that each incremental data task has a dedicated processing unit, i.e., the aforementioned N-1 processes, which provide structural support for parallel processing.

[0068] Then, the N-1 threads start parallel processing of the compressed files allocated to them, i.e., performing decompression and parsing operations to generate structured data. It is worth noting that this parallel processing follows a specific dependency relationship, i.e., after the decompression and parsing of the compressed file corresponding to the Xth data processing task are completed, the decompression and parsing process of the X+2th data processing task is started, and X is a positive integer greater than 0 and less than N-2. By controlling the execution order between threads, resource conflicts and data confusion caused by concurrent operations can be avoided, ensuring the accuracy and consistency of data processing. Wherein, N, N-1, N-2, N-3, X and X+2 are positive integers.

[0069] Finally, the N-1th data processing task and the N-2th data processing task in the incremental data can be executed simultaneously. Subsequently, the structured data of all N-1 incremental data processing tasks is generated one by one, and the processing of each task is performed under the premise of following the data publishing time sequence, providing a data basis for the subsequent data grouping and data reconstruction stage.

[0070] In an alternative embodiment, Figure 4 is a flowchart of processing the data processing task corresponding to the incremental data according to the optional embodiment provided in Embodiment One of the present application. As Figure 4As shown, the incremental data processing flow is similar to the full data processing flow, with the difference being that the execution of each Job needs to wait for the processing status of the previous Job. Because, all Jobs are not started at the same time, if all are started, it will consume all CPU and memory resources, causing the system to be unavailable. In addition, in the data reconstruction phase, the business logic requires that the data editing of the previous Job must be completed, otherwise it will cause data errors.

[0071] Specifically, in the data decompression and analysis phase, if the current execution Job Z is observed, the previous Job, i.e. Job Z-2, is completed, if not, it is put to sleep and waits. Therefore, although multiple processes are created at the same time, because the execution of the processes depends on the relationship (such as Figure 4 the start of the Job Z process in the above depends on the end of the execution of the Z-2 process), most of the processes are in sleep state and do not occupy CPU resources. Further explanation, although in the waiting state, the CPU resources are not idle and can process the data reconstruction process of Job Z-1 or Job Z-2 (data processing operation and index updating operation in the following).

[0072] Through the above steps, the technical effect of fully utilizing the multi-thread parallel processing capability on the basis of ensuring the data processing timing is achieved, which significantly improves the processing efficiency of incremental data, and creates favorable conditions for efficient reconstruction of international civil aviation price database. This strategy not only optimizes the processing flow and reduces the waiting time, but also avoids the bottleneck that may occur when processing large-scale data sets, thereby speeding up the update speed of the database, enhancing the recovery capability of international price data and the feasibility of off-site database reconstruction.

[0073] Optionally, in the international price database reconstruction method provided in Embodiment One of the present application, the structured data of each task in the N tasks is grouped based on the business primary key, obtaining M groups corresponding to the N tasks, comprising: calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in the N tasks; grouping the structured data of the data processing task corresponding to the full data according to the hash value corresponding to each data record in the full data, obtaining M first groups; calculating the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; in the case that the hash value of the business primary key corresponding to any target group in the M first groups is the same as the hash value corresponding to any target data record in the structured data of the data processing task corresponding to the incremental data, the target data in the incremental data is divided into the target group in the M first groups, obtaining M groups corresponding to the N tasks.

[0074] In the first embodiment, in order to uniformly group the data, the hash value corresponding to the business primary key of each record in the full data can be read and calculated to ensure that the same business primary key produces consistent hash values. The business primary key can be one or more fields in the data table of the international freight database, which is not specifically limited in the first embodiment.

[0075] Then, according to the generated first hash value, the structured data of the full data is divided to form M first groups, and M is the total number of groups preset. The total number of groups can be determined according to the data types contained in the international freight database, and can also be determined according to the data amount of the international freight database, which is not specifically limited in the first embodiment. It should be noted that due to the dispersion characteristics of the hash value, this grouping process enables the data to be uniformly distributed in each group, and the data amount processed by each process is generally equivalent, and there is no case that a process is delayed for a very long time compared with other processes. This uniform distribution strategy can reduce the overall processing delay of the system.

[0076] The general grouping strategy in the prior art is based on data types, and the same data type is processed in one process, but the disadvantage is that the data amount of each data type is very different, and even the data amount of a certain type (installfare) accounts for more than 30% of the whole, which results in that other processes have been completed, but this process is still not completed, thereby leading to low overall processing efficiency of data reconstruction.

[0077] Finally, for the processing of the incremental data, the second hash value of the business primary key of each record is calculated by applying the hash function, and the structured data in each data processing task in the incremental data is distributed to the above-mentioned M first groups according to these values. The results of grouping the full data and the incremental data are combined to determine the M groups corresponding to the N tasks.

[0078] In an optional embodiment, for the structured data of the P02 data type in a certain task, M is 3, the structured data of the P02 data type is grouped according to the business primary key fields A, field B and field C, the hash value is calculated, and then the total number of groups is taken as the remainder. The calculation result can be shown in Table 1. The calculation result of the hash calculation can represent the partition number of different data, as shown in Table 1, and the three data in Table 1 are split into partition 2, partition 0 and partition 1. Each partition sub-process independently performs data reconstruction.

[0079] Table 1

[0080]

[0081] Through the above steps, the technical effect of efficiently grouping structured data is achieved, which is helpful for subsequent parallel data editing and index updating operations, because the data is reasonably dispersed into multiple groups, each group can be processed independently, reducing resource competition in the processing process, while ensuring the correctness and integrity of data processing. In addition, the hash grouping method based on the business primary key can also dynamically adapt to the processing needs of different types of data, which is beneficial to improve the reconstruction efficiency of high-performance databases.

[0082] Optionally, in the international freight rate database reconstruction method provided in Embodiment One of the present application, the data reconstruction operation is performed on the data in the M groups corresponding to the N tasks in parallel through M sub-processes, to generate M group file databases corresponding to the N tasks, including: for the data in the M groups, creating M sub-processes, and using the M sub-processes to perform data processing operations and index updating operations on the structured data of the data processing tasks corresponding to the full data in the N tasks, wherein the data processing operation is used to reconstruct the international freight rate database; after detecting that the data processing operation and the index updating operation on the full data are executed, using the M sub-processes to perform data processing operations and index updating operations on the structured data of N-1 data processing tasks corresponding to the incremental data in the N tasks in parallel, and after completing the data processing operation and the index updating operation on the structured data corresponding to the Yth data processing task corresponding to the incremental data, performing the data processing operation and the index updating operation on the structured data corresponding to the Y+1th data processing task corresponding to the incremental data, wherein Y is a positive integer greater than 0 and less than N-1; after completing the data processing operation and the index updating operation on the structured data corresponding to the N-2th data processing task corresponding to the incremental data, performing the data processing operation and the index updating operation on the structured data corresponding to the N-1th data processing task corresponding to the incremental data, to obtain M group file databases.

[0083] In Embodiment One, for the above M groups, M independent sub-processes are started, and each sub-process is assigned to perform data processing operations and index updating operations on specific group data. It should be noted that the sub-process of each task (including the data processing task corresponding to the full data and the data processing task corresponding to the incremental data, i.e. the above-mentioned Job) can be created by using the fork method, so that each sub-process can inherit the memory information of the parent process, including the task information of the N tasks (i.e. the information of all Jobs), and the task sequence number (i.e. the Job number) allocated to the current sub-process. Each sub-process inherits the memory data of the parent process, so it can read the structured data processed by the parent process, i.e. each sub-process gets all the decompressed and parsed data without memory copying, globally occupying one physical memory, saving CPU and memory resources.

[0084] Then, each sub-process respectively performs data processing operations on the grouped data assigned to itself, including but not limited to data editing, rule application, and comparison and update with existing database records. Exemplarily, the data processing operations include but are not limited to searching for data with the same primary key in the text database according to the data service primary key, directly adding the record if there is no such record, and deleting or modifying the record based on the record if there is such record. This parallel processing stage greatly improves the speed of data processing and reduces the time consumption of a single process in processing large amounts of data. After the data processing operation is completed, the index update operation is continued using the M sub-processes to establish or update the index position of the data record, ensuring the query performance of the database.

[0085] Specifically, Figure 4 is a flowchart of an optional process of processing data processing tasks corresponding to the incremental data according to the embodiment one of the present application. As Figure 4 shown, in the execution stage of the data processing operation and the index update operation, if the current execution JobZ, the previous Job, i.e. JobZ-1, is observed to determine whether it is completed, and if not, it is put to sleep and waits. Therefore, although multiple processes are created at the same time, the execution of the processes depends on the relationship (e.g. Figure 4 the start of the execution of the JobZ sub-process depends on the end of the execution of the JobZ-1 sub-process).

[0086] Finally, after the data processing operation and the index update operation of the M groups of data, M groups of corresponding file databases are obtained, which are then merged into the reconstructed international civil aviation fare database, completing the reconstruction task of the international civil aviation fare database.

[0087] Through the above steps, the technical effect of accelerating the reconstruction speed of the international civil aviation fare database through multi-process parallel processing and index update is achieved. This strategy not only improves the efficiency of data processing, but also simplifies the formation of the final data set through the merging operation, avoids redundant data processing and storage, and improves the performance of the entire international civil aviation fare database reconstruction process. The parallel sub-process processing and index update operation fully utilizes the computing power of the multi-core processor, providing a more efficient method for database maintenance and update.

[0088] Optionally, in the method for reconstructing the international freight database according to the first embodiment of the present application, the M groups of file databases corresponding to the N tasks are merged according to the partition order, and the reconstructed international freight database is determined according to the merged file databases, including: determining the partition information of each file database in the M groups of file databases in the current storage space, wherein the partition information at least includes the partition size and the partition order; merging the M groups of file databases in different partitions according to the business primary key according to the partition order to obtain the merged file databases; updating the source index information of the M groups of file databases before merging according to the partition size, the partition order and the business primary key to obtain the target index information; and determining the reconstructed international freight database according to the target index information and the merged file databases.

[0089] In the first embodiment, Figure 5 is a flowchart of an optional process for merging file databases corresponding to different data processing tasks according to the first embodiment of the present application. In order to merge and reconstruct the data, the characteristics of the current storage space, including the storage capacity and the structure layout, need to be evaluated to determine the detailed information of the partition of each group of file databases (such as file database group 1 or file database group M in Figure 5 ) in the storage space, i.e. the partition information mentioned above, which at least covers the size of the partition and the order in the storage architecture, to ensure that the physical framework of the data merging operation is constructed.

[0090] Then, according to the determined partition order, the business primary key is used as the merging identifier to sequentially connect the data records belonging to the same business primary key in different partitions (such as record merging in Figure 5 ), which realizes the logical integration of the data and forms the merged file databases, providing data basis for constructing the complete database.

[0091] Secondly, in order to adapt to the change of the storage location after data merging, the source index information of each group of file databases needs to be updated, and the index offset information is adjusted according to the partition size, the partition order and the business primary key (such as index merging in Figure 5 ), to obtain the target index information mentioned above, to ensure that the new data location is accurately recorded to ensure the database query performance.

[0092] Finally, after integrating the target index information and the merged file databases, the final state of the reconstructed international civil aviation freight database is determined, and the update and reconstruction of the international civil aviation freight database are completed.

[0093] Through the above steps, the technical effects of optimizing data storage and indexing and efficiently rebuilding the international civil aviation fare database are achieved. This strategy not only ensures the accuracy of data merging, but also ensures the efficient query performance of the database by dynamically adjusting the index information, providing a fast and reliable processing flow for data storage space maintenance and data recovery. At the same time, the partition merging method customized based on the storage space characteristics fully utilizes the storage resources, reduces the time consumption of data migration and copying, and significantly improves the overall efficiency of database reconstruction.

[0094] In an optional embodiment, taking P02 data type as an example, Table 2 is the partition information of P02 data type corresponding to data processing task before merging, and before merging, it is three file databases, namely P02 data files of partition 0, partition 1 and partition 2. Among them, each file is essentially a memory image of an array of P02 data structure, therefore, the three files can be directly merged in first-to-last order according to the partition order (i.e. the size order of partition serial number) to obtain the final file of P02 data type.

[0095] The index file can be a red-black binary tree data structure, containing key and value, key is a data business primary key string, and value is a data record offset (relative to the first character of the data file) array.

[0096] Table 2

[0097]

[0098] As shown in Table 2, the file size of partition 2 is 40000 Bytes, and a certain primary key (001EK LLCDPTN1) contains three data records with offsets of 0, 40 and 200 (assuming each record length is 40 Bytes). After merging the data of partitions 0, 1 and 2 in the order of partitions 0, 1 and 2, the offsets of the index can be as shown in Table 3.

[0099] Table 3

[0100]

[0101] As can be seen from Table 3, for the data of the original partition 2, the data of partitions 0 and 1 is inserted in front, and all the data is equivalent to moving back 44000+48000=92000 Bytes, so the value part of the index is uniformly increased by 92000 Bytes of offset.

[0102] Optionally, in the first embodiment of the present application, Figure 6 is a flowchart of the optional reconstruction of international fare data provided by the first embodiment of the present application. As Figure 6As shown, the main process is used to read and sort the full data and incremental data according to the publishing time into multiple Jobs. The thread pool in the Job is used to parallel decompression and parsing data. For the data processing task corresponding to the full data, in the data decompression and parsing phase, multiple threads are created through the thread pool technology, and the data decompression and parsing operation is performed by multiple threads. For the data processing task corresponding to the incremental data, in the data decompression and parsing phase, if the current execution Job Z is observed, the previous Job, that is, Job Z-2 is completed, if not, it is slept and waits (for example Figure 4 The start of the execution of the Job Z process in the above depends on the end of the execution of the Z-2 process). It is further explained that although in the waiting state, the CPU resources are not idle, and can process the data reconstruction process of Job Z-1 or Job Z-2 (the data processing operation and index updating operation in the following text).

[0103] Then, according to the hash value of the business primary key, grouping is performed, multiple sub-processes are created in each Job, and each sub-process independently performs data editing and index updating to obtain the file database corresponding to M groups (for example Figure 6 The first group file database and the Mth group file database in the above text). Specifically, for the data processing task corresponding to the full data, in the execution phase of the data processing operation and the index updating operation, the M sub-processes are used to perform the data processing operation and the index updating operation on the structured data in the M groups in parallel. For the data processing task corresponding to the incremental data, in the execution phase of the data processing operation and the index updating operation, if the current execution Job Z is observed, the previous Job, that is, Job Z-1 is completed, if not, it is slept and waits. Therefore, although multiple processes are created at the same time, because of the execution dependency relationship of the processes (for example Figure 4 The start of the execution of the Job Z sub-process in the above depends on the end of the execution of the Job Z-1 sub-process).

[0104] Finally, the file databases processed by all the sub-processes (that is, the M group corresponding file databases in the above text) are merged in the partition order, and the offset of the data record in the index file is adjusted, to obtain the reconstructed international freight rate database.

[0105] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0106] Embodiment Two

[0107] The embodiment two of the application further provides a reconstruction device of an international freight database. It should be noted that the reconstruction device of the international freight database of the embodiment two of the application can be used to execute the reconstruction method for the international freight database provided in the embodiment one of the application. The reconstruction device of the international freight database provided in the embodiment two of the application is introduced as follows.

[0108] Figure 7 FIG. 1 is a schematic diagram of the reconstruction device of the international freight database according to the embodiment two of the application. As shown in the figure, the device comprises an analysis unit 701, a grouping unit 702, a reconstruction unit 703 and a merging unit 704. Figure 7

[0109] Specifically, the analysis unit 701 is configured to perform data analysis operations on data in N tasks in parallel through N processes, to generate structured data of each of the N tasks, wherein the N tasks comprise data processing tasks corresponding to full data and incremental data in the international freight database to be reconstructed, and N is a positive integer.

[0110] The grouping unit 702 is configured to group the structured data of each of the N tasks based on a business primary key, to obtain M groups corresponding to the N tasks, wherein M is a positive integer.

[0111] The reconstruction unit 703 is configured to perform data reconstruction operations on data in the M groups corresponding to the N tasks in parallel through M sub-processes, to generate M file databases corresponding to the N tasks.

[0112] The merging unit 704 is configured to merge the M file databases corresponding to the N tasks according to a partition order, and to determine the reconstructed international freight database according to the merged file database.

[0113] ​The international freight rate database reconstruction device provided in the second embodiment of the present application is configured to perform data analysis operations on data in N tasks in parallel through N processes by the analysis unit 701, to generate structured data of each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in the international freight rate database to be reconstructed, and N is a positive integer; the grouping unit 702 groups the structured data of each of the N tasks based on a business primary key, to obtain M groups corresponding to the N tasks, wherein M is a positive integer; the reconstruction unit 703 performs data reconstruction operations on data in the M groups corresponding to the N tasks in parallel through M sub-processes, to generate M file databases corresponding to the N tasks; and the merging unit 704 merges the M file databases corresponding to the N tasks according to a partition order, to determine the reconstructed international freight rate database according to the merged file databases, thereby solving the problem of low database reconstruction efficiency in the related art due to large data volume and the necessity of processing data in time sequence during reconstruction of the international freight rate database.

[0114] By implementing the concurrent data reconstruction and partition merging strategy, all data processing tasks can be synchronized and advanced, the generation efficiency of structured data is significantly improved, the reconstruction time of the international freight rate database can be significantly shortened, and the technical effect of accelerating the data processing process is achieved. Meanwhile, intelligent partition merging of each task is realized through the thread pool technology, so that the data integration process is efficient and accurate, the international freight rate data can be quickly recovered and the consistency of the cross-regional database can be reconstructed, the low-efficiency bottleneck of the traditional serial processing mode is overcome, especially when processing large-scale data sets, the multi-core processing capability of the server can be fully utilized, the waste of system resources is avoided, the data integrity and the ordered storage according to the release time are ensured, and the flexibility of data storage space maintenance and the reliability of data service are further improved.

[0115] Optionally, in the international freight rate database reconstruction device provided in the second embodiment of the present application, the device further includes an acquisition unit configured to acquire full data and incremental data from a file directory corresponding to the international freight rate database to be reconstructed before performing data analysis operations on data in N tasks in parallel through N processes; a first determination unit configured to determine data processing tasks corresponding to the full data according to the full data; a second determination unit configured to determine modification times of each file included in the incremental data, and to determine N-1 data processing tasks corresponding to the incremental data according to the modification times of each file included in the incremental data; and a third determination unit configured to determine the N tasks according to the data processing tasks corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0116] Optionally, in the international freight rate database reconstruction device provided in Embodiment Two of the present application, the parsing unit 701 comprises: a first creation subunit configured to create a first process for a data processing task corresponding to full data in N tasks; a decompression subunit configured to decompress a plurality of compressed files in the data processing task corresponding to the full data in parallel using a plurality of threads in the first process to obtain a plurality of readable data; and a first parsing subunit configured to parse the plurality of readable data into structured data in parallel using the plurality of threads based on a predefined data structure to generate structured data of the data processing task corresponding to the full data.

[0117] Optionally, in the international freight rate database reconstruction device provided in Embodiment Two of the present application, the parsing unit 701 comprises: a second creation subunit configured to create N-1 processes for N-1 data processing tasks corresponding to incremental data in N tasks; a second parsing subunit configured to decompress and parse compressed files corresponding to the N-1 data processing tasks corresponding to the incremental data in parallel using the N-1 processes, and after the decompression and parsing of the compressed files corresponding to an Xth data processing task corresponding to the incremental data are completed, decompress and parse compressed files corresponding to an X+2th data processing task corresponding to the incremental data, wherein X is a positive integer greater than 0 and less than N-2; and a third parsing subunit configured to, after the decompression and parsing of the compressed files corresponding to an N-3th data processing task corresponding to the incremental data are executed, decompress and parse compressed files corresponding to an N-2th data processing task and compressed files corresponding to an N-1th data processing task corresponding to the incremental data to obtain structured data of the N-1 data processing tasks corresponding to the incremental data.

[0118] Optionally, in the international freight rate database reconstruction device provided in Embodiment Two of the present application, the grouping unit 702 comprises: a first calculation subunit configured to calculate hash values of business primary keys of each data record in structured data of a data processing task corresponding to full data in N tasks; a grouping subunit configured to group the structured data of the data processing task corresponding to the full data according to the hash values corresponding to each data record in the full data to obtain M first groups; a second calculation subunit configured to calculate hash values of business primary keys of each data record in structured data of a data processing task corresponding to incremental data in N tasks; and a division subunit configured to, in a case where a hash value of a target business primary key corresponding to a target group in the M first groups is the same as a hash value corresponding to a target data record in the structured data of the data processing task corresponding to the incremental data, divide the target data in the incremental data into the target group in the M first groups to obtain M groups corresponding to the N tasks.

[0119] Optionally, in the international freight rate database reconstruction device provided in Embodiment Two of the present application, the reconstruction unit 703 comprises: a third creation sub-unit, configured to create M sub-processes for the data in the M groups, and perform data processing operation and index update operation on the structured data of the data processing tasks corresponding to the full-amount data in the N tasks by using the M sub-processes, wherein the data processing operation is used to reconstruct the international freight rate database; a first processing sub-unit, configured to perform data processing operation and index update operation on the structured data of N-1 data processing tasks corresponding to the incremental data in the N tasks in parallel by using the M sub-processes after detecting that the data processing operation and index update operation on the full-amount data are performed, and perform data processing operation and index update operation on the structured data of the Y+1th data processing task corresponding to the incremental data after the data processing operation and index update operation on the structured data of the Yth data processing task corresponding to the incremental data are completed, wherein Y is a positive integer greater than 0 and less than N-1; and a second processing sub-unit, configured to perform data processing operation and index update operation on the structured data of the N-1th data processing task corresponding to the incremental data after the data processing operation and index update operation on the structured data of the N-2th data processing task corresponding to the incremental data are completed, to obtain the M groups of file databases.

[0120] Optionally, in the international freight rate database reconstruction device provided in Embodiment Two of the present application, the merging unit 704 comprises: a first determination sub-unit, configured to determine the partition information of each file database in the M groups of file databases according to the current storage space, wherein the partition information at least comprises partition size and partition order; a merging sub-unit, configured to merge the M groups of file databases in different partitions according to the business primary key in the partition order, to obtain the merged file database; an update sub-unit, configured to update the source index information of the M groups of file databases before merging according to the partition size, the partition order and the business primary key, to obtain the target index information; and a second determination sub-unit, configured to determine the reconstructed international freight rate database according to the target index information and the merged file database.

[0121] The international freight rate database reconstruction device comprises a processor and a memory, and the above-mentioned analysis unit 701, grouping unit 702, reconstruction unit 703 and merging unit 704 are all stored in the memory as program units, and the corresponding functions are realized by the processor executing the above-mentioned program units stored in the memory.

[0122] The processor comprises a core, and the core retrieves the corresponding program units from the memory. One or more than one core can be set, and the international freight rate database reconstruction efficiency can be improved by adjusting the core parameters.

[0123] The memory can include non-persistent memory in a computer readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory, including at least one memory chip.

[0124] Embodiment three of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the method for reconstructing an international freight rate database.

[0125] Embodiment four of the present application provides a processor, which is used to run a program, and the program is executed to implement the method for reconstructing an international freight rate database.

[0126] As shown in Figure 8 Embodiment five of the present application provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program: performing data parsing operations on data in N tasks in parallel through N processes to generate structured data of each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in an international freight rate database to be reconstructed, and N is a positive integer; grouping the structured data of each of the N tasks based on a business primary key to obtain M groups corresponding to the N tasks, wherein M is a positive integer; performing data reconstruction operations on data in the M groups corresponding to the N tasks in parallel through M sub-processes to generate M file databases corresponding to the N tasks; and merging the M file databases corresponding to the N tasks according to a partition order to determine a reconstructed international freight rate database according to the merged file databases.

[0127] The processor further implements the following steps when executing the program: before performing the data parsing operations on the data in the N tasks in parallel through the N processes, the above method further includes: reading a file directory corresponding to the international freight rate database to be reconstructed to obtain the full data and the incremental data from the file directory; determining a data processing task corresponding to the full data according to the full data; determining modification times of each file included in the incremental data, and determining N-1 data processing tasks corresponding to the incremental data for each file included in the incremental data according to the modification times; and determining the N tasks according to the data processing task corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0128] The processor further implements the following steps when executing the program: performing data parsing operations on data in the N tasks in parallel through N processes to generate structured data of each of the N tasks, including: creating a first process for a data processing task corresponding to full data in the N tasks; decompressing a plurality of compressed files in the data processing task corresponding to the full data in parallel using a plurality of threads in the first process to obtain a plurality of readable data; and parsing the plurality of readable data into structured data based on a predefined data structure to generate structured data of the data processing task corresponding to the full data.

[0129] The processor further implements the following steps when executing the program: performing data parsing operations on data in the N tasks in parallel through N processes to generate structured data of each of the N tasks, including: creating a first process for a data processing task corresponding to full data in the N tasks; decompressing a plurality of compressed files in the data processing task corresponding to the full data in parallel using a plurality of threads in the first process to obtain a plurality of readable data; and parsing the plurality of readable data into structured data based on a predefined data structure to generate structured data of the data processing task corresponding to the full data.

[0130] The processor further implements the following steps when executing the program: grouping structured data of each of the N tasks based on a business primary key to obtain M groups corresponding to the N tasks, including: calculating a hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full data in the N tasks; grouping the structured data of the data processing task corresponding to the full data according to the hash value corresponding to each data record in the full data to obtain M first groups; calculating a hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; and dividing target data in the incremental data into a target group in the M first groups when the hash value of the business primary key corresponding to any target group in the M first groups is the same as the hash value corresponding to any target data record in the structured data of the data processing task corresponding to the incremental data.

[0131] The processor further implements the following steps when executing the program: performing data reconstruction operations on data in the M groups corresponding to the N tasks in parallel through the M sub-processes to generate the M groups of file databases corresponding to the N tasks, including: creating the M sub-processes for the data in the M groups, and performing data processing operations and index updating operations on structured data of the data processing tasks corresponding to the full data in the N tasks using the M sub-processes, wherein the data processing operations are used to reconstruct the international freight rate database; after detecting that the data processing operations and the index updating operations on the full data are completed, performing data processing operations and index updating operations on structured data of N-1 data processing tasks corresponding to the incremental data in the N tasks in parallel using the M sub-processes, and after the data processing operations and the index updating operations on the structured data of the Yth data processing task corresponding to the incremental data are completed, performing data processing operations and index updating operations on structured data of the Y+1th data processing task corresponding to the incremental data, wherein Y is a positive integer greater than 0 and less than N-1; after the data processing operations and the index updating operations on the structured data of the N-2th data processing task corresponding to the incremental data are completed, performing data processing operations and index updating operations on structured data of the N-1th data processing task corresponding to the incremental data, to obtain the M groups of file databases.

[0132] The processor further implements the following steps when executing the program: merging the M groups of file databases corresponding to the N tasks according to the partition order, and determining the reconstructed international freight rate database according to the merged file databases, including: determining partition information of each file database in the M groups of file databases according to the current storage space, wherein the partition information at least includes: partition size, partition order; merging the M groups of file databases in different partitions according to the business primary key according to the partition order to obtain the merged file databases; updating source index information of the M groups of file databases before merging according to the partition size, the partition order and the business primary key to obtain target index information; and determining the reconstructed international freight rate database according to the target index information and the merged file databases.

[0133] The device herein can be a server, a PC, a PAD, a mobile phone, etc.

[0134] The application further provides a computer program product, which is adapted to execute a program initializing the following method steps when executed on a data processing device: performing data parsing operations on data in N tasks in parallel through N processes to generate structured data of each of the N tasks, wherein the N tasks comprise data processing tasks corresponding to full data and incremental data of an international freight rate database to be reconstructed, N is a positive integer; grouping the structured data of each of the N tasks based on a business primary key to obtain M groups corresponding to the N tasks, wherein M is a positive integer; performing data reconstruction operations on data in the M groups corresponding to the N tasks in parallel through M sub-processes to generate M file databases corresponding to the N tasks; and merging the M file databases corresponding to the N tasks according to a partition order to determine the reconstructed international freight rate database according to the merged file databases.

[0135] When executed on a data processing device, the program is also adapted to initialize the following method steps: before performing data parsing operations on data in N tasks in parallel through N processes, the above method further comprises: reading a file directory corresponding to an international freight rate database to be reconstructed to obtain full data and incremental data from the file directory; determining a data processing task corresponding to the full data according to the full data; determining modification times of each file contained in the incremental data, and determining N-1 data processing tasks corresponding to the incremental data according to the modification times of each file contained in the incremental data; and determining the N tasks according to the data processing task corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

[0136] When executed on a data processing device, the program is also adapted to initialize the following method steps: performing data parsing operations on data in N tasks in parallel through N processes to generate structured data of each of the N tasks, comprising: creating a first process for a data processing task corresponding to full data in the N tasks; decompressing a plurality of compressed files in the data processing task corresponding to the full data in parallel through a plurality of threads in the first process to obtain a plurality of readable data; and parsing the plurality of readable data into structured data in parallel through the plurality of threads based on a predefined data structure to generate structured data of the data processing task corresponding to the full data.

[0137] When executed on the data processing device, the program is further adapted to execute the following method steps: performing, by N processes, data parsing operations on data in N tasks in parallel to generate structured data of each of the N tasks, comprising: creating N-1 processes for N-1 data processing tasks corresponding to incremental data in the N tasks; performing, by the N-1 processes, decompression and parsing on compressed files corresponding to the N-1 data processing tasks of the incremental data in parallel, and after completing decompression and parsing on a compressed file corresponding to an Xth data processing task of the incremental data, performing decompression and parsing on a compressed file corresponding to an X+2th data processing task of the incremental data, where X is a positive integer greater than 0 and less than N-2; after performing decompression and parsing on a compressed file corresponding to an N-3th data processing task of the incremental data ends, performing decompression and parsing on compressed files corresponding to an N-2th data processing task and an N-1th data processing task of the incremental data to obtain structured data of the N-1 data processing tasks of the incremental data.

[0138] When executed on the data processing device, the program is further adapted to execute the following method steps: grouping structured data of each of the N tasks based on a business primary key to obtain M groups corresponding to the N tasks, comprising: calculating a hash value of the business primary key of each data record in structured data of a data processing task corresponding to full data in the N tasks; grouping the structured data of the data processing task corresponding to the full data according to the hash value corresponding to each data record in the full data to obtain M first groups; calculating a hash value of the business primary key of each data record in structured data of a data processing task corresponding to incremental data in the N tasks; in a case where a hash value of a business primary key corresponding to any target group in the M first groups is the same as a hash value corresponding to any target data record in the structured data of the data processing task corresponding to the incremental data, dividing the target data in the incremental data into the target group in the M first groups to obtain the M groups corresponding to the N tasks.

[0139] When executed on the data processing device, the program is also adapted to perform the following method steps: performing, by M sub-processes in parallel, data reconstruction operations on data in M groups corresponding to N tasks, to generate M groups of file databases corresponding to the N tasks, including: for the data in the M groups, creating M sub-processes, and using the M sub-processes to perform data processing operations and index update operations on structured data of data processing tasks corresponding to full data in the N tasks, wherein the data processing operations are used to reconstruct the international freight rate database; after detecting that the data processing operations and the index update operations on the full data are performed, using the M sub-processes in parallel to perform data processing operations and index update operations on structured data of N-1 data processing tasks corresponding to incremental data in the N tasks, and after the data processing operations and the index update operations on structured data of a Yth data processing task corresponding to the incremental data are completed, performing data processing operations and index update operations on structured data of a (Y+1)th data processing task corresponding to the incremental data, wherein Y is a positive integer greater than 0 and less than N-1; after the data processing operations and the index update operations on structured data of an (N-2)th data processing task corresponding to the incremental data are completed, performing data processing operations and index update operations on structured data of an (N-1)th data processing task corresponding to the incremental data, to obtain the M groups of file databases.

[0140] When executed on the data processing device, the program is also adapted to perform the following method steps: merging, according to a partition order, M groups of file databases corresponding to N tasks, and determining a reconstructed international freight rate database according to the merged file databases, including: determining, according to a current storage space, partition information of a partition in which each file database in the M groups of file databases is located, wherein the partition information at least includes a partition size and a partition order; merging, according to the partition order, the M groups of file databases in different partitions according to a business primary key, to obtain a merged file database; updating source index information of the M groups of file databases before merging according to the partition size, the partition order, and the business primary key, to obtain target index information; and determining the reconstructed international freight rate database according to the target index information and the merged file database.

[0141] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) containing computer-usable program code.

[0142] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0143] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0144] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0145] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or the like, which can be used to store the code and / or data structures for use while the device is in the on mode. The memory can also include non-volatile memory, such as read-only memory (ROM) and / or the like, which can be used to store instructions and / or data structures when the device is in the off mode.

[0146] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0147] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0148] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0149] The above merely provides embodiments of the present application and is not intended to limit the present application. Various modifications and changes can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A method for rebuilding an international freight rate database, characterized in that: include: Performing data parsing operations on data in N tasks in parallel through N processes to generate structured data for each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in the international freight rate database to be rebuilt, and N is a positive integer; Grouping the structured data of each of the N tasks based on the business primary key to obtain M groups corresponding to the N tasks, where M is a positive integer; Performing data reconstruction operations on the data in the M groups corresponding to the N tasks in parallel through M subprocesses to generate M groups of file databases corresponding to the N tasks; The M groups of file databases corresponding to the N tasks are merged in a partition order, and a reconstructed international freight rate database is determined based on the merged file databases.

2. The method according to claim 1, characterized in that Before performing data parsing operations on the data in the N tasks in parallel through the N processes, the method further includes: Reading a file directory corresponding to the international freight rate database to be rebuilt, and obtaining the full data and the incremental data from the file directory; Determining a data processing task corresponding to the full amount of data based on the full amount of data; Determine the modification time of each file included in the incremental data, and determine N-1 data processing tasks corresponding to the incremental data for each file included in the incremental data according to the modification time; The N tasks are determined based on the data processing tasks corresponding to the full data and the N-1 data processing tasks corresponding to the incremental data.

3. The method according to claim 1, characterized in that Performing data parsing operations on the data in N tasks in parallel through N processes to generate structured data for each of the N tasks, including: Creating a first process for the data processing task corresponding to the full amount of data in the N tasks; Decompressing multiple compressed files in the data processing task corresponding to the full amount of data in parallel using multiple threads in the first process to obtain multiple readable data; Based on a predefined data structure, the multiple threads are used in parallel to parse the multiple readable data into structured data, and generate structured data of the data processing task corresponding to the full amount of data.

4. The method according to claim 1, wherein Performing data parsing operations on the data in N tasks in parallel through N processes to generate structured data for each of the N tasks, including: For N-1 data processing tasks corresponding to the incremental data in the N tasks, create N-1 processes; Decompressing and parsing the compressed files corresponding to the N-1 data processing tasks corresponding to the incremental data in parallel using the N-1 processes, and after completing the decompression and parsing of the compressed files corresponding to the Xth data processing task corresponding to the incremental data, decompressing and parsing the compressed files corresponding to the X+2th data processing task corresponding to the incremental data, where X is a positive integer greater than 0 and less than N-2; After the compressed file corresponding to the N-3th data processing task corresponding to the incremental data is decompressed and parsed, the compressed file corresponding to the N-2th data processing task corresponding to the incremental data and the compressed file corresponding to the N-1th data processing task are decompressed and parsed to obtain structured data of the N-1 data processing task corresponding to the incremental data.

5. The method according to claim 1, wherein The structured data of each of the N tasks is grouped based on the business primary key to obtain M groups corresponding to the N tasks, including: Calculate the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the full amount of data in the N tasks; Grouping the structured data of the data processing task corresponding to the full data according to the hash value corresponding to each data record in the full data to obtain M first groups; Calculate the hash value of the business primary key of each data record in the structured data of the data processing task corresponding to the incremental data in the N tasks; When the hash value of the business primary key corresponding to any target group in the M first groups is the same as the hash value corresponding to any target data record in the structured data of the data processing task corresponding to the incremental data, the target data in the incremental data is divided into the target groups in the M first groups to obtain M groups corresponding to the N tasks.

6. The method according to claim 1, characterized in that Performing a data reconstruction operation on the data in the M groups corresponding to the N tasks in parallel through M subprocesses to generate M groups of file databases corresponding to the N tasks, including: For the data in the M groups, create M subprocesses, and use the M subprocesses to perform data processing operations and index update operations on structured data of the data processing tasks corresponding to the full data in the N tasks, wherein the data processing operations are used to rebuild the international freight rate database; After detecting that the data processing operation and index update operation on the full data are completed, the M sub-processes are used to perform data processing operations and index update operations on the structured data of the N-1 data processing tasks corresponding to the incremental data in the N tasks in parallel, and after the data processing operation and index update operation on the structured data corresponding to the Y-th data processing task corresponding to the incremental data are completed, the data processing operation and index update operation are performed on the structured data corresponding to the Y+1-th data processing task corresponding to the incremental data, where Y is a positive integer greater than 0 and less than N-1; After completing the data processing operation and index update operation on the structured data corresponding to the N-2th data processing task corresponding to the incremental data, perform data processing operation and index update operation on the structured data corresponding to the N-1th data processing task corresponding to the incremental data to obtain the M group file database.

7. The method according to claim 1, characterized in that Merging the M groups of file databases corresponding to the N tasks in a partition order, and determining a reconstructed international freight rate database based on the merged file databases, including: Determining partition information of a partition where each file database in the M groups of file databases is located based on the current storage space, wherein the partition information at least includes: a partition size and the partition order; Merging M groups of file databases in different partitions according to the partition order and the business primary key to obtain the merged file database; updating the source index information of the M groups of file databases before merging according to the partition size, the partition order, and the business primary key to obtain target index information; The reconstructed international freight rate database is determined based on the target index information and the merged file database.

8. A device for rebuilding an international freight rate database, characterized in that: include: a parsing unit, configured to perform data parsing operations on data in N tasks in parallel through N processes to generate structured data for each of the N tasks, wherein the N tasks include data processing tasks corresponding to full data and incremental data in the international freight rate database to be rebuilt, and N is a positive integer; a grouping unit, configured to group the structured data of each of the N tasks based on a business primary key to obtain M groups corresponding to the N tasks, where M is a positive integer; a reconstruction unit, configured to perform a data reconstruction operation on the data in the M groups corresponding to the N tasks in parallel through M subprocesses, to generate M groups of file databases corresponding to the N tasks; The merging unit is used to merge the M groups of file databases corresponding to the N tasks in a partition order, and determine the reconstructed international freight rate database based on the merged file databases.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes stored computer instructions, wherein when the computer instructions are executed by a processor, the method for reconstructing the international freight rate database according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The system comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for reconstructing an international freight rate database according to any one of claims 1 to 7.

Images