Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment, and relates to the technical field of data processing. The method comprises the steps of obtaining multiple pieces of source data from multiple databases, and determining a current version of each piece of source data; obtaining a source data updating time table and dynamic registration information of each computing node from a registration configuration center; updating the current version of each piece of source data according to the source data updating time table and the dynamic registration information of each computing node to obtain business data of a target version; and performing calculation processing on the business data of the target version to obtain target freight rate data. Through application of the method and the device, the problem that the efficiency of calculating the freight rate data by using the updated business data is relatively low because the data volume of the business data is too large and the time for updating the data is relatively long in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device, a storage medium, and an electronic device. Background Art

[0002] In the related art, the architecture of air freight rate calculation is AllInOne architecture, that is, the program and data are on the same machine, and data access in the calculation process is performed locally. Moreover, in the related art, a memory-mapped database (MemoryMap) is generally used as a carrier of freight rate and rule data, and its data update process is: first complete the data update at the update node, and then send the changed memory-mapped file to the computing cluster (a cluster composed of multiple nodes) through the multicast network. However, the amount of business data in the related art is getting higher and higher, and the overall data update process in the related art is data construction (update node) -> data compression (update node) -> data sending / receiving (multicast between update node and computing node) -> data decompression and aggregation (computing node). And as the amount of data increases, the data update process in the related art takes longer and longer time, which leads to low efficiency in calculating freight rate data using updated business data.

[0003] Currently, no effective solution has been proposed to the problem that the amount of business data in related technologies is too large, resulting in a long time to update the data, which in turn leads to low efficiency in calculating freight rate data using the updated business data. Summary of the invention

[0004] The main purpose of the present application is to provide a data processing method and device, a storage medium and an electronic device to solve the problem in the related art that the amount of business data is too large, resulting in a long time to update the data, and then resulting in low efficiency in calculating freight rate data using the updated business data.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data processing method is provided. The method comprises: obtaining multiple source data from multiple databases, and determining the current version of each source data, wherein the multiple source data include business data for calculating freight rate data; obtaining the source data update schedule and the dynamic registration information of each computing node from the registration configuration center, wherein the source data update schedule includes the update time of each source data version, and the dynamic registration information includes the version information of the data in each computing node; updating the current version of each source data according to the source data update schedule and the dynamic registration information of each computing node to obtain the business data of the target version; and calculating the business data of the target version to obtain the target freight rate data.

[0006] Furthermore, before obtaining multiple source data from multiple databases, the method also includes: classifying the multiple source data according to type information of each source data to obtain multiple types of source data; based on the type of source data stored in each database, storing the multiple types of source data in the multiple databases.

[0007] Furthermore, based on the source data update schedule and the dynamic registration information of each computing node, the current version of each source data is updated to obtain the target version of business data, including: based on the source data update schedule, the current version of each source data is updated to obtain the latest version information of each source data; based on the dynamic registration information of each computing node, the latest version of each source data is updated to obtain the target version information of each source data; based on the target version information of each source data, a data version table is constructed, wherein the data version table is used to record the version information of the business data as the target version; based on the data version table, the business data of the target version is obtained.

[0008] Furthermore, based on the source data update schedule, the current version of each source data is updated to obtain the latest version information of each source data, including: obtaining the update time of the current version of each source data, and obtaining the update time of each source data version from the source data update schedule; judging whether the update time of the current version of each source data is earlier than the update time of each source data version in the source data update schedule to obtain a judgment result; based on the judgment result, the current version of each source data is updated to obtain the latest version information of each source data.

[0009] Furthermore, based on the judgment result, the current version of each source data is updated to obtain the latest version information of each source data, including: if the judgment result indicates that the update time of the current version of each source data is not earlier than the update time of the version of each source data in the source data update schedule, it means that there is no need to update the current version of each source data; if the judgment result indicates that the update time of the current version of each source data is earlier than the update time of the version of each source data in the source data update schedule, the current version of each source data is updated to obtain the latest version information of each source data.

[0010] Furthermore, based on the dynamic registration information of each computing node, the latest version of each source data is updated to obtain the target version information of each source data, including: obtaining the version information of the data in each computing node from the dynamic registration information of each computing node; judging whether the version of the data in each computing node has been updated to the target version based on the version information of the data in each computing node; if there is at least one computing node whose data version has not been updated to the target version, updating the latest version of each source data is prohibited; if the version of the data in each computing node has been updated to the target version, updating the latest version of each source data to obtain the target version information of each source data.

[0011] Furthermore, based on the data version table, obtaining the target version of the business data includes: acquiring version information of the business data from the data version table; and acquiring the target version of the business data from the multiple databases according to the version information of the business data.

[0012] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a data processing device is provided. The device includes: a first processing unit, which is used to obtain multiple source data from multiple databases and determine the current version of each source data, wherein the multiple source data include business data for calculating freight rate data; a first acquisition unit, which is used to obtain the source data update schedule and the dynamic registration information of each computing node from the registration configuration center, wherein the source data update schedule includes the update time of each source data version, and the dynamic registration information includes the version information of the data in each computing node; a second processing unit, which is used to update the current version of each source data according to the source data update schedule and the dynamic registration information of each computing node to obtain the target version of business data; a third processing unit, which is used to calculate the target version of business data to obtain the target freight rate data.

[0013] Furthermore, the device also includes: a fourth processing unit, which is used to classify the multiple source data according to type information of each source data before obtaining the multiple source data from the multiple databases, so as to obtain multiple types of source data; and a first storage unit, which is used to store the multiple types of source data in the multiple databases based on the type of source data stored in each database.

[0014] Furthermore, the second processing unit includes: a first processing sub-unit, used to update the current version of each source data according to the source data update schedule, and obtain the latest version information of each source data; a second processing sub-unit, used to update the latest version of each source data according to the dynamic registration information of each computing node, and obtain the target version information of each source data; a first construction sub-unit, used to construct a data version table based on the target version information of each source data, wherein the data version table is used to record the version information of the business data as the target version; a first determination sub-unit, used to obtain the business data of the target version based on the data version table.

[0015] Furthermore, the first processing sub-unit includes: a first acquisition module, used to obtain the update time of the current version of each source data, and obtain the update time of each source data version from the source data update schedule; a first judgment module, used to judge whether the update time of the current version of each source data is earlier than the update time of each source data version in the source data update schedule, and obtain a judgment result; a first processing module, used to update the current version of each source data according to the judgment result, and obtain the latest version information of each source data.

[0016] Furthermore, the first processing module includes: a first processing sub-module, which is used to indicate that there is no need to update the current version of each source data if the judgment result indicates that the update time of the current version of each source data is not earlier than the update time of the version of each source data in the source data update schedule; and a second processing sub-module, which is used to update the current version of each source data to obtain the latest version information of each source data if the judgment result indicates that the update time of the current version of each source data is earlier than the update time of the version of each source data in the source data update schedule.

[0017] Furthermore, the second processing sub-unit includes: a second acquisition module, used to obtain version information of the data in each computing node from the dynamic registration information of each computing node; a second judgment module, used to judge whether the version of the data in each computing node has been updated to the target version based on the version information of the data in each computing node; a second processing module, used to prohibit updating the latest version of each source data if the version of the data in at least one computing node has not been updated to the target version; a third processing module, used to update the latest version of each source data if the version of the data in each computing node has been updated to the target version to obtain the target version information of each source data.

[0018] Furthermore, the first determination subunit includes: a third acquisition module, used to obtain version information of the business data from the data version table; and a fourth acquisition module, used to obtain the business data of the target version from the multiple databases according to the version information of the business data.

[0019] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a program, wherein the program executes any one of the above-mentioned data processing methods.

[0020] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, which includes one or more processors and a memory, and the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the data processing methods described above.

[0021] Through the present application, the following steps are adopted: multiple source data are obtained from multiple databases, and the current version of each source data is determined, wherein the multiple source data include business data for calculating freight rate data; the source data update schedule and the dynamic registration information of each computing node are obtained from the registration configuration center, wherein the source data update schedule includes the update time of each source data version, and the dynamic registration information includes the version information of the data in each computing node; according to the source data update schedule and the dynamic registration information of each computing node, the current version of each source data is updated to obtain the business data of the target version; the business data of the target version is calculated to obtain the target freight rate data, which solves the problem in the related art that the amount of business data is too large, resulting in a long time to update the data, and then resulting in low efficiency in calculating the freight rate data using the updated business data. By splitting a set of databases into multiple databases, and using multiple databases to store different business data respectively, and then updating the version of the business data stored in each database according to the source data update schedule and the dynamic registration information of each computing node, and then using the updated business data to calculate the freight data, compared with the prior art of using a set of databases to store data and update the data, the time for updating data can be reduced, and when updating the data in multiple databases, the final data consistency of the entire set of databases can be maintained, thereby achieving the effect of improving the efficiency of calculating freight data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a flow chart of a data processing method provided according to an embodiment of the present application;

[0024] Figure 2 It is a schematic diagram of the data update process under the AllInOne architecture provided according to the prior art;

[0025] Figure 3 is a schematic diagram of the logic of the data control node in this embodiment;

[0026] Figure 4 is a schematic diagram of the business logic of the computing node in this embodiment;

[0027] Figure 5 is a schematic diagram of a data processing device provided according to an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0033] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of a data processing method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0034] Step S101, obtaining a plurality of source data from a plurality of databases, and determining a current version of each source data, wherein the plurality of source data include business data for calculating freight rate data.

[0035] For example, one or more source data may be obtained from each database, and the current data version of each source data may be obtained, and the one or more source data obtained from each database may include business data for calculating air fares.

[0036] Step S102, obtaining a source data update schedule and dynamic registration information of each computing node from a registration configuration center, wherein the source data update schedule includes the update time of each source data version, and the dynamic registration information includes the version information of the data in each computing node.

[0037] For example, the source data update schedule of the registration configuration center is read, and the dynamic registration information of each node is read from the registration configuration center, and the dynamic registration information of each node read from the registration configuration center includes the basic information of each node and the current data version status.

[0038] Step S103 , updating the current version of each source data according to the source data update schedule and the dynamic registration information of each computing node to obtain the target version of business data.

[0039] For example, according to the source data update schedule and the dynamic registration information of each node, the current version of each source data is updated.

[0040] Step S104, calculating and processing the target version of the business data to obtain target freight rate data.

[0041] For example, after the data version is updated, the updated business data can be used to calculate the airline freight rate data.

[0042] Through the above steps S101 to S104, by splitting a set of databases into multiple databases, and using the multiple databases to store different business data respectively, and then updating the version of the business data stored in each database according to the source data update schedule and the dynamic registration information of each computing node, and then using the updated business data to calculate the freight data, compared with using a set of databases to store data and update the data in the prior art, the time for updating data can be reduced, and when updating the data in multiple databases, the final data consistency of the entire set of databases can be maintained, thereby achieving the effect of improving the efficiency of calculating freight data.

[0043] Optionally, in the data processing method provided in the embodiment of the present application, before obtaining multiple source data from multiple databases, the method also includes: classifying the multiple source data according to the type information of each source data to obtain multiple types of source data; based on the type of source data stored in each database, storing the multiple types of source data in multiple databases.

[0044] For example, if the first database stores source data of type A, and the second database stores source data of type B, then when storing data 1, data 2, and data 3 in the database, you can first determine which type of data data 1, data 2, and data 3 belong to, and then store data 1, data 2, and data 3 in the corresponding database. For example, if data 1 and data 2 belong to source data of type A, data 1 and data 2 can be stored in the first database; if data 3 belongs to source data of type B, data 3 can be stored in the second database.

[0045] Through the above solution, data can be stored in the corresponding database quickly and accurately according to the type information of the data.

[0046] Optionally, in the data processing method provided in the embodiment of the present application, the current version of each source data is updated according to the source data update schedule and the dynamic registration information of each computing node to obtain the target version of the business data, including: according to the source data update schedule, the current version of each source data is updated to obtain the latest version information of each source data; according to the dynamic registration information of each computing node, the latest version of each source data is updated to obtain the target version information of each source data; based on the target version information of each source data, a data version table is constructed, wherein the data version table is used to record the version information of the business data as the target version; based on the data version table, the target version of the business data is obtained.

[0047] For example, according to the source data update schedule, the current version of each source data is updated to the latest version, and then according to the dynamic registration information of each node, the latest version of each source data is updated to the target version, and the data version table is used to record the data of the target version. Finally, the data of the target version that is ultimately required can be obtained from the data version table.

[0048] Through the above solution, the business data of the updated target version can be obtained quickly and accurately.

[0049] Optionally, in the data processing method provided in the embodiment of the present application, the current version of each source data is updated according to the source data update schedule to obtain the latest version information of each source data, including: obtaining the update time of the current version of each source data, and obtaining the update time of each source data version from the source data update schedule; judging whether the update time of the current version of each source data is earlier than the update time of each source data version in the source data update schedule to obtain a judgment result; and based on the judgment result, updating the current version of each source data to obtain the latest version information of each source data.

[0050] For example, the update time of each source data version is obtained from the source data update schedule, and the update time of the current version of the source data is compared to see whether it is earlier than the update time of each source data version, and the current version of each source data is updated according to the comparison result to obtain the updated version of the source data.

[0051] In summary, according to the update time of the data version, it is possible to quickly and accurately determine whether the current version of the source data needs to be updated.

[0052] Optionally, in the data processing method provided in the embodiment of the present application, based on the judgment result, the current version of each source data is updated to obtain the latest version information of each source data, including: if the judgment result indicates that the update time of the current version of each source data is not earlier than the update time of the version of each source data in the source data update schedule, it means that there is no need to update the current version of each source data; if the judgment result indicates that the update time of the current version of each source data is earlier than the update time of the version of each source data in the source data update schedule, then the current version of each source data is updated to obtain the latest version information of each source data.

[0053] For example, if the update time of the current version of the source data is not earlier than the update time of each version of the source data, there is no need to update the current version of each source data; if the update time of the current version of the source data is earlier than the update time of each version of the source data, then the current version of each source data needs to be updated.

[0054] In summary, by comparing the update time of data versions, the data versions can be updated conveniently.

[0055] Optionally, in the data processing method provided in the embodiment of the present application, the latest version of each source data is updated based on the dynamic registration information of each computing node to obtain the target version information of each source data, including: obtaining the version information of the data in each computing node from the dynamic registration information of each computing node; judging whether the version of the data in each computing node has been updated to the target version based on the version information of the data in each computing node; if there is at least one computing node whose data version has not been updated to the target version, updating the latest version of each source data is prohibited; if the version of the data in each computing node has been updated to the target version, updating the latest version of each source data to obtain the target version information of each source data.

[0056] For example, determine whether the version of the data in the computing node has been updated to the target version. If the version of the data in the computing node has not been updated to the target version, it is prohibited to update the latest version of the source data; if the version of the data in each computing node has been updated to the target version, the latest version of the source data is updated.

[0057] Through the above scheme, the target version information of the source data can be obtained quickly and accurately according to the version status of the compared data.

[0058] Optionally, in the data processing method provided in an embodiment of the present application, obtaining the target version of business data based on the data version table includes: obtaining version information of the business data from the data version table; and obtaining the target version of business data from multiple databases based on the version information of the business data.

[0059] For example, a data version table may be used to record the version of business data, and then the corresponding version of data may be obtained from the database based on the version of the business data recorded in the data version table.

[0060] Through the above solution, the business data of the required version can be quickly and accurately obtained according to the data version record table.

[0061] In this embodiment, the data is split from the AllInOne stand-alone mode to a distributed mode, allowing multiple servers to store different business data for distributed parallel construction and calculation. The data growth problem can be easily addressed by continuously adjusting and refining the split granularity and horizontally expanding the number of nodes.

[0062] For example, this embodiment mainly supports the distributed data update of the freight rate calculation service, and its workflow mainly includes data slicing, data grouping, data construction and data set creation. It uses four components: source data management node (provides original data management and download services), registration configuration center (a key-value data storage service, such as Redis / etcd), basic container scheduling service (k8s or self-built container management service), data control node (Master node or service).

[0063] For example, Figure 2 It is a schematic diagram of the data update process under the AllInOne architecture provided by the prior art, such as Figure 2 As shown in the figure, under the AllInOne architecture, since the database itself is completely built and generated by a single data construction node and transmitted to the computing cluster for use, the entire cluster naturally has data consistency, and the data on all nodes is exactly the same.

[0064] In addition, data updates under a distributed architecture are relatively complex. Different data is built on different machines and used directly by the computing program locally, rather than being transmitted through a multicast network. Since the types and amounts of data on each node are different, and the build time is also different, a technical solution is needed to ensure consistent updates of the entire cluster's overall data. In this embodiment, this part of the function is completed by the data control node with the help of the registration configuration center. Figure 3 is a schematic diagram of the logic of the data control node in this embodiment, such as Figure 3 As shown in the figure, the unified data version update process of the cluster is as follows:

[0065] Step 1: The data control node keeps polling the source data management node, records the "latest source data version", and reads the "source data update schedule" of the registration configuration center. According to the source data update schedule, it determines whether to refresh the "latest source data version" of a certain business data to the "latest data version".

[0066] Step 2: The data control node reads the dynamic registration information of each node from the registration configuration center. The information contains the basic information of each node and the "current data version". If all computing nodes have been updated to the "current data version", proceed to step 3, otherwise return to step 1.

[0067] Step 3: The data control node updates its own "latest data version" to the "current data version" of the registration configuration center.

[0068] Step 4: Each node in the computing cluster reads the data version configuration table (obtains the latest "current data version") and reads the "physical node and data grouping table" from the registration configuration center. Each node queries which grouped data should be created based on its own hostname and executes the build task.

[0069] Step 5: The computing node (cluster) will continuously report its heartbeat information (including basic node information and construction task status) to the registration configuration center for use by the data control node for decision-making.

[0070] The above steps 1 to 5 describe the process of data version update from the data control node as the main body.

[0071] For example, Figure 4 is a schematic diagram of the business logic of the computing node in this embodiment, such as Figure 4 As shown, the logic of the computing node is as follows:

[0072] Step 1: The computing node obtains the data version table from the registration configuration center to obtain the "current data version", completes the download of the source data from the source data management node according to the "current data version", and stores it in the / rawdata directory of the local machine.

[0073] Step 2: The computing node obtains the physical node and data grouping table from the registration configuration center, queries the table based on its own hostname, and obtains the data grouping it should build.

[0074] Step 3: According to the data grouping and its own physical configuration, start the corresponding construction container to build the database. The number of construction containers corresponds to the number of groups.

[0075] Step 4: When the container is started, the environment variable cluster will be set, that is, the group to be built, and the host's / rawdata will be mounted in read-only mode to the container's / rawdata directory, and the host's / opt / app / dbstore / {cluster_version} / cluster will be mounted in read-write mode to the container's / databases directory. Cluster_version refers to a specific data splitting method, which is equivalent to determining how many groups there are in total and what data each group contains.

[0076] Step 5: Start the specific build program in the build container, and store the database build in its own / databases directory according to the cluster grouping.

[0077] Step 6: After the build container is executed, it automatically exits. The computing node checks the results of the database build and updates the information to the registration configuration center.

[0078] In this implementation, in order to solve the problem of insufficient physical resources of the machine due to data growth in the old AllInOne architecture and the need for vertical expansion (upgrading the machine's memory and disk, etc.), a set of MemoryMap databases is split according to business dimensions to form dozens of smaller group databases, which are then distributedly constructed and provided with distributed computing services in a set of computing node clusters.

[0079] Moreover, under the premise of splitting a set of databases into dozens of small group databases, this embodiment can still ensure the orderly update of data and maintain the final data consistency of the entire set of databases, that is, all group databases are built with the same source data and are at the same source data version, so that the calculation results are consistent with the AllInOne architecture, and the data growth problem can be easily addressed by horizontal expansion (increasing the number of physical nodes) or increasing the split groups. At the same time, the construction process of small groups can be faster, making the data effective time shorter than the original architecture.

[0080] In summary, the data processing method provided by the embodiment of the present application obtains multiple source data from multiple databases and determines the current version of each source data, wherein the multiple source data include business data for calculating freight rate data; obtains the source data update schedule and the dynamic registration information of each computing node from the registration configuration center, wherein the source data update schedule includes the update time of each source data version, and the dynamic registration information includes the version information of the data in each computing node; updates the current version of each source data according to the source data update schedule and the dynamic registration information of each computing node to obtain the business data of the target version; calculates and processes the business data of the target version to obtain the target freight rate data, which solves the problem in the related art that the amount of business data is too large, resulting in a long time to update the data, and then resulting in low efficiency in calculating the freight rate data using the updated business data. By splitting a set of databases into multiple databases, and using multiple databases to store different business data respectively, and then updating the version of the business data stored in each database according to the source data update schedule and the dynamic registration information of each computing node, and then using the updated business data to calculate the freight data, compared with the prior art of using a set of databases to store data and update the data, the time for updating data can be reduced, and when updating the data in multiple databases, the final data consistency of the entire set of databases can be maintained, thereby achieving the effect of improving the efficiency of calculating freight data.

[0081] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0082] The embodiment of the present application also provides a data processing device. It should be noted that the data processing device of the embodiment of the present application can be used to execute the data processing method provided by the embodiment of the present application. The data processing device provided by the embodiment of the present application is introduced below.

[0083] Figure 5 Schematic diagram of a data processing device provided according to an embodiment of the present application. Figure 5 As shown, the device includes: a first processing unit 501, a first acquiring unit 502, a second processing unit 503 and a third processing unit 504.

[0084] Specifically, the first processing unit 501 is used to obtain multiple source data from multiple databases and determine the current version of each source data, wherein the multiple source data include business data used to calculate freight rate data;

[0085] A first acquisition unit 502 is used to acquire a source data update schedule and dynamic registration information of each computing node from a registration configuration center, wherein the source data update schedule includes an update time of each source data version, and the dynamic registration information includes version information of data in each computing node;

[0086] The second processing unit 503 is used to update the current version of each source data according to the source data update schedule and the dynamic registration information of each computing node to obtain the business data of the target version;

[0087] The third processing unit 504 is used to calculate and process the business data of the target version to obtain target freight rate data.

[0088] In summary, the data processing device provided in the embodiment of the present application obtains multiple source data from multiple databases through the first processing unit 501, and determines the current version of each source data, wherein the multiple source data include business data for calculating freight data; the first acquisition unit 502 obtains the source data update schedule and the dynamic registration information of each computing node from the registration configuration center, wherein the source data update schedule includes the update time of each source data version, and the dynamic registration information includes the version information of the data in each computing node; the second processing unit 503 updates the current version of each source data according to the source data update schedule and the dynamic registration information of each computing node to obtain the business data of the target version; the third processing unit 504 calculates and processes the business data of the target version to obtain the target freight data, which solves the problem in the related art that the amount of business data is too large, resulting in a long time to update the data, and then resulting in low efficiency in calculating the freight data using the updated business data. By splitting a set of databases into multiple databases, and using multiple databases to store different business data respectively, and then updating the version of the business data stored in each database according to the source data update schedule and the dynamic registration information of each computing node, and then using the updated business data to calculate the freight data, compared with the prior art of using a set of databases to store data and update the data, the time for updating data can be reduced, and when updating the data in multiple databases, the final data consistency of the entire set of databases can be maintained, thereby achieving the effect of improving the efficiency of calculating freight data.

[0089] Optionally, in the data processing device provided in the embodiment of the present application, the device also includes: a fourth processing unit, used to classify the multiple source data according to the type information of each source data before obtaining the multiple source data from the multiple databases, so as to obtain multiple types of source data; a first storage unit, used to store the multiple types of source data in the multiple databases based on the type of the source data stored in each database.

[0090] Optionally, in the data processing device provided in the embodiment of the present application, the second processing unit includes: a first processing sub-unit, used to update the current version of each source data according to the source data update schedule, and obtain the latest version information of each source data; a second processing sub-unit, used to update the latest version of each source data according to the dynamic registration information of each computing node, and obtain the target version information of each source data; a first construction sub-unit, used to construct a data version table based on the target version information of each source data, wherein the data version table is used to record the version information of the business data as the target version; a first determination sub-unit, used to obtain the business data of the target version based on the data version table.

[0091] Optionally, in the data processing device provided in the embodiment of the present application, the first processing sub-unit includes: a first acquisition module, used to obtain the update time of the current version of each source data, and obtain the update time of each source data version from the source data update schedule; a first judgment module, used to judge whether the update time of the current version of each source data is earlier than the update time of each source data version in the source data update schedule, and obtain a judgment result; a first processing module, used to update the current version of each source data based on the judgment result, and obtain the latest version information of each source data.

[0092] Optionally, in the data processing device provided in the embodiment of the present application, the first processing module includes: a first processing sub-module, which is used to indicate that there is no need to update the current version of each source data if the judgment result indicates that the update time of the current version of each source data is not earlier than the update time of the version of each source data in the source data update schedule; and a second processing sub-module, which is used to update the current version of each source data if the judgment result indicates that the update time of the current version of each source data is earlier than the update time of the version of each source data in the source data update schedule, so as to obtain the latest version information of each source data.

[0093] Optionally, in the data processing device provided in the embodiment of the present application, the second processing sub-unit includes: a second acquisition module, used to obtain the version information of the data in each computing node from the dynamic registration information of each computing node; a second judgment module, used to judge whether the version of the data in each computing node has been updated to the target version based on the version information of the data in each computing node; a second processing module, used to prohibit updating the latest version of each source data if the version of the data in at least one computing node has not been updated to the target version; a third processing module, used to update the latest version of each source data if the version of the data in each computing node has been updated to the target version, so as to obtain the target version information of each source data.

[0094] Optionally, in the data processing device provided in an embodiment of the present application, the first determination subunit includes: a third acquisition module, used to obtain version information of business data from a data version table; and a fourth acquisition module, used to obtain business data of a target version from multiple databases based on the version information of the business data.

[0095] The data processing device includes a processor and a memory. The above-mentioned first processing unit 501, first acquisition unit 502, second processing unit 503 and third processing unit 504 are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0096] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the efficiency of calculating freight rate data can be improved by adjusting kernel parameters.

[0097] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0098] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the data processing method is implemented.

[0099] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data processing method is executed when the program is run.

[0100] like Figure 6 As shown, an embodiment of the present invention provides an electronic device, the device includes a processor, a memory and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program: obtaining multiple source data from multiple databases, and determining the current version of each source data, wherein the multiple source data include business data for calculating freight rate data; obtaining a source data update schedule and dynamic registration information of each computing node from a registration configuration center, wherein the source data update schedule includes the update time of each source data version, and the dynamic registration information includes the version information of the data in each computing node; updating the current version of each source data according to the source data update schedule and the dynamic registration information of each computing node to obtain the business data of the target version; and calculating the business data of the target version to obtain the target freight rate data.

[0101] When the processor executes the program, the following steps are also implemented: before obtaining multiple source data from multiple databases, it also includes: classifying the multiple source data according to the type information of each source data to obtain multiple types of source data; based on the type of source data stored in each database, storing the multiple types of source data in multiple databases.

[0102] When the processor executes the program, the following steps are also implemented: according to the source data update schedule and the dynamic registration information of each computing node, the current version of each source data is updated to obtain the target version of the business data, including: according to the source data update schedule, the current version of each source data is updated to obtain the latest version information of each source data; according to the dynamic registration information of each computing node, the latest version of each source data is updated to obtain the target version information of each source data; based on the target version information of each source data, a data version table is constructed, wherein the data version table is used to record the version information of the business data as the target version; based on the data version table, the target version of the business data is obtained.

[0103] When the processor executes the program, the following steps are also implemented: according to the source data update schedule, the current version of each source data is updated to obtain the latest version information of each source data, including: obtaining the update time of the current version of each source data, and obtaining the update time of each source data version from the source data update schedule; judging whether the update time of the current version of each source data is earlier than the update time of each source data version in the source data update schedule, and obtaining the judgment result; according to the judgment result, the current version of each source data is updated to obtain the latest version information of each source data.

[0104] When the processor executes the program, the following steps are also implemented: based on the judgment result, the current version of each source data is updated to obtain the latest version information of each source data, including: if the judgment result indicates that the update time of the current version of each source data is not earlier than the update time of each source data version in the source data update schedule, it means that there is no need to update the current version of each source data; if the judgment result indicates that the update time of the current version of each source data is earlier than the update time of each source data version in the source data update schedule, then the current version of each source data is updated to obtain the latest version information of each source data.

[0105] When the processor executes the program, the following steps are also implemented: based on the dynamic registration information of each computing node, the latest version of each source data is updated to obtain the target version information of each source data, including: obtaining the version information of the data in each computing node from the dynamic registration information of each computing node; based on the version information of the data in each computing node, judging whether the version of the data in each computing node has been updated to the target version; if there is at least one computing node whose data version has not been updated to the target version, updating the latest version of each source data is prohibited; if the version of the data in each computing node has been updated to the target version, updating the latest version of each source data to obtain the target version information of each source data.

[0106] When the processor executes the program, the following steps are also implemented: based on the data version table, obtaining the target version of business data includes: obtaining version information of the business data from the data version table; and obtaining the target version of business data from multiple databases according to the version information of the business data.

[0107] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0108] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the following method steps: obtaining multiple source data from multiple databases and determining the current version of each source data, wherein the multiple source data include business data for calculating freight rate data; obtaining a source data update schedule and dynamic registration information of each computing node from a registration configuration center, wherein the source data update schedule includes the update time of each source data version, and the dynamic registration information includes the version information of the data in each computing node; updating the current version of each source data according to the source data update schedule and the dynamic registration information of each computing node to obtain the business data of the target version; and calculating the business data of the target version to obtain the target freight rate data.

[0109] When executed on a data processing device, it is also suitable for executing an initialization program having the following method steps: before obtaining multiple source data from multiple databases, it also includes: classifying the multiple source data according to the type information of each source data to obtain multiple types of source data; based on the type of source data stored in each database, storing the multiple types of source data in multiple databases.

[0110] When executed on a data processing device, it is also suitable for executing an initialization program having the following method steps: updating the current version of each source data according to the source data update schedule and the dynamic registration information of each computing node to obtain the target version of business data, including: updating the current version of each source data according to the source data update schedule to obtain the latest version information of each source data; updating the latest version of each source data according to the dynamic registration information of each computing node to obtain the target version information of each source data; constructing a data version table based on the target version information of each source data, wherein the data version table is used to record the version information of the business data as the target version; obtaining the target version of business data based on the data version table.

[0111] When executed on a data processing device, it is also suitable for executing an initialized program having the following method steps: updating the current version of each source data according to the source data update schedule to obtain the latest version information of each source data, including: obtaining the update time of the current version of each source data, and obtaining the update time of each source data version from the source data update schedule; judging whether the update time of the current version of each source data is earlier than the update time of each source data version in the source data update schedule to obtain a judgment result; based on the judgment result, updating the current version of each source data to obtain the latest version information of each source data.

[0112] When executed on a data processing device, it is also suitable for executing an initialized program having the following method steps: based on the judgment result, the current version of each source data is updated to obtain the latest version information of each source data, including: if the judgment result indicates that the update time of the current version of each source data is not earlier than the update time of each source data version in the source data update schedule, it means that there is no need to update the current version of each source data; if the judgment result indicates that the update time of the current version of each source data is earlier than the update time of each source data version in the source data update schedule, then the current version of each source data is updated to obtain the latest version information of each source data.

[0113] When executed on a data processing device, it is also suitable for executing an initialized program having the following method steps: based on the dynamic registration information of each computing node, the latest version of each source data is updated to obtain the target version information of each source data, including: obtaining the version information of the data in each computing node from the dynamic registration information of each computing node; based on the version information of the data in each computing node, judging whether the version of the data in each computing node has been updated to the target version; if there is at least one computing node whose data version has not been updated to the target version, updating the latest version of each source data is prohibited; if the data version in each computing node has been updated to the target version, updating the latest version of each source data to obtain the target version information of each source data.

[0114] When executed on a data processing device, it is also suitable for executing an initialization program having the following method steps: based on the data version table, obtaining the target version of business data includes: obtaining the version information of the business data from the data version table; and obtaining the target version of business data from multiple databases based on the version information of the business data.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0119] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0120] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0121] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0122] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0123] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0124] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire multiple source data from multiple databases, and determine a current version of each source data, wherein the multiple source data include business data used to calculate freight rate data; Obtaining a source data update schedule and dynamic registration information of each computing node from a registration configuration center, wherein the source data update schedule includes an update time of each source data version, and the dynamic registration information includes version information of data in each computing node; According to the source data update schedule and the dynamic registration information of each computing node, the current version of each source data is updated to obtain the target version of business data; The target version of the business data is calculated and processed to obtain target freight rate data.

2. The method according to claim 1, characterized in that Before acquiring the plurality of source data from the plurality of databases, the method further comprises: According to the type information of each source data, the plurality of source data are classified and processed to obtain multiple types of source data; The multiple types of source data are stored in the multiple databases based on the types of source data stored in each database.

3. The method according to claim 1, characterized in that According to the source data update schedule and the dynamic registration information of each computing node, the current version of each source data is updated to obtain the target version of business data including: According to the source data update schedule, the current version of each source data is updated to obtain the latest version information of each source data; According to the dynamic registration information of each computing node, the latest version of each source data is updated to obtain the target version information of each source data; Based on the target version information of each source data, a data version table is constructed, wherein the data version table is used to record the version information of the business data as the target version; Based on the data version table, the business data of the target version is obtained.

4. The method according to claim 3, characterized in that According to the source data update schedule, the current version of each source data is updated to obtain the latest version information of each source data, including: Obtaining the update time of the current version of each source data, and obtaining the update time of each source data version from the source data update schedule; Determine whether the update time of the current version of each source data is earlier than the update time of each version of the source data in the source data update schedule, and obtain a determination result; According to the judgment result, the current version of each source data is updated to obtain the latest version information of each source data.

5. The method according to claim 4, characterized in that According to the judgment result, the current version of each source data is updated to obtain the latest version information of each source data, including: If the judgment result indicates that the update time of the current version of each source data is not earlier than the update time of each source data version in the source data update schedule, it means that there is no need to update the current version of each source data; If the judgment result indicates that the update time of the current version of each source data is earlier than the update time of each source data version in the source data update schedule, the current version of each source data is updated to obtain the latest version information of each source data.

6. The method according to claim 3, characterized in that According to the dynamic registration information of each computing node, the latest version of each source data is updated, and the target version information of each source data is obtained, including: Obtaining version information of data in each computing node from dynamic registration information of each computing node; According to the version information of the data in each computing node, determine whether the version of the data in each computing node has been updated to the target version; If the version of the data in at least one computing node has not been updated to the target version, it is prohibited to update the latest version of each source data; If the version of the data in each computing node has been updated to the target version, the latest version of each source data is updated to obtain the target version information of each source data.

7. The method according to claim 3, characterized in that Based on the data version table, obtaining the business data of the target version includes: Acquire the version information of the business data from the data version table; The business data of the target version is acquired from the multiple databases according to the version information of the business data.

8. A data processing device, characterized in that: include: A first processing unit is used to obtain a plurality of source data from a plurality of databases and determine a current version of each source data, wherein the plurality of source data includes business data used to calculate freight rate data; A first acquisition unit is used to acquire a source data update schedule and dynamic registration information of each computing node from a registration configuration center, wherein the source data update schedule includes an update time of each source data version, and the dynamic registration information includes version information of data in each computing node; A second processing unit is used to update the current version of each source data according to the source data update schedule and the dynamic registration information of each computing node to obtain the business data of the target version; The third processing unit is used to calculate and process the business data of the target version to obtain target freight rate data.

9. A computer-readable storage medium, characterized in that: The storage medium stores a program, wherein the program executes the data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method described in any one of claims 1 to 7.

Images