Warehouse counting model timeliness improvement method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a warehouse counting model timeliness improvement method and device, electronic equipment and a computer readable storage medium. The method comprises the steps of obtaining waybill data to be updated; determining an original node process for updating the waybill data to be updated to the target waybill model, wherein the original node process comprises at least two original task nodes with an original dependency relationship; for each original task node, adjusting an original dependency relationship corresponding to the original task node according to a direct source of data on which the original task node depends to obtain a target node process; and updating the waybill data to be updated to the target waybill model according to the target node flow. The dependency relationship between the task nodes is adjusted through the direct source of the data on which the task nodes depend, and the execution priority of the task nodes is improved, so that the data output timeliness of the waybill model is improved, and sufficient time is reserved for subsequent business data consumption.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of computer technology, and specifically to a method, device, electronic device, and storage medium for improving the timeliness of a data warehouse model, wherein the storage medium includes a computer-readable storage medium. Background Art

[0002] As the business continues to develop, the concept of data-driven business is also being continuously strengthened. Core data analysis is increasingly affecting key decisions. The waybill model contains the waybill data of the entire network, serving more than 1,000 users and more than 4,000 downstream tasks, covering most business scenarios. For example, management morning meetings, operation briefings, business forecasts and other applications all rely on the timeliness of the waybill model.

[0003] The timeliness of the waybill model output fluctuates around 3:40 in the morning. After the downstream application tasks receive the waybill data, they still need to go through complex link processing, which takes a long time. Once the waybill model output is delayed and the buffer time reserved for remediation is insufficient, the timeliness of data for many downstream core applications will not be achieved, affecting business data such as management morning meetings, operation briefings, and business forecasts.

[0004] Therefore, it is necessary to optimize the waybill model scheduling tasks, improve the timeliness of the waybill model output, and reserve sufficient time for subsequent business use of the data. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device and computer-readable storage medium for improving the timeliness of a data warehouse model, which can improve the timeliness of the waybill model to produce data and reserve sufficient time for subsequent business use of the data.

[0006] The present application embodiment provides a method for improving the timeliness of a data warehouse model, including:

[0007] Get the waybill data to be updated;

[0008] Determine an original node process for updating the waybill data to be updated to the target waybill model, wherein the original node process includes at least two original task nodes having an original dependency relationship;

[0009] For each original task node, according to the direct source of the data on which the original task node depends, the original dependency relationship corresponding to the original task node is adjusted to obtain the target node process;

[0010] According to the target node process, the waybill data to be updated is updated to the target waybill model.

[0011] Accordingly, the embodiment of the present application also provides a data warehouse model time-efficiency improvement device, including:

[0012] The acquisition module is used to obtain the waybill data to be updated;

[0013] A determination module, used to determine an original node process for updating the waybill data to be updated to a target waybill model, wherein the original node process includes at least two original task nodes having an original dependency relationship;

[0014] An adjustment module, for adjusting, for each original task node, the original dependency relationship corresponding to the original task node according to the direct source of the data on which the original task node depends, to obtain a target node process;

[0015] An updating module is used to update the waybill data to be updated to the target waybill model according to the target node process.

[0016] Optionally, in some embodiments of the present application, the adjustment module includes:

[0017] A determination unit, configured to determine, for each original task node, an upstream task node on which the original task node depends according to a direct source of data on which the original task node depends;

[0018] An establishing unit, used to establish a direct dependency relationship between the original task node and the upstream task node;

[0019] An updating unit is used to update the original dependency relationship corresponding to the original task node according to the direct dependency relationship to obtain a target node process.

[0020] Optionally, in some embodiments of the present application, the updating unit includes:

[0021] A determination subunit, used to determine a transition task node between the original task node and the upstream task node from the original node process;

[0022] The deletion subunit is used to delete the transition task node and the original dependency relationship between the transition task node and the original task node and the upstream task node, respectively, to obtain a target node process.

[0023] In some embodiments of the present application, the updating unit includes:

[0024] An acquisition subunit, used to acquire the end execution time of the upstream task node and the start execution time of the original task node;

[0025] An adjusting subunit, configured to adjust a target start execution time of the original task node according to a time difference between the start execution time and the end execution time;

[0026] The updating subunit is used to update the original dependency relationship corresponding to the original task node according to the direct dependency relationship, and adjust the original task node to the target start execution time to obtain the target node process.

[0027] Among them, in some embodiments of the present application, the acquisition module includes:

[0028] The acquisition unit is used to acquire the waybill data to be updated from the distributed system through the distributed processing engine, and the distributed system acquires the waybill data to be updated from the source system in the form of binary logs.

[0029] Among them, in some embodiments of the present application, the waybill data to be updated includes incremental data to be updated for the first network, the target node process includes a historical data processing node, a waybill information completion node, a historical data acquisition node, a waybill data combination node and an incremental data insertion node, the historical data processing node, the waybill information completion node and the historical data acquisition node are executed in parallel, and the update module includes:

[0030] An information determination unit, used to determine the newly added waybill data and the waybill data to be updated in the incremental data to be updated;

[0031] A clearing unit, configured to clear the waybill data corresponding to the waybill data to be updated in the target waybill model through the historical data processing node according to the waybill number corresponding to the incremental data to be updated;

[0032] An incremental data acquisition unit, used to acquire historical waybill data through a historical data acquisition node;

[0033] A completion unit, configured to complete and associate the newly added waybill data with the waybill data to be updated through the waybill information completion node to obtain completed associated data;

[0034] A combining unit, configured to combine the supplemented associated data with the historical waybill data through the waybill data combining node to obtain combined waybill data;

[0035] An insertion unit is used to insert the node through the incremental data and insert the combined waybill data into the target waybill model according to the waybill number.

[0036] Among them, in some embodiments of the present application, the target node process also includes a full data acquisition node and an invalid data elimination node, the invalid data elimination node is a downstream task node of the historical data acquisition node, and the invalid data elimination node is executed in parallel with the waybill information completion node, and the device also includes a full data insertion module, and the full data insertion module includes:

[0037] A full data acquisition unit, configured to acquire full waybill data for the second network through the full data acquisition node;

[0038] A removal unit, configured to remove the waybill data of the non-first network contained in the historical waybill data acquired by the historical data acquisition node through an invalid data removal node, obtain the removed historical waybill data, and obtain the preliminary updated historical waybill data according to the removed historical waybill data and the full waybill data;

[0039] Combination unit, comprising:

[0040] The combining subunit is used to combine the completed associated data and the preliminarily updated historical waybill data through the waybill data combining node to obtain combined waybill data.

[0041] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the above-mentioned method for improving the timeliness of the data warehouse model are implemented.

[0042] In a fourth aspect, an embodiment of the present application further provides a storage medium, the storage medium comprising a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned method for improving the timeliness of a data warehouse model are implemented.

[0043] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0044] The embodiment of the present application obtains the waybill data to be updated, determines the original node process for updating the waybill data to be updated to the target waybill model, and the original node process includes at least two original task nodes with original dependencies. For each original task node, the original dependency corresponding to the original task node is adjusted according to the direct source of the data on which the original task node depends, and the target node process is obtained. According to the target node process, the waybill data to be updated is updated to the target waybill model. Among them, the dependency between task nodes is adjusted by the direct source of the data on which the task node depends, and the priority of task node execution is improved, thereby improving the timeliness of the waybill model, and reserving sufficient time for subsequent business data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 It is a scenario diagram of a method for improving the timeliness of a data warehouse model provided in an embodiment of the present application;

[0047] Figure 2 It is a flowchart of a method for improving the timeliness of a data warehouse model provided in an embodiment of the present application;

[0048] Figure 3 It is a process-oriented diagram of the original node process provided in the embodiment of the present application;

[0049] Figure 4 It is a first process guide diagram of the target node process provided by an embodiment of the present application;

[0050] Figure 5 It is a second process guide diagram of the target node process provided in an embodiment of the present application;

[0051] Figure 6 It is a structural schematic diagram of a data warehouse model aging improvement device provided in an embodiment of the present application;

[0052] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0054] The embodiments of the present application provide a method, device, electronic device and storage medium for improving the timeliness of a data warehouse model. Specifically, the embodiments of the present application provide a device for improving the timeliness of a data warehouse model suitable for an electronic device, wherein the electronic device includes a server, which can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The server can be directly or indirectly connected via wired or wireless communication.

[0055] See also Figure 1 , Figure 1 : is a schematic diagram of a scenario in which a server according to an embodiment of the present application executes the method for improving the timeliness of a data warehouse model, wherein the specific execution process of the server executing the method for improving the timeliness of a data warehouse model is as follows:

[0056] The server 10 obtains the waybill data to be updated from the database, and determines the original node process for updating the waybill data to be updated to the target waybill model, wherein the original node process includes at least two original task nodes with original dependency relationships, and for each original task node, according to the direct source of the data on which the original task node depends, the original dependency relationship corresponding to the original task node is adjusted to obtain the target node process, and according to the target node process, the waybill data to be updated is updated to the target waybill model.

[0057] Subsequently, the server 10 obtains corresponding waybill data from the target waybill data in response to the business data viewing request.

[0058] It is understandable that by adjusting the dependency between task nodes through the direct source of the data on which the task nodes depend, the priority of task node execution can be improved, thereby improving the timeliness of the waybill model and reserving sufficient time for subsequent business use of the data.

[0059] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0060] See also Figure 2 , Figure 2 This is a flow chart of a method for improving the timeliness of a data warehouse model provided in an embodiment of the present application. The specific process of the method for improving the timeliness of a data warehouse model is as follows:

[0061] 101. Get the waybill data to be updated.

[0062] It can be understood that the waybill data to be updated is the data that needs to be updated to the waybill model. By obtaining the waybill data to be updated, it is convenient to use the waybill data to be updated to update the waybill data in the waybill model, thereby ensuring the accuracy of the waybill data in the waybill model.

[0063] 102. Determine an original node process for updating the waybill data to be updated to a target waybill model, wherein the original node process includes at least two original task nodes having an original dependency relationship.

[0064] It can be understood that the original node process is the node process before adjustment. Based on this node process, the updated waybill data to be updated can also be updated to the waybill model. However, based on the embodiment of the present application, the original node process has the disadvantage of slow timeliness in outputting data from the waybill model. It is necessary to adjust the task nodes to optimize the update process of the waybill data to be updated.

[0065] 103. For each original task node, according to the direct source of the data on which the original task node depends, the original dependency relationship corresponding to the original task node is adjusted to obtain a target node process.

[0066] It is understandable that the execution of each task node often depends on the corresponding data. Therefore, in an embodiment of the present application, by adjusting the dependency relationship of the task nodes through the direct source of the data on which the task nodes depend, it is possible to adjust the execution priority of the task nodes. For example, the task nodes located downstream are adjusted to the upstream, thereby improving the execution timeliness of the task nodes, ensuring that the waybill data is quickly updated to the waybill model, and improving the data output timeliness of the waybill model.

[0067] Among them, the dependency of the task nodes is adjusted based on the direct source of the dependent data to ensure the accuracy of the adjusted dependency and ensure the effective execution of each task node.

[0068] 104. According to the target node process, update the waybill data to be updated to the target waybill model.

[0069] It is understandable that, through the node process after dependency adjustment, the updating of the waybill data to be updated in the waybill model can be accelerated, which helps to improve the timeliness of the waybill model and ensure the execution of downstream business.

[0070] It can be understood that the embodiment of the present application obtains the waybill data to be updated, determines the original node process for updating the waybill data to be updated to the target waybill model, and the original node process includes at least two original task nodes with original dependency relationships. For each original task node, the original dependency relationship corresponding to the original task node is adjusted according to the direct source of the data on which the original task node depends, and the target node process is obtained. According to the target node process, the waybill data to be updated is updated to the target waybill model, so as to adjust the dependency relationship between the task nodes through the direct source of the data on which the task nodes depend, improve the priority of task node execution, and then improve the timeliness of the waybill model, reserving sufficient time for subsequent business data use.

[0071] Optionally, in an embodiment of the present application, the upstream task node of the task node can be determined according to the direct source of the data on which the task node depends, and then a new dependency relationship is established according to the upstream task node, and the original node process is updated through the new dependency relationship to obtain the target node process, that is, optionally, in some embodiments of the present application, the step of "for each original task node, according to the direct source of the data on which the original task node depends, the original dependency relationship corresponding to the original task node is adjusted to obtain the target node process" includes:

[0072] For each original task node, determine the upstream task node that the original task node depends on according to the direct source of the data that the original task node depends on;

[0073] Establishing a direct dependency relationship between the original task node and the upstream task node;

[0074] The original dependency relationship corresponding to the original task node is updated according to the direct dependency relationship to obtain a target node process.

[0075] It can be understood that the direct source corresponds to a task node, and the data generated by the task node after execution can be directly used by the corresponding downstream task node. For example, for a certain original task node, the task node corresponding to the direct source of the data that the original task node depends on is used as the upstream task node.

[0076] Among them, determining the upstream task node according to the direct source of the data on which the original task node depends can improve the execution timeliness of the original task node and speed up the execution of the original task node, thereby avoiding the slow execution timeliness of the original task node due to too many intermediate task nodes that need to be waited for, resulting in a long time for the waybill data to be updated to the waybill model.

[0077] Accordingly, by establishing a direct dependency relationship between the original task node and the corresponding upstream task node, and updating the original node process according to the direct dependency relationship, the dependency relationship between the task nodes in the original node process is adjusted, and the construction of a new target node process is achieved.

[0078] Optionally, in an embodiment of the present application, corresponding elimination processing may be performed on the intermediate task nodes between the original task node and the upstream task node to avoid the execution of invalid task nodes during the execution of the node process, thereby further accelerating the timeliness of updating the waybill data to the waybill model. That is, optionally, in some embodiments of the present application, the step of "updating the original dependency corresponding to the original task node according to the direct dependency to obtain the target node process" includes:

[0079] Determine a transition task node between the original task node and the upstream task node from the original node process;

[0080] The transition task node is deleted, and the original dependency relationships between the transition task node and the original task node and the upstream task node are deleted, to obtain a target node process.

[0081] Among them, since the direct source of the data that the original task node depends on is the upstream task node, the direct source of the data that the original task node depends on is not the task node located between the original task node and the upstream task node. Therefore, this part of the task nodes can be eliminated.

[0082] For example, the task node between the original task node and the upstream task node is used as a transition task node. Correspondingly, since the original task node does not directly depend on the data of the transition task node, it can be understood that the role of the transition task node is the transit processing of data. In order to improve the efficiency of updating the waybill data to the waybill model, in an embodiment of the present application, these task nodes can be deleted to reduce the number of task nodes that the data update complies with, speed up the update of data to the waybill model, and thereby improve the timeliness of the waybill model.

[0083] Accordingly, in an embodiment of the present application, the direct source of the data that the original task node depends on may be more than one task node, that is, the original task node may correspond to multiple upstream task nodes. Accordingly, a direct dependency relationship may be established between the original task node and each upstream task node respectively.

[0084] For example, see Figure 3 , Figure 3It is a process-oriented diagram of an original node process provided by an embodiment of the present application, wherein the original node process includes an original task node a, an original task node b, an original task node c, an original task node d, and an original task node e, wherein the original task node a, the original task node b, and the original task node c are upstream task nodes of the original task node d, and the original task node d is an upstream task node of the original task node e;

[0085] Accordingly, by analyzing the direct source of the data that the original task node e depends on, assuming that the direct source of the data that the original task node e depends on is the original task node a, the original task node b and the original task node c, then the original task node d is a transition task node. Accordingly, the transition task node d can be eliminated, and the direct dependency relationship between the original task node e and the original task node a, the original task node b and the original task node c can be directly established, and the target node process including the original task node e and its corresponding upstream task nodes a, b and c is updated, as shown in FIG. Figure 4 As shown, Figure 4 It is the first process-oriented diagram of the target node process provided by an embodiment of the present application, which includes upstream task nodes a, b and c, and downstream task node e.

[0086] Optionally, in an embodiment of the present application, for an original task node, the time difference between the original task node and its corresponding upstream task node may also be calculated, and the execution time of the original task node may be adjusted based on the time difference, thereby accelerating the timeliness of updating the waybill data to the waybill model. That is, optionally, in some embodiments of the present application, the step of "updating the original dependency corresponding to the original task node according to the direct dependency to obtain the target node process" includes:

[0087] Obtaining the end execution time of the upstream task node and the start execution time of the original task node;

[0088] Adjusting the target start execution time of the original task node according to the time difference between the start execution time and the end execution time;

[0089] The original dependency relationship corresponding to the original task node is updated according to the direct dependency relationship, and the original task node is adjusted to the target start execution time to obtain the target node process.

[0090] It can be understood that in the embodiment of the present application, the task corresponding to the task node is an offline task. Accordingly, in order to balance the occupancy of system resources (such as memory and CPU) by the execution of the task, each task node corresponds to a corresponding execution time, including a start execution time and an end execution time, which is used to control the execution of each offline task, for example, to automatically update the waybill data offline, such as in the early morning hours.

[0091] Accordingly, for offline tasks, the execution time of the task can be adjusted to avoid the drawback that the downstream node cannot execute the corresponding task due to long waiting time for data. For example, in the embodiment of the present application, the target start execution time to be adjusted is determined by the difference between the end execution time of the upstream task node and the start execution time of the original task node, in order to speed up the execution of the entire node process by processing the original task node in advance. For example, when updating the waybill data, it is necessary to obtain the historical waybill data, and since the acquisition of the historical waybill data depends on the update summary of the last waybill data, which is often the data update result of the previous day, when the waybill data needs to be updated, the task node for obtaining the waybill data can be executed in advance.

[0092] Optionally, data usually needs to be obtained from the database and then updated to the waybill model. Therefore, you can try to replace the data acquisition engine to improve the efficiency of obtaining data from the database. That is, optionally, in some embodiments of the present application, the step of "obtaining the waybill data to be updated" includes:

[0093] The waybill data to be updated is obtained from the distributed system through a distributed processing engine, and the distributed system obtains the waybill data to be updated from the source system in the form of a binary log.

[0094] Among them, in the embodiment of the present application, the distributed processing engine refers to the flink program, the distributed system refers to kafka, and the binary log refers to binlog. Compared with the offline architecture in the related art, which mainly synchronizes data to the Hive source layer through ETL (jdbc) and synchronizes once a day, the source system data (database data) of the embodiment of the present application is pushed to kafka through binlog, and then processed in real time by the flink program and falls into the source layer (Hive). At the same time, the flink program realizes the function of filtering and removing dirty data, which improves the timeliness of the source layer (Hive) to obtain the waybill data.

[0095] Among them, the embodiment of the present application pushes the data in the source layer (Hive) to the waybill model through the target node process. By improving the timeliness of the source layer (Hive) obtaining the waybill data, the timeliness of the waybill model receiving the waybill data can be improved.

[0096] Optional, see Figure 5 , Figure 5 It is the second process-oriented diagram of the target node process provided by an embodiment of the present application, wherein the second process-oriented diagram shows that the target node process includes a first incremental data acquisition node A, a historical data processing node B, a waybill information completion node C, a historical data acquisition node D, a waybill data combination node E and an incremental data insertion node F, wherein the historical data processing node B, the waybill information completion node C and the historical data acquisition node D are executed in parallel, the incremental data insertion node F is the downstream task node of the historical data processing node B, the waybill information completion node C and the historical data acquisition node D, and the first incremental data acquisition node A is the upstream task node of the historical data processing node B and the waybill information completion node C.

[0097] Accordingly, in the embodiment of the present application, the waybill data to be updated includes incremental data to be updated for the first network, and the first incremental data acquisition node A is used to acquire the incremental data to be updated for the first network. Specifically, the step of "updating the waybill data to be updated to the target waybill model according to the target node process" includes:

[0098] Determine the newly added waybill data and the waybill data to be updated in the incremental data to be updated;

[0099] According to the waybill number corresponding to the incremental data to be updated, clearing the waybill data corresponding to the waybill data to be updated in the target waybill model through the historical data processing node;

[0100] Obtain historical waybill data through the historical data acquisition node;

[0101] The newly added waybill data and the waybill data to be updated are complemented and associated with each other through the waybill information complementing node to obtain complement associated data;

[0102] Combining the completed associated data with the historical waybill data through the waybill data combination node to obtain combined waybill data;

[0103] The completed waybill data is inserted into the target waybill model according to the waybill number through the incremental data insertion node.

[0104] It can be understood that, through the parallel execution of the historical data processing node B, the waybill information completion node C and the historical data acquisition node D, the historical data processing node B and the historical data acquisition node D are executed in advance, and the early execution of the historical data processing node B speeds up the clearing of the corresponding waybill data, making it convenient for the incremental data to be updated to be updated to the waybill model in an incremental update manner, and the early execution of the historical data acquisition node D facilitates the acceleration of the acquisition of historical waybill data, and facilitates the combination of the historical waybill data obtained by the historical data acquisition node D and the completed associated data after the waybill information completion node C is completed through the waybill data combination node E, thereby facilitating the incremental data insertion node F to insert the combined waybill data into the waybill model.

[0105] It should be noted that completing the association refers to the process of associating data through unique identifiers such as waybill numbers to form multiple complete data sets.

[0106] It can be understood that in an embodiment of the present application, before the waybill information completes node C, a fee type conversion node K can also be included, and the fee type conversion is performed based on the fee type conversion node K, for example, the currency in the waybill data is converted into a unified currency.

[0107] Among them, in an embodiment of the present application, the first incremental data acquisition node A is configured with two tasks. The first task A1 is used to obtain and operate the waybill, such as the waybill change information formed by the delivery of goods; the second task A2 is used to operate the waybill return slip, for example, some waybill information returned by the end, such as waybill receipt information, etc.

[0108] Optionally, the data warehouse model time-efficiency improvement method also includes updating the full amount of waybill data for the second largest network. For details, please refer to Figure 5 The target node process also includes a full data acquisition node G and an invalid data elimination node H. Specifically, the full data acquisition node G and the historical data acquisition node D are executed in parallel, the invalid data elimination node H is the downstream task node of the full data acquisition node G and the historical data acquisition node D, and the waybill data combination node E is the downstream task node of the historical data processing node B, the waybill information completion node C and the invalid data elimination node H.

[0109] Correspondingly, the method for improving the timeliness of the data warehouse model also includes:

[0110] Acquire the full waybill data for the second network through the full data acquisition node;

[0111] By using an invalid data elimination node, the waybill data not belonging to the first network included in the historical waybill data obtained by the historical data acquisition node is eliminated to obtain the eliminated historical waybill data, and the preliminary updated historical waybill data is obtained according to the eliminated historical waybill data and the full waybill data;

[0112] Correspondingly, the step of “combining the supplemented associated data with the historical waybill data through the waybill data combination node to obtain combined waybill data” includes:

[0113] The completed associated data and the preliminarily updated historical waybill data are combined through the waybill data combination node to obtain combined waybill data.

[0114] It can be understood that, through the invalid data elimination node H, the waybill data other than the first network contained in the historical waybill data is eliminated. After the elimination, the historical waybill data for the first network and the newly processed waybill data for the second network can be obtained. Subsequently, the newly added waybill information is supplemented into the preliminary updated historical waybill data obtained by the eliminated historical waybill data and the full waybill data through the waybill information completion node C, so as to obtain the total amount data (temporary table storage). Subsequently, according to the waybill number, the newly added waybill data is obtained from the total amount data and updated to the target waybill model, that is, updated to the target waybill model after the waybill data is cleared by the historical data processing node.

[0115] Correspondingly, in the embodiment of the present application, update nodes for other waybill data may also be configured, for example, see Figure 5 As shown, a value-added data acquisition node I for obtaining value-added data and a third network waybill data acquisition node J for obtaining waybill data of the third network are configured, wherein the value-added data acquisition node I is executed in parallel with the first incremental data acquisition node A and the third network waybill data acquisition node J, the value-added data acquisition node I is the upstream task node of the waybill data combination node E, and the third network waybill data acquisition node J is the upstream task node of the waybill information completion node C. Among them, the comprehensiveness and accuracy of the waybill data in the waybill model are improved by configuring the value-added data acquisition node I and the third network waybill data acquisition node J.

[0116] Among them, in an embodiment of the present application, the insured data acquisition node I, the third network waybill data acquisition node J and the full data acquisition node G can be configured as ETL-based task nodes to perform data acquisition based on ETL.

[0117] In order to facilitate the better implementation of the data warehouse model time-efficiency improvement method of the present application, the present application also provides a data warehouse model time-efficiency improvement device based on the above data warehouse model time-efficiency improvement method. The meaning of the nouns is the same as that in the above data warehouse model time-efficiency improvement method, and the specific implementation details can refer to the description in the method embodiment.

[0118] See also Figure 6 , Figure 6 : is a schematic diagram of the structure of the data warehouse model aging improvement device provided in an embodiment of the present application, wherein the data warehouse model aging improvement device can be specifically as follows:

[0119] The acquisition module 301 is used to acquire the waybill data to be updated;

[0120] A determination module 302 is used to determine an original node process for updating the waybill data to be updated to a target waybill model, wherein the original node process includes at least two original task nodes having an original dependency relationship;

[0121] The adjustment module 303 is used to adjust the original dependency relationship corresponding to each original task node according to the direct source of the data on which the original task node depends, so as to obtain a target node process;

[0122] The updating module 304 is used to update the waybill data to be updated to the target waybill model according to the target node process.

[0123] Optionally, in some embodiments of the present application, the adjustment module 303 includes:

[0124] A determination unit, configured to determine, for each original task node, an upstream task node on which the original task node depends according to a direct source of data on which the original task node depends;

[0125] An establishing unit, used to establish a direct dependency relationship between the original task node and the upstream task node;

[0126] An updating unit is used to update the original dependency relationship corresponding to the original task node according to the direct dependency relationship to obtain a target node process.

[0127] Optionally, in some embodiments of the present application, the updating unit includes:

[0128] A determination subunit, used to determine a transition task node between the original task node and the upstream task node from the original node process;

[0129] The deletion subunit is used to delete the transition task node and the original dependency relationship between the transition task node and the original task node and the upstream task node, respectively, to obtain a target node process.

[0130] In some embodiments of the present application, the updating unit includes:

[0131] An acquisition subunit, used to acquire the end execution time of the upstream task node and the start execution time of the original task node;

[0132] An adjusting subunit, configured to adjust a target start execution time of the original task node according to a time difference between the start execution time and the end execution time;

[0133] The updating subunit is used to update the original dependency relationship corresponding to the original task node according to the direct dependency relationship, and adjust the original task node to the target start execution time to obtain the target node process.

[0134] In some embodiments of the present application, the acquisition module 301 includes:

[0135] The acquisition unit is used to acquire the waybill data to be updated from the distributed system through the distributed processing engine, and the distributed system acquires the waybill data to be updated from the source system in the form of binary logs.

[0136] Among them, in some embodiments of the present application, the waybill data to be updated includes incremental data to be updated for the first network, the target node process includes a historical data processing node, a waybill information completion node, a historical data acquisition node, a waybill data combination node and an incremental data insertion node, the historical data processing node, the waybill information completion node and the historical data acquisition node are executed in parallel, and the update module includes:

[0137] An information determination unit, used to determine the newly added waybill data and the waybill data to be updated in the incremental data to be updated;

[0138] A clearing unit, configured to clear the waybill data corresponding to the waybill data to be updated in the target waybill model through the historical data processing node according to the waybill number corresponding to the incremental data to be updated;

[0139] An incremental data acquisition unit, used to acquire historical waybill data through a historical data acquisition node;

[0140] A completion unit, used to complete and associate the newly added waybill data through the waybill information completion node to obtain completed associated data;

[0141] A combining unit, configured to combine the supplemented associated data with the historical waybill data through the waybill data combining node to obtain combined waybill data;

[0142] An insertion unit is used to insert the node through the incremental data and insert the combined waybill data into the target waybill model according to the waybill number.

[0143] Among them, in some embodiments of the present application, the target node process also includes a full data acquisition node and an invalid data elimination node, the invalid data elimination node is a downstream task node of the historical data acquisition node, and the invalid data elimination node is executed in parallel with the waybill information completion node, and the device also includes a full data insertion module, and the full data insertion module includes:

[0144] A full data acquisition unit, configured to acquire full waybill data for the second network through the full data acquisition node;

[0145] A removal unit, configured to remove the waybill data of the non-first network contained in the historical waybill data acquired by the historical data acquisition node through an invalid data removal node, obtain the removed historical waybill data, and obtain the preliminary updated historical waybill data according to the removed historical waybill data and the full waybill data;

[0146] Combination unit, comprising:

[0147] The combining subunit is used to combine the completed associated data and the preliminarily updated historical waybill data through the waybill data combining node to obtain combined waybill data.

[0148] In the embodiment of the present application, the acquisition module 301 first acquires the waybill data to be updated, and then the determination module 302 determines the original node process of updating the waybill data to be updated to the target waybill model, and the original node process includes at least two original task nodes with original dependency relationships. Subsequently, the adjustment module 303 adjusts the original dependency relationship corresponding to each original task node according to the direct source of the data on which the original task node depends, and obtains the target node process. Then, the update module 304 is used to update the waybill data to be updated to the target waybill model according to the target node process.

[0149] It is understandable that by adjusting the dependency between task nodes through the direct source of the data on which the task nodes depend, the priority of task node execution can be improved, thereby improving the timeliness of the waybill model and reserving sufficient time for subsequent business use of the data.

[0150] In addition, the present application also provides an electronic device, such as Figure 7As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0151] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0152] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0153] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0154] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0155] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0156] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 will run the application stored in the memory 402, thereby implementing any step in the data warehouse model time-efficiency improvement method provided in the embodiment of the present application.

[0157] The embodiment of the present application obtains the waybill data to be updated, determines the original node process for updating the waybill data to be updated to the target waybill model, the original node process includes at least two original task nodes with original dependency relationships, for each original task node, according to the direct source of the data on which the original task node depends, the original dependency relationship corresponding to the original task node is adjusted to obtain the target node process, and according to the target node process, the waybill data to be updated is updated to the target waybill model.

[0158] Among them, the dependency relationship between task nodes is adjusted through the direct source of the data on which the task nodes depend, the priority of task node execution is improved, and then the timeliness of the waybill model is improved, reserving sufficient time for subsequent business use of the data.

[0159] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0160] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer storage medium and loaded and executed by a processor.

[0161] To this end, the present application provides a storage medium, which includes a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any one of the data warehouse model timeliness improvement methods provided in the present application.

[0162] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0163] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0164] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the data warehouse model timeliness improvement methods provided in the present application, the beneficial effects that can be achieved by any of the data warehouse model timeliness improvement methods provided in the present application can be achieved. Please see the previous embodiments for details, which will not be repeated here.

[0165] The above is a detailed introduction to a method, device, electronic device and computer-readable storage medium for improving the timeliness of a data warehouse model provided in the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0166] It should be noted that in the specific implementation of this application, related data such as the user's waybill data is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A method for improving the timeliness of a data warehouse model, characterized in that: include: Get the waybill data to be updated; Determine an original node process for updating the waybill data to be updated to the target waybill model, wherein the original node process includes at least two original task nodes having an original dependency relationship; For each original task node, according to the direct source of the data on which the original task node depends, the original dependency relationship corresponding to the original task node is adjusted to obtain the target node process; According to the target node process, the waybill data to be updated is updated to the target waybill model.

2. The method according to claim 1, characterized in that For each original task node, adjusting the original dependency relationship corresponding to the original task node according to the direct source of the data on which the original task node depends to obtain the target node process includes: For each original task node, determine the upstream task node that the original task node depends on according to the direct source of the data that the original task node depends on; Establishing a direct dependency relationship between the original task node and the upstream task node; The original dependency relationship corresponding to the original task node is updated according to the direct dependency relationship to obtain a target node process.

3. The method according to claim 2, characterized in that The updating of the original dependency relationship corresponding to the original task node according to the direct dependency relationship to obtain the target node process includes: Determine a transition task node between the original task node and the upstream task node from the original node process; The transition task node is deleted, and the original dependency relationships between the transition task node and the original task node and the upstream task node are deleted, to obtain a target node process.

4. The method according to claim 2, characterized in that: The updating of the original dependency relationship corresponding to the original task node according to the direct dependency relationship to obtain the target node process includes: Obtaining the end execution time of the upstream task node and the start execution time of the original task node; Adjusting the target start execution time of the original task node according to the time difference between the start execution time and the end execution time; The original dependency relationship corresponding to the original task node is updated according to the direct dependency relationship, and the original task node is adjusted to the target start execution time to obtain the target node process.

5. The method according to claim 1, characterized in that The obtaining of the waybill data to be updated includes: The waybill data to be updated is obtained from the distributed system through a distributed processing engine, and the distributed system obtains the waybill data to be updated from the source system in the form of a binary log.

6. The method according to claim 1, characterized in that The waybill data to be updated includes incremental data to be updated for the first network, the target node process includes a historical data processing node, a waybill information completion node, a historical data acquisition node, a waybill data combination node and an incremental data insertion node, and the historical data processing node, the waybill information completion node and the historical data acquisition node are executed in parallel; The updating of the to-be-updated waybill data to the target waybill model according to the target node process includes: Determine the newly added waybill data and the waybill data to be updated in the incremental data to be updated; According to the waybill number corresponding to the incremental data to be updated, clearing the waybill data corresponding to the waybill data to be updated in the target waybill model through the historical data processing node; Obtain historical waybill data through the historical data acquisition node; The newly added waybill data and the waybill data to be updated are complemented and associated with each other through the waybill information complementing node to obtain complement associated data; Combining the completed associated data with the historical waybill data through the waybill data combination node to obtain combined waybill data; The combined waybill data is inserted into the target waybill model according to the waybill number through the incremental data insertion node.

7. The method according to claim 6, characterized in that The target node process also includes a full data acquisition node and an invalid data elimination node, the invalid data elimination node is a downstream task node of the historical data acquisition node, and the invalid data elimination node is executed in parallel with the waybill information completion node, and the method also includes: Acquire the full waybill data for the second network through the full data acquisition node; By using an invalid data elimination node, the waybill data not belonging to the first network included in the historical waybill data obtained by the historical data acquisition node is eliminated to obtain the eliminated historical waybill data, and the preliminary updated historical waybill data is obtained by combining the eliminated historical waybill data with the full waybill data; The combining of the supplemented associated data with the historical waybill data through the waybill data combining node to obtain combined waybill data includes: The completed associated data and the preliminarily updated historical waybill data are combined through the waybill data combination node to obtain combined waybill data.

8. A device for improving the timeliness of a data warehouse model, characterized in that: include: The acquisition module is used to obtain the waybill data to be updated; A determination module, used to determine an original node process for updating the waybill data to be updated to a target waybill model, wherein the original node process includes at least two original task nodes having an original dependency relationship; An adjustment module, for adjusting, for each original task node, the original dependency relationship corresponding to the original task node according to the direct source of the data on which the original task node depends, to obtain a target node process; An updating module is used to update the waybill data to be updated to the target waybill model according to the target node process.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the method for improving the timeliness of a data warehouse model as described in any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the method for improving the timeliness of the data warehouse model as described in any one of claims 1 to 7 are implemented.

Images