Data processing method, apparatus, device, storage medium, and program product

Abstract

The application provides a data processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: for any target data model in a plurality of data models of a data warehouse, determining a first cost of the target data model, wherein the first cost comprises a calculation cost generated in a process of generating the target data model according to at least one first data model, and a storage cost of storing the target data model, the at least one first data model being a data model in the plurality of data models for generating the target data model; processing actual input costs of the at least one first data model respectively to obtain a second cost of the target data model; and determining an actual input cost of the target data model according to the first cost of the target data model and the second cost of the target data model. The data model cost can be determined more accurately, so that the data model can be evaluated more accurately.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] A data model is a model built during the data warehouse construction process, based on statistical analysis and data mining of raw business data. It is designed to support business analysis and decision-making in various business scenarios. A data model typically consists of one or more tables.

[0003] Data layering refers to a logical or physical layering process in data warehouse construction. Generally, a data warehouse can be divided into four layers: Operation Data Store (ODS), Data Warehouse Details (DWD), Data Warehouse Service (DWS), and Application Data Service (ADS). Each layer can include one or more data models.

[0004] When building the data model for each layer of a data warehouse, it is necessary to process the data model of the layer above it or the original business data to obtain the data model for that layer. When calculating the cost of the data model for that layer, the computational cost is often evaluated using the time consumed by the Central Processing Unit (CPU) and memory in the above processing, and the storage cost is evaluated using the disk capacity consumed to store the data model. Then, the sum of the above computational cost and storage cost is determined as the total cost of the model.

[0005] However, since building the data model above this data model requires investment, and the cost of the data model above was not considered in the process of determining the total cost, the method for determining the cost of the data model is inaccurate. Summary of the Invention

[0006] This application provides a data processing method, apparatus, device, storage medium, and program product to more accurately determine the cost of a data model, thereby enabling a more accurate evaluation of the data model.

[0007] In a first aspect, this application provides a data processing method, which includes: determining a first cost of a target data model for any target data model among multiple data models in a data warehouse, wherein the first cost includes: computational costs incurred in generating the target data model based on at least one first data model, and storage costs for storing the target data model, wherein at least one first data model is a data model used to generate the target data model among multiple data models; processing the actual input costs of each of the at least one first data model to obtain a second cost of the target data model; and determining the actual input cost of the target data model based on the first cost and the second cost of the target data model.

[0008] Optionally, the above-mentioned processing of the actual input cost of each of the at least one first data model to obtain the second cost of the target data model includes: for any one of the at least one first data model, identifying at least one downstream data model among multiple data models that has a data lineage with the first data model, wherein the at least one downstream data model includes the target data model; processing the actual input cost of the first data model according to the at least one downstream data model to obtain a third cost; and summing the third costs corresponding to each of the at least one first data model to obtain the second cost of the target data model.

[0009] Optionally, the above-mentioned processing of the actual input cost of the first data model based on at least one downstream data model to obtain the third cost includes: processing the actual input cost of the first data model based on the number of downstream data models included in the at least one downstream data model to obtain the third cost.

[0010] Optionally, the above-mentioned processing of the actual input cost of the first data model based on the number of downstream data models included in at least one downstream data model to obtain a third cost includes: determining the number of downstream data models included in at least one downstream data model to obtain a first value; summing the first value and a preset value to obtain a second value; and calculating the ratio of the actual input cost of the first data model to the second value to obtain the third cost.

[0011] Optionally, the above-mentioned processing of the actual input cost of the first data model based on at least one downstream data model to obtain a third cost includes: for any one of the at least one downstream data models, determining the amount of data in the first data model used when generating the downstream data model based on the first data model; determining a third value corresponding to the target data model based on the data amount corresponding to each of the at least one downstream data model; and processing the actual input cost of the first data model based on the third value corresponding to the target data model to obtain the third cost.

[0012] Optionally, determining the third value corresponding to the target data model based on the data volume corresponding to at least one downstream data model includes: normalizing the data volume corresponding to each of the at least one downstream data model; determining the normalized result of the data volume corresponding to the target data model as the third value corresponding to the target data model; and processing the actual input cost of the first data model based on the third value corresponding to the target data model to obtain the third cost, including: calculating the product of the actual input cost of the first data model and the third value corresponding to the target data model to obtain the third cost.

[0013] Optionally, before determining the first cost of any target data model among multiple data models in the data warehouse, the method further includes: generating a directed graph structure of multiple data models, wherein multiple nodes of the directed graph structure correspond one-to-one with multiple data models, and the directed edges of the directed graph structure are represented as nodes pointing from the upstream data model in two data models with data lineage to the downstream data model in the other two data models; performing a topological sort on the directed graph structure to obtain a sorting result; determining the first cost of any target data model among multiple data models in the data warehouse includes: calculating the first cost of the data model corresponding to any target node in the sorting result; processing the actual input cost of each of the at least one first data model to obtain the second cost of the target data model includes: traversing the directed graph structure to determine at least one upstream node corresponding to the target node; processing the actual input cost of each of the at least one data model corresponding to the at least one upstream node to obtain the second cost of the data model corresponding to the target node.

[0014] Secondly, this application provides a data processing apparatus, comprising: a first determining module, a processing module, and a second determining module, wherein the first determining module is used to determine a first cost of a target data model for any target data model among multiple data models in a data warehouse, wherein the first cost includes: the computational cost incurred in generating the target data model based on at least one first data model, and the storage cost of storing the target data model, wherein at least one first data model is a data model used to generate the target data model among multiple data models; the processing module is used to process the actual input cost of each of the at least one first data model to obtain a second cost of the target data model; and the second determining module is used to determine the actual input cost of the target data model based on the first cost and the second cost of the target data model.

[0015] Optionally, the processing module is specifically used for: for any one of the at least one first data models, determining at least one downstream data model among multiple data models that has a data lineage with the first data model, wherein the at least one downstream data model includes the target data model; processing the actual input cost of the first data model according to the at least one downstream data model to obtain a third cost; and summing the third costs corresponding to each of the at least one first data model to obtain a second cost of the target data model.

[0016] Optionally, the processing module is specifically used to: process the actual input cost of the first data model based on the number of downstream data models included in at least one downstream data model, to obtain the third cost.

[0017] Optionally, the processing module is specifically used to: determine the number of downstream data models included in at least one downstream data model to obtain a first value; sum the first value and a preset value to obtain a second value; and calculate the ratio of the actual input cost of the first data model to the second value to obtain a third cost.

[0018] Optionally, the processing module is specifically used for: determining, for any one of the at least one downstream data models, the amount of data used in the first data model when generating the downstream data model based on the first data model; determining the third value corresponding to the target data model based on the data amount corresponding to each of the at least one downstream data model; and processing the actual input cost of the first data model based on the third value corresponding to the target data model to obtain the third cost.

[0019] Optionally, the processing module is specifically used to: normalize the data volume corresponding to at least one downstream data model; determine the normalized result of the data volume corresponding to the target data model as the third value corresponding to the target data model; and calculate the product of the actual input cost of the first data model and the third value corresponding to the target data model to obtain the third cost.

[0020] Optionally, the data processing device further includes: a generation module and a sorting module, wherein the generation module is used to generate a directed graph structure of multiple data models, wherein multiple nodes of the directed graph structure correspond one-to-one with multiple data models, and the directed edges of the directed graph structure are represented as nodes corresponding to upstream data models in two data models that have a data lineage relationship, pointing to nodes corresponding to downstream data models in the two data models; the sorting module is used to perform topological sorting on the directed graph structure to obtain a sorting result; the first determination module is specifically used to: calculate the first cost of the data model corresponding to any target node in the sorting result; the processing module is specifically used to: traverse the directed graph structure to determine at least one upstream node corresponding to the target node; process the actual input cost of each of the at least one data model corresponding to the at least one upstream node to obtain the second cost of the data model corresponding to the target node.

[0021] Thirdly, this application provides an electronic device, including: a processor and a memory, the memory for storing a computer program, and the processor for calling and running the computer program stored in the memory to perform the methods as described in the first aspect or its various implementations.

[0022] Fourthly, this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0023] Fifthly, this application provides a computer program product including computer program instructions that cause a computer to perform the methods as described in the first aspect or its various implementations.

[0024] Sixthly, this application provides a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0025] According to the technical solution of this application, for any target data model among multiple data models in a data warehouse, the electronic device can first determine the first cost of the target data model. The first cost includes: the computational cost incurred in generating the target data model from at least one first data model, and the storage cost of storing the target data model. The at least one first data model is the data model used to generate the target data model among multiple data models. Next, the electronic device can process the actual input cost of each of the at least one first data model to obtain the second cost of the target data model. Finally, the electronic device can determine the actual input cost of the target data model based on the first cost and the second cost. In the above process, when determining the actual input cost of the target data model, the electronic device considers not only the computational cost incurred in generating the target data model and the storage cost of storing the target data model, but also the actual input cost of the data model used to generate the target data model. Therefore, the technical solution of this application can more accurately determine the data model cost, thereby enabling a more accurate evaluation of the data model. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0028] Figure 2 A schematic diagram illustrating a data processing method provided in an embodiment of this application;

[0029] Figure 3 A schematic diagram of a data processing apparatus 300 provided in an embodiment of this application;

[0030] Figure 4 This is a schematic block diagram of an electronic device 400 provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0034] As mentioned above, since building the upper-level data model of the target data model requires investment, and the existing technology does not consider the cost of the upper-level data model when determining the total cost of the target data model, the existing technology has the problem of inaccurate determination of data model cost.

[0035] To address the aforementioned technical problems, the inventive concept of this application is that, when determining the actual investment cost of a target data model, electronic devices consider not only the computational costs incurred in generating the target data model and the storage costs of storing the target data model, but also the actual investment cost of the data model used to generate the target data model. Therefore, the technical solution of this application can more accurately determine the cost of the data model, thereby enabling a more accurate evaluation of the data model.

[0036] It should be understood that the technical solution of this application can be applied to the following scenarios, but is not limited to:

[0037] In some possible implementations, the technical solution of this application can be applied to scenarios for evaluating data models.

[0038] For example, after constructing the data models for each layer of the data warehouse, the electronic device can determine the input cost of each data model. It can then evaluate the data models based on these costs. For instance, the electronic device can assess the resources consumed and the value generated in producing the data models, such as the storage and computing resources consumed. Based on the evaluation results, it can perform targeted optimizations on the data models. For example, the electronic device can optimize the generation and storage methods for data models with higher input costs to reduce the input costs and increase the value of the data models. The input costs of the data models may include, but are not limited to, the first cost, the second cost, and the actual input cost in this embodiment.

[0039] For example, the device used to build the data model for each layer of the data warehouse and the electronic device in this application may be the same device or different devices, and this application does not limit this.

[0040] It should be noted that the solutions provided in this application can be executed by any electronic device with data processing capabilities, and this application does not impose specific limitations on such electronic devices. For example, the electronic device can be a server. The server can be a single server, a server cluster consisting of multiple servers, or a cloud platform control center, but is not limited thereto. Alternatively, the electronic device can be a terminal device. The terminal device can be a tablet computer, a laptop computer, or a desktop computer, etc. Alternatively, the electronic device can be implemented as a combination of a server and a terminal device, wherein the server and the terminal device can communicate wirelessly or via wired means.

[0041] After introducing the application scenarios of the embodiments of this application, the technical solution of this application will be described in detail below:

[0042] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. This method can be executed by an electronic device in the aforementioned application scenario, but is not limited thereto. Figure 1 As shown, the method may include the following steps:

[0043] S110: For any target data model among multiple data models in a data warehouse, determine a first cost of the target data model, wherein the first cost includes: the computational cost incurred in generating the target data model based on at least one first data model, and the storage cost of storing the target data model, wherein at least one first data model is a data model used to generate the target data model among multiple data models;

[0044] S120: Process the actual input cost of each of at least one first data model to obtain the second cost of the target data model;

[0045] S130: Determine the actual input cost of the target data model based on the first cost and the second cost of the target data model.

[0046] Understandingly, data lineage, also known as kinship, refers to an upstream and downstream relationship in the data computation and production process. Generally, if a data model (hereinafter referred to as the input model) is the input to another data model (hereinafter referred to as the output model), that is, if the output model can be generated from the input model, then there is a lineage relationship between the input and output models. The input model is called the upstream model of the output model, and the output model is called the downstream model of the input model. For example, suppose a data warehouse includes: Data Model 1 and Data Model 2, and Data Model 2 is generated from Data Model 1. Then Data Model 1 and Data Model 2 have a lineage relationship, and Data Model 1 is the upstream data model of Data Model 2, while Data Model 2 is the downstream data model of Data Model 1.

[0047] In S110, since at least one first data model is a data model used to generate the target data model among multiple data models, the target data model and at least one first data model have a data lineage relationship. At least one first data model is an upstream data model of the target data model, and the target data model is a downstream data model of at least one first data model.

[0048] It should be noted that the electronic device can determine the actual investment cost of at least one first data model according to the method for determining the actual investment cost of the target data model in the embodiments of this application, which will not be elaborated upon in this application. If, when building a certain data model in a data warehouse, it is only necessary to generate the data model based on the original business data, and it is not necessary to generate the data model based on other data models, then the electronic device can determine the actual investment cost of the data model as the computing cost and the storage cost of storing the data model generated in the process of generating the data model based on the original business data.

[0049] In some possible implementations, the electronic device can first determine the CPU time and memory time consumed when generating the target data model based on at least one first data model, and determine the disk capacity consumed when storing the target data model. Then, the electronic device can determine the computation cost based on the CPU time and memory time consumed, and determine the storage cost based on the disk capacity consumed. The sum of the computation cost and the storage cost is determined as the first cost of the target data model.

[0050] For example, assuming that when generating the target data model based on at least one first data model, the time consumed by memory is 10 seconds, the time consumed by CPU is 15 seconds, and the disk capacity consumed when storing the target data model is 20 bytes, then the electronic device can determine that the computation cost is 10 + 15 = 25 seconds, the storage cost is 20 bytes, and the first cost is 25 + 20 = 45.

[0051] It should be noted that this application does not impose any restrictions on the determination method or specific content of computing costs, storage costs, primary costs, etc.

[0052] In some possible implementations, the above-mentioned S120 may include: for any one of the at least one first data models, determining at least one downstream data model among a plurality of data models that has a data lineage with the first data model, wherein the at least one downstream data model includes the target data model; processing the actual input cost of the first data model according to the at least one downstream data model to obtain a third cost; and summing the third costs corresponding to each of the at least one first data model to obtain a second cost of the target data model.

[0053] For example, suppose a data warehouse includes data model 1, data model 2, data model 3, and data model 4, wherein data model 2, data model 3, and data model 4 are all generated based on data model 1. Then, at least one downstream data model that has a data lineage relationship with data model 1 includes: data model 2, data model 3, and data model 4. In this embodiment, if the target data model is data model 2, then the first data model is data model 1, and at least one downstream data model that has a data lineage relationship with the first data model includes: data model 2, data model 3, and data model 4.

[0054] In some feasible implementations, the electronic device may process the actual input cost of the first data model based on at least one downstream data model to obtain a third cost, but is not limited to this:

[0055] Method 1: Electronic devices can process the actual input cost of the first data model based on the number of downstream data models included in at least one downstream data model to obtain the third cost.

[0056] For example, the electronic device can first determine the number of downstream data models included in at least one downstream data model to obtain a first value. Then, the electronic device can sum the first value and a preset value to obtain a second value. Next, the electronic device can calculate the ratio of the actual input cost of the first data model to the second value to obtain a third cost. The preset value can be any value; for example, it can be 0, 0.1, 1, -0.1, or -1, and this application does not impose any restrictions on it.

[0057] For example, in conjunction with the above embodiments, assuming the target data model is data model 2, the first data model is data model 1, the actual investment cost of data model 1 is 40, and at least one downstream data model that has a data lineage relationship with the first data model includes: data model 2, data model 3, and data model 4, with a preset value of 1, then the electronic device can determine that the number of downstream data models included in at least one downstream data model is 3, and obtain the first value as 3. Then, the electronic device can sum the first value and the preset value to obtain the second value as 3 + 1 = 4. Then, the electronic device can calculate the ratio of the actual investment cost of data model 1 to the second value to obtain the third cost as 40 ÷ 4 = 10. If, among the multiple data models in the data warehouse, only data model 1 is used to generate data model 2, meaning at least one first data model includes only data model 1, then the electronic device can directly determine the third cost of data model 1 as the second cost of data model 2. If, among the multiple data models in the data warehouse, data model 1 and data model 5 are used to generate data model 2, meaning at least one first data model includes both data model 1 and data model 5, the electronic device can determine the third cost of data model 5 using a method similar to that used to determine the third cost of data model 1. The sum of the third costs of data model 1 and data model 5 is then determined as the second cost of data model 2. For example, assuming the third cost of data model 5 is 9, the electronic device can determine the second cost of data model 2 as 10 + 9 = 19.

[0058] Method 2: For any one of the at least one downstream data models, the electronic device can first determine the amount of data in the first data model used when generating the downstream data model based on the first data model. Then, based on the amount of data corresponding to each of the at least one downstream data model, the electronic device can determine the third value corresponding to the target data model. Next, the electronic device can process the actual input cost of the first data model based on the third value corresponding to the target data model to obtain the third cost.

[0059] For example, the electronic device can first normalize the data volume corresponding to at least one downstream data model, and determine the normalized result of the data volume corresponding to the target data model as the third value corresponding to the target data model. Then, the electronic device can calculate the product of the actual input cost of the first data model and the third value corresponding to the target data model to obtain the third cost.

[0060] For example, the amount of data used in the first data model can be a percentage or a specific value. For instance, assuming the first data model has a data size of 20 bytes, and 15 bytes of data from the first data model are used when generating the target data model, the electronic device can determine that the target data model uses 15 bytes of data from the first data model, or determine that the target data model uses 15 / 20 = 75% of the data from the first data model. That is, the data size corresponding to the target data model is 15% or 75%.

[0061] For example, in conjunction with the above embodiments, assuming the target data model is data model 2, the first data model is data model 1, the actual investment cost of data model 1 is 40, the data size of data model 1 is 20 bytes, and at least one downstream data model with a data lineage relationship to the first data model includes: data model 2, data model 3, and data model 4. When generating data model 2, data model 3, and data model 4 based on data model 1, the data amounts used in the first data model are 40%, 10%, and 50%, respectively. That is, data model 2, data model 3, and data model 4 are all related to the first data model. The data amounts corresponding to each of the four models are 40%, 10%, and 50%, respectively. The electronic device can normalize the data amounts corresponding to each of the three models, obtaining normalization results of 0.4, 0.1, and 0.5, respectively. The electronic device can then determine that the third value corresponding to the model 2 is 0.4. Then, the electronic device can calculate the product of the actual input cost of the model 1 and the third value corresponding to the model 2, obtaining the third cost as 40 × 0.4 = 10. If, among the multiple data models in the data warehouse, only data model 1 is used to generate data model 2, meaning at least one first data model includes only data model 1, then the electronic device can directly determine the third cost of data model 1 as the second cost of data model 2, i.e., determine the second cost of data model 2 as 10. If, among the multiple data models in the data warehouse, data model 1 and data model 5 are used to generate data model 2, meaning at least one first data model includes both data model 1 and data model 5, the electronic device can determine the third cost of data model 5 using a method similar to that used to determine the third cost of data model 1. The sum of the third costs of data model 1 and data model 5 is then determined as the second cost of data model 2. For example, assuming the third cost of data model 5 is 9, the electronic device can determine the second cost of data model 2 as 10 + 9 = 19.

[0062] In some feasible methods, when determining the actual investment cost of the target data model based on the first cost and the second cost of the target data model, the electronic device can directly determine the sum of the first cost and the second cost as the actual investment cost, or it can first determine the weight coefficients of the first cost and the second cost, and then perform a weighted summation of the first cost and the second cost based on the weight coefficients, and determine the result of the weighted summation as the actual investment cost.

[0063] For example, assuming the first cost of the target data model is 45 and the second cost of the target data model is 19, the electronic device can determine the actual investment cost of the target data model as 45 + 19 = 64; or, if the weight coefficients of the first cost and the second cost of the target data model are 0.7 and 0.3 respectively, the electronic device can determine the actual investment cost of the target data model as 45 × 0.7 + 19 × 0.3 = 37.2.

[0064] In some implementations, the electronic device can determine the input-output ratio of the target data model based on its third cost and actual input cost, thereby evaluating the value of the target data model. For example, assuming the actual input cost of data model 2 is 64, and data model 2 is an upstream data model of data model 6, the electronic device can determine, through the method in the above embodiments, that the cost of data model 2 consumed when generating data model 6 from data model 2 is 24, i.e., the third cost of data model 2 is 24. Therefore, the input-output ratio of the target data model can be determined as 64 / 24. It should be noted that if the target data model has multiple downstream data models, the electronic device can determine the third cost of the target data model by the average, summation, maximum, or minimum cost of each downstream data model consumed when determining the target data model. This application does not impose any restrictions on this.

[0065] In the above embodiments, when determining the actual investment cost of the target data model, the electronic device considers not only the computational cost incurred in generating the target data model and the storage cost of storing the target data model, but also the actual investment cost of the data model used to generate the target data model. Therefore, the technical solution of this application can more accurately determine the data model cost, thereby enabling a more accurate evaluation of the data model.

[0066] It is understandable that in two data models with a data lineage, the upstream data model can serve as the basis for generating the downstream data model. The generation of the downstream data model depends on the upstream data model. Therefore, the actual cost of the downstream data model is accurate only when it includes its computational and storage costs as well as the cost of consuming the upstream data model.

[0067] Furthermore, since the second cost of the target data model can represent the cost of at least one first data model consumed when generating the target data model, and the at least one first data model is an upstream data model of the target data model, the technical solution of this application can determine the second cost of the target data model by calculating the third cost corresponding to each of the at least one first data model based on the actual input cost of each of the at least one first data model, thereby reflecting the transitivity and multi-source nature of data model costs. The third cost of the first data model can be used to represent the cost of the first data model consumed when generating the target data model based on the first data model.

[0068] Since the actual input cost of the target data model determined by the technical solution of this application is relatively accurate, the input-output ratio of the target data model determined according to the third cost of the target data model and the actual input cost of the target data model in the above embodiments is also relatively accurate, so that the use value of the data model can be evaluated more reasonably and effectively.

[0069] In other words, the technical solution of this application can allocate the actual input cost of each data model in the entire data warehouse to its downstream data models through data lineage and the data model cost allocation method. By using the step-by-step allocation method, the cost in the data model production chain can be passed down and traced up, thereby accurately assessing the input cost and use value of each data model.

[0070] In some implementations, before S110 above, the following steps may be included: generating a directed graph structure of multiple data models, wherein multiple nodes of the directed graph structure correspond one-to-one with multiple data models, and the directed edges of the directed graph structure are represented as nodes corresponding to upstream data models in two data models that have a data lineage relationship pointing to nodes corresponding to downstream data models in the two data models; performing a topological sort on the directed graph structure to obtain a sorting result. Correspondingly, S110 above may include: calculating the first cost of the data model corresponding to any target node in the sorting result; correspondingly, S120 above may include: traversing the directed graph structure to determine at least one upstream node corresponding to the target node; processing the actual input costs of each of the at least one data model corresponding to the at least one upstream node to obtain the second cost of the data model corresponding to the target node.

[0071] It is understandable that performing topological sorting on a directed graph structure can ensure that the in-degree of each node is zero, that is, ensure that each node in the sorted result has no other upstream nodes. Thus, based on the sorted result, all upstream nodes of each node can be determined more accurately, and in turn, all upstream data models of each data model can be determined more accurately, thereby improving the accuracy of determining the actual input cost of the data model.

[0072] For example, the electronic device can traverse the above directed graph structure according to a graph traversal algorithm, such as a breadth-first traversal algorithm, but is not limited to this.

[0073] For example, suppose a data warehouse is divided into four layers: ODS, DWD, DWS, and ADS. ODS includes five data models: ODS1, ODS2, ODS3, ODS4, and ODS5; DWD includes three data models: DWD1, DWD2, and DWD3; DWS includes three data models: DWS1, DWS2, and DWS3; and ADS includes two data models: ADS1 and ADS2. ODS1 and ODS2 are both upstream data models of DWD1, DWD1 and ODS2 are both upstream data models of DWD2, ODS3 is an upstream data model of DWD3, DWD1 is an upstream data model of DWS1, DWD2 and DWD3 are both upstream data models of DWS2, ODS4 and DWD3 are both upstream data models of DWS3, DWS1 and DWS2 are both upstream data models of ADS1, and DWS3 and ODS5 are both upstream data models of ADS2. Then, electronic devices can generate data based on the above data lineage and multiple data models, such as... Figure 2 The directed graph structure is shown below. Taking ODS1, ODS2, and DWD1 as an example, as follows... Figure 2 As shown, ODS1, ODS2, and DWD1 can correspond to nodes 1, 2, and 3, respectively. Since ODS1 and ODS2 are both upstream data models of DWD1, there are directed edges from node 1 to node 3 and from node 2 to node 3 in the directed graph structure. The determination method of other directed edges is similar to that here, and will not be elaborated in this application.

[0074] The above embodiment will be described using nodes 1, 2, and 3 as examples. The method for determining the first and second costs corresponding to each node based on other nodes and directed edges in the directed graph structure is similar to that described here, and will not be elaborated upon further. Figure 2As shown, the first cost of the data model corresponding to node 3 is the first cost of DWD1. A topological sort is performed on nodes 1, 2, and 3, resulting in a sorting result of (node ​​1, node 2, node 3). Based on this sorting result, the upstream nodes of node 3 can be determined to be nodes 1 and 2. Therefore, the actual investment costs of at least one data model corresponding to at least one upstream node are the actual investment costs of ODS1 and ODS2, respectively. Then, the electronic device can process the actual investment costs of ODS1 and ODS2 using the method described in the above embodiment to obtain the third cost corresponding to ODS1 and the third cost corresponding to ODS2. Next, the electronic device can sum the third cost corresponding to ODS1 and the third cost corresponding to ODS2 to obtain the second cost of DWD1. Finally, the electronic device can determine the actual investment cost of DWD1 based on the second cost of DWD1 and the first cost of DWD1.

[0075] For example, electronic devices can use NodeName, NextNodeCount, NodeActualCost, NodeCostBefore, and NodeCostAfter to represent the node name of the target node, the number of downstream nodes of the target upstream node of the target node, the first cost of the data model corresponding to the target node, the actual investment cost of the data model corresponding to the target upstream node of the target node, and the actual investment cost of the data model corresponding to the target node, respectively. Electronic devices can determine the directed graph structure based on multiple data models and the data lineage between them. Then, they can perform a topological sort on the directed graph structure to obtain the sorting result. Next, the electronic devices can determine the first cost of the target node NodeName as NodeActualCost, and based on the sorting result, determine all upstream nodes of the target node NodeName. Taking the upstream node of the target node as an example, assuming the upstream node of the target node NodeName is the target upstream node, the electronic devices can determine the actual investment cost of the target upstream node as NodeCostBefore. Then, the electronic devices can traverse the directed graph structure to determine the number of downstream nodes of the target upstream node as NextNodeCount. The electronic devices can then determine the second cost of the data model corresponding to the target node as NodeCostBefore / (NextNodeCount+1). Therefore, the electronic devices can determine the actual investment cost of the data model corresponding to the target node as NodeCostAfter = NodeActualCost + NodeCostBefore / (NextNodeCount+1).

[0076] It should be noted that all the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here. For example, when an electronic device determines the first cost, second cost, and actual investment cost of a data model based on a directed graph structure, the specific technical solutions and effects used can refer to any embodiment in this application, and will not be described in detail here.

[0077] Figure 3 This is a schematic diagram of a data processing apparatus 300 provided in an embodiment of this application.

[0078] like Figure 3 As shown, the data processing device 300 includes: a first determining module 310, a processing module 320, and a second determining module 330. The first determining module 310 is used to determine a first cost of a target data model for any target data model among multiple data models in a data warehouse. The first cost includes: the computational cost incurred in generating the target data model from at least one first data model, and the storage cost of storing the target data model. The at least one first data model is a data model used to generate the target data model among multiple data models. The processing module 320 is used to process the actual input costs of each of the at least one first data model to obtain a second cost of the target data model. The second determining module 330 is used to determine the actual input cost of the target data model based on the first cost and the second cost of the target data model.

[0079] In some possible implementations, the processing module 320 is specifically used to: for any one of the at least one first data models, determine at least one downstream data model among a plurality of data models that has a data lineage with the first data model, wherein the at least one downstream data model includes the target data model; process the actual input cost of the first data model according to the at least one downstream data model to obtain a third cost; and sum the third costs corresponding to each of the at least one first data model to obtain a second cost of the target data model.

[0080] In some implementations, the processing module 320 is specifically used to: process the actual input cost of the first data model based on the number of downstream data models included in at least one downstream data model, to obtain a third cost.

[0081] In some implementations, the processing module 320 is specifically used to: determine the number of downstream data models included in at least one downstream data model to obtain a first value; sum the first value and a preset value to obtain a second value; and calculate the ratio of the actual input cost of the first data model to the second value to obtain a third cost.

[0082] In some possible implementations, the processing module 320 is specifically used to: determine, for any one of the at least one downstream data models, the amount of data in the first data model used when generating the downstream data model based on the first data model; determine the third value corresponding to the target data model based on the data amount corresponding to each of the at least one downstream data model; and process the actual input cost of the first data model based on the third value corresponding to the target data model to obtain the third cost.

[0083] In some implementations, the processing module 320 is specifically used to: normalize the data volume corresponding to at least one downstream data model; determine the normalized result of the data volume corresponding to the target data model as the third value corresponding to the target data model; and calculate the product of the actual input cost of the first data model and the third value corresponding to the target data model to obtain the third cost.

[0084] In some implementations, the data processing device 300 further includes: a generation module 340 and a sorting module 350. The generation module 340 generates a directed graph structure of multiple data models, where each node in the directed graph structure corresponds one-to-one with a data model. The directed edges of the graph structure are represented as nodes pointing from the upstream data model to the downstream data model in two data models with a data lineage relationship. The sorting module 350 performs a topological sort on the directed graph structure to obtain a sorting result. The first determining module 310 is specifically used to: calculate the first cost of the data model corresponding to any target node in the sorting result. The processing module 320 is specifically used to: traverse the directed graph structure to determine at least one upstream node corresponding to the target node; and process the actual input costs of each of the at least one data model corresponding to the at least one upstream node to obtain the second cost of the data model corresponding to the target node.

[0085] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 3 The apparatus 300 shown can execute the above-described method embodiments, and the aforementioned and other operations and / or functions of each module in the apparatus 300 are respectively for implementing the corresponding processes in the above-described methods. For the sake of brevity, they will not be described in detail here.

[0086] The apparatus 300 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0087] Figure 4 This is a schematic block diagram of an electronic device 400 provided in an embodiment of this application.

[0088] like Figure 4 As shown, the electronic device 400 may include:

[0089] The system includes a memory 410 and a processor 420. The memory 410 stores computer programs and transfers the program code to the processor 420. In other words, the processor 420 can retrieve and run the computer program from the memory 410 to implement the methods described in the embodiments of this application.

[0090] For example, the processor 420 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0091] In some embodiments of this application, the processor 420 may include, but is not limited to:

[0092] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0093] In some embodiments of this application, the memory 410 includes, but is not limited to:

[0094] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0095] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 410 and executed by the processor 420 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0096] like Figure 4 As shown, the electronic device may further include:

[0097] Transceiver 430, which can be connected to processor 420 or memory 410.

[0098] The processor 420 can control the transceiver 430 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include antennas, and the number of antennas may be one or more.

[0099] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0100] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0101] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., Digital Video Disc (DVD)), or a semiconductor medium (e.g., Solid State Disk (SSD)).

[0102] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0104] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0105] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, include: For any target data model among multiple data models in a data warehouse, a first cost of the target data model is determined, wherein the first cost includes: the computational cost incurred in generating the target data model based on at least one first data model, and the storage cost of storing the target data model. The at least one first data model is a data model among the multiple data models used to generate the target data model. The data model is a model in the data warehouse used to support business analysis and decision-making. There is a data lineage relationship between the data models, which represents an upstream and downstream relationship between the data models in the process of data calculation and production. Based on the data lineage, the actual input costs of each of the at least one first data model are processed through a step-by-step allocation method in the data model generation chain to obtain the second cost of the target data model. This includes: for any one of the at least one first data model, identifying at least one downstream data model among the plurality of data models that has a data lineage with the first data model, the at least one downstream data model including the target data model; processing the actual input costs of the first data model according to the at least one downstream data model to obtain a third cost; summing the third costs corresponding to each of the at least one first data model to obtain the second cost of the target data model; wherein, the second cost represents the allocation of the actual input costs of the at least one first data model by the target data model, and this allocation is used to realize the downward transmission and upward tracing of costs in the data model production chain. Based on the first cost and the second cost of the target data model, the actual input cost of the target data model is determined for the purpose of evaluating the target data model.

2. The method according to claim 1, characterized in that, The step of processing the actual input cost of the first data model based on the at least one downstream data model to obtain the third cost includes: Based on the number of downstream data models included in the at least one downstream data model, the actual input cost of the first data model is processed to obtain the third cost.

3. The method according to claim 2, characterized in that, The step of processing the actual investment cost of the first data model based on the number of downstream data models included in the at least one downstream data model to obtain the third cost includes: Determine the number of downstream data models included in the at least one downstream data model to obtain a first value; The first value and the preset value are summed to obtain the second value; The third cost is obtained by calculating the ratio of the actual input cost of the first data model to the second value.

4. The method according to claim 1, characterized in that, The step of processing the actual input cost of the first data model based on the at least one downstream data model to obtain the third cost includes: For any of the at least one downstream data models, determine the amount of data in the first data model used when generating the downstream data model based on the first data model; Based on the data volume corresponding to each of the at least one downstream data model, determine the third value corresponding to the target data model; Based on the third value corresponding to the target data model, the actual investment cost of the first data model is processed to obtain the third cost.

5. The method according to claim 4, characterized in that, The step of determining the third value corresponding to the target data model based on the data volume corresponding to each of the at least one downstream data model includes: The data volume corresponding to each of the at least one downstream data model is normalized. The normalized result of the data volume corresponding to the target data model is determined as the third value corresponding to the target data model; The step of processing the actual investment cost of the first data model based on the third value corresponding to the target data model to obtain the third cost includes: The third cost is obtained by multiplying the actual cost of the first data model by the third value corresponding to the target data model.

6. The method according to any one of claims 1-5, characterized in that, Before determining the first cost of any target data model among multiple data models for the data warehouse, the method further includes: A directed graph structure is generated for the multiple data models. The multiple nodes of the directed graph structure correspond one-to-one with the multiple data models. The directed edges of the directed graph structure are represented as nodes pointing from the upstream data model in two data models that have a data lineage relationship to the downstream data model in the two data models. Perform a topological sort on the directed graph structure to obtain the sorting result; For any target data model among multiple data models in the data warehouse, determining the first cost of the target data model includes: For any target node in the sorting results, calculate the first cost of the data model corresponding to the target node; The step of processing the actual input costs of each of the at least one first data model to obtain the second cost of the target data model includes: Traverse the directed graph structure to determine at least one upstream node corresponding to the target node; The actual investment cost of each of the at least one data model corresponding to the at least one upstream node is processed to obtain the second cost of the data model corresponding to the target node.

7. A data processing apparatus, characterized in that, The data processing device includes: a first determining module, a processing module, and a second determining module, wherein, The first determining module is used to determine a first cost of the target data model for any target data model among multiple data models in the data warehouse. The first cost includes: the computational cost incurred in generating the target data model based on at least one first data model, and the storage cost of storing the target data model. The at least one first data model is a data model among the multiple data models used to generate the target data model. The data model is a model in the data warehouse used to support business analysis and decision-making. There is a data lineage relationship between the data models, which represents an upstream and downstream relationship between the data models in the process of data calculation and production. The processing module is used to process the actual input cost of each of the at least one first data model in the data model generation chain based on the data lineage relationship, through a step-by-step allocation method, to obtain the second cost of the target data model. This includes: for any one of the at least one first data model, determining at least one downstream data model among the plurality of data models that has a data lineage relationship with the first data model, the at least one downstream data model including the target data model; processing the actual input cost of the first data model according to the at least one downstream data model to obtain a third cost; and summing the third costs corresponding to each of the at least one first data model to obtain the second cost of the target data model. The second cost represents the allocation of the actual input cost of the at least one first data model by the target data model, and this allocation is used to realize the downward transmission and upward tracing of costs in the data model production chain. The second determining module is used to determine the actual input cost of the target data model based on the first cost and the second cost of the target data model, so as to evaluate the target data model.

8. The apparatus according to claim 7, characterized in that, The processing module is specifically used for: Based on the number of downstream data models included in the at least one downstream data model, the actual input cost of the first data model is processed to obtain the third cost.

9. The apparatus according to claim 8, characterized in that, The processing module is specifically used for: Determine the number of downstream data models included in the at least one downstream data model to obtain a first value; The first value and the preset value are summed to obtain the second value; The third cost is obtained by calculating the ratio of the actual input cost of the first data model to the second value.

10. The apparatus according to claim 8, characterized in that, The processing module is specifically used for: For any of the at least one downstream data models, determine the amount of data in the first data model used when generating the downstream data model based on the first data model; Based on the data volume corresponding to each of the at least one downstream data model, determine the third value corresponding to the target data model; Based on the third value corresponding to the target data model, the actual investment cost of the first data model is processed to obtain the third cost.

11. The apparatus according to claim 10, characterized in that, The processing module is specifically used for: The data volume corresponding to each of the at least one downstream data model is normalized. The normalized result of the data volume corresponding to the target data model is determined as the third value corresponding to the target data model; The third cost is obtained by multiplying the actual cost of the first data model by the third value corresponding to the target data model.

12. The apparatus according to any one of claims 7-11, characterized in that, The data processing device further includes: a generation module and a sorting module, wherein... The generation module is used to generate a directed graph structure of the multiple data models. The multiple nodes of the directed graph structure correspond one-to-one with the multiple data models. The directed edges of the directed graph structure are represented as nodes pointing from the upstream data model in two data models that have a data lineage relationship to the downstream data model in the two data models. The sorting module is used to perform topological sorting on the directed graph structure to obtain the sorting result; The first determining module is specifically used for: For any target node in the sorting results, calculate the first cost of the data model corresponding to the target node; The processing module is specifically used for: Traverse the directed graph structure to determine at least one upstream node corresponding to the target node; The actual investment cost of each of the at least one data model corresponding to the at least one upstream node is processed to obtain the second cost of the data model corresponding to the target node.

13. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-6 by executing the executable instructions.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

15. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, the electronic device performs the method of any one of claims 1-6.

Images