Data processing method and device, computer equipment and storage medium

Abstract

The invention discloses a data processing method and device, computer equipment and a storage medium, and the method comprises the steps: determining the total cost of resources consumed by a first task unit according to the resource information of the resources consumed by the first task unit in a cluster for processing a task; according to the data blood relationship between the first task unit and each second task unit in the cluster, determining a cost allocation proportion for allocating the total cost of the first task unit; and finally, according to the cost allocation proportion, performing cost allocation on the total cost of the resources consumed by the first task unit. According to the invention, the accuracy of cost allocation of resources can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] Big data clusters have become a key tool for processing and analyzing data. After a task is completed, the total cost of the cluster's resource consumption will be periodically counted. After the total cost is allocated to each task unit, the cost allocation result will be sent to the business department for subsequent processing.

[0003] However, existing big data cluster cost allocation usually adopts static or simple allocation strategies, which are difficult to cope with dynamically changing task execution, data processing and resource adjustment in the cluster, resulting in inaccurate cost allocation results. Summary of the invention

[0004] In response to the above technical problems, embodiments of the present application provide a data processing method, apparatus, computer device and storage medium.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] Determine the total cost of resources consumed by the first task unit according to resource information of resources consumed by the first task unit in the cluster for processing tasks;

[0007] Determining a cost sharing ratio for sharing a total cost of resources consumed by the first task unit according to a data lineage relationship between the first task unit and each second task unit in the cluster;

[0008] The total cost of the resources consumed by the first task unit is apportioned according to the cost apportionment ratio.

[0009] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0010] A cost determination module, configured to determine a total cost of resources consumed by a first task unit in a cluster according to resource information of resources consumed by the first task unit in processing tasks;

[0011] a ratio determination module, configured to determine a cost sharing ratio for sharing a total cost of resources consumed by the first task unit according to a data lineage relationship between the first task unit and each second task unit in the cluster;

[0012] The cost allocation module is used to allocate the total cost of the resources consumed by the first task unit according to the cost allocation ratio.

[0013] In a third aspect, an embodiment of the present application further provides a computer device, comprising a memory storing a plurality of instructions; a processor loads instructions from the memory to execute the steps of any data processing method provided in the embodiment of the present application.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps of any data processing method provided in the embodiment of the present application.

[0015] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps of any data processing method provided in the embodiment of the present application.

[0016] According to the scheme of the embodiment of the present application, the total cost of the resources consumed by the first task unit is determined according to the amount of different resources consumed by the first task unit in the cluster to process the task, and the unit price of the cost calculated uniformly for different types of resources. Thus, the cost association between different types of resources consumed by the task unit can be comprehensively and accurately considered, and the cost to be shared of each task unit can be accurately counted based on the cost association of different resource consumption. According to the data lineage relationship between the first task unit and each second task unit in the cluster, the cost sharing ratio for sharing the total cost of the first task unit is determined, so that the cost sharing can more accurately reflect the actual resource usage and data flow of the task unit based on the dynamic data lineage relationship between each task unit, strengthen the correlation between cost sharing and data lineage, and make the cost sharing result more accurate and more in line with the actual situation. According to the cost sharing ratio, the total cost of the resources consumed by the first task unit is shared, and then more accurate cost sharing is achieved through the above method, which can better reflect the cost of each task unit in resource use, help identify and optimize high-cost operations, and thus improve the overall resource utilization efficiency of the big data cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a schematic diagram of the application environment of the data processing method provided in the embodiment of the present application;

[0019] Figure 2 It is a software architecture diagram of the existing cost sharing execution program;

[0020] Figure 3 It is a schematic diagram of the existing process of executing cost allocation;

[0021] Figure 4 It is a schematic diagram of an embodiment of the data processing method provided in the embodiments of the present application;

[0022] Figure 5 It is a software architecture diagram of the cost sharing execution program provided in the embodiment of the present application;

[0023] Figure 6 It is a flowchart of the execution cost allocation provided in the embodiment of the present application;

[0024] Figure 7 is a schematic diagram of the data flow between each task unit provided in the embodiment of the present application;

[0025] Figure 8 It is a schematic diagram of the data kinship relationship between each task unit provided in the embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0027] Fig.10 It is a schematic diagram of the internal structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0029] In one embodiment of the present application, the data processing method can be run on a local terminal device or a server. When the data processing method is run on a server, the method can be implemented and executed based on a data processing system, wherein the data processing system includes a server and a client device.

[0030] In order to better understand a data processing method, device, electronic device and storage medium provided in an embodiment of the present application, an application environment applicable to the embodiment of the present application is first described below.

[0031] See also Figure 1 , Figure 1 1 shows a schematic diagram of an application environment of a data processing method provided by an embodiment of the present application. As an implementation mode, the data processing method provided by an embodiment of the present application can be applied to an electronic device. The electronic device can be as follows: Figure 1 The server 110 shown in the figure can communicate with the terminal device 120 through a network. The network is a medium for providing a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as wired communication links, wireless communication links, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the electronic device can also be a smart phone, a laptop computer, etc.

[0032] It should be understood that Figure 1 The server 110, network and terminal device 120 are merely illustrative, and any number of servers, networks and terminal devices may be provided as required. For example, the server 110 may be a physical server or a server cluster composed of multiple servers, and the terminal device 120 may be a mobile phone, a tablet, a desktop computer, a laptop computer, and the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 at the same time.

[0033] In some embodiments, the terminal device 120 may obtain the total cost of resources consumed by the first task unit. Further, the terminal device 120 sends the total cost of resources consumed by the first task unit to the server 110 through the network. After receiving the total cost data, the server 110 may analyze it to determine the cost sharing ratio for sharing the total cost of the first task unit. The total cost data of the first task unit may be resource allocated based on the cost sharing ratio using the data processing method described in the embodiment of the present application.

[0034] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is a terminal device as an example. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0035] The data processing method of this embodiment can be applied to any scenario that needs to be executed, and the method can be applied to products in these scenarios, such as cluster data table systems, data processing systems, etc. Before introducing the data processing method of this embodiment, the relevant technical data in the common scenarios of the data processing method of this embodiment are first explained.

[0036] First of all, a big data cluster (also referred to as a "cluster") refers to a group of interconnected computer servers used to store, process and analyze big data. A cluster usually includes hundreds or even thousands of servers, which work together to handle large amounts of data, high-speed data streams and complex data analysis tasks. Clusters can provide high availability, high scalability and high-efficiency computing capabilities, so they have a wide range of application value in the field of big data collection.

[0037] Big data clusters usually use distributed computing frameworks (such as Apache Hadoop, Apache Spark, etc.) to process data, and the above frameworks allow parallel processing of data and improve computing efficiency by distributing tasks to multiple nodes.

[0038] Big data clusters usually use distributed file systems (such as Hadoop's HDFS, Apache HBase, etc.) to store data. The above systems can store data in a distributed manner on multiple nodes, providing high reliability and high capacity.

[0039] Data lineage, also known as data lineage, data origin, and data pedigree, refers to the recording and tracking of all paths and steps in the process of data transmission, conversion, and processing from the source to the destination. For example, in the entire life cycle of data, data is generated, processed, processed, integrated, and circulated to the final extinction, and a relationship is naturally formed between the data. It describes in detail the origin, change process, and final storage location of the data, allowing users to understand the generation and circulation process of the data and track the life cycle of the data. Because it records the link relationship generated by the data, these relationships are similar to human blood relationships, so they are called data lineage relationships.

[0040] In a big data cluster, data usually undergoes multiple processing and conversions and spans multiple nodes and systems, so data lineage is particularly important and can be applied to cost allocation, global view, resource optimization, performance monitoring and optimization of big data clusters. In terms of cost allocation, based on the data lineage relationship between data in a big data cluster, cost allocation can be performed more accurately to reflect the actual resource usage and data processing costs.

[0041] Existing cost allocation for big data clusters usually adopts static or simple rules, which fail to consider the actual resource usage, resulting in allocation results that are inconsistent with reality. For example, cost allocation is usually based only on rough resource consumption statistics, and fails to monitor and allocate costs in a fine-grained dimension, making it difficult to accurately track the specific resource consumption of each task or data table.

[0042] For another example, the existing cost allocation ignores the importance of data lineage and data flow, and cannot reflect the dependency of data in the processing process, resulting in a lack of scientific basis for cost allocation and low accuracy of cost allocation. Another situation is that even if the data flow between data is recorded, the cost will still be allocated with a simple strategy. For example, the data of data table A in the cluster flows to data tables B and C respectively, but because the rules of finer granularity dimensions are not used as the basis for cost allocation, the number of data flows, data volume and other information are not considered. Therefore, the total cost generated by data table A is usually only roughly allocated to data tables B and C, and the accuracy of cost allocation is still low.

[0043] In addition, the existing cost allocation method still has the problem of unclear calculation logic of the unit price of cluster resource consumption cost. Figure 2 As shown, even if the collection layer of some existing cost sharing software will also collect the data lineage relationship between data, the calculation layer of the existing cost sharing software will calculate the cost unit price of the cluster's consumption of computing resources and the cost unit price of the consumption of storage resources. Figure 3 As shown, after determining the two types of cost unit prices, the total cost of computing resources and the total cost of storage resources are determined according to the two types of cost unit prices, and subsequent cost allocation is performed in combination with the data lineage relationship in the cluster.

[0044] However, this method ignores the relationship between computing resource consumption and storage resource consumption, and does not include the relationship between the two in the cost allocation calculation, which cannot correctly reflect the cost of the cluster. For example, in the process of big data processing, the use of computing resources and storage resources is closely related. For example, some computing tasks require a large amount of reading and writing data, which not only consumes computing resources (such as CPU and memory), but also occupies a large amount of storage resources. If only the consumption of computing resources is considered separately, the cost of storage resources occupied by computing resource-consuming tasks may be ignored, and vice versa. The total cost of cluster resource consumption is not accurately calculated, and it is impossible to achieve accurate cost allocation for the cluster.

[0045] Based on this, in order to solve the problems of inaccurate cost unit price calculation, simple allocation rules, lack of scientific basis and low accuracy of allocation results in the existing cost allocation of resource consumption of big data clusters, the data processing method, device, computer equipment and storage medium of the present application are proposed.

[0046] Please refer to Figure 4 The specific process of the data processing method provided in the embodiment of the present application may include the following steps 201 to 203, which are specifically introduced below.

[0047] Step 201 : determining the total cost of resources consumed by a first task unit in a cluster according to resource information of resources consumed by the first task unit in processing tasks.

[0048] The task unit refers to a task execution object or an associated object in task execution. In some embodiments, the task unit can be a data table and a database in a cluster. The data table and the database are independent data storage entities that contain a certain set of data and can be used or processed by one or more tasks. For example, operations such as querying, writing, and cleaning a data table are performed.

[0049] In some embodiments, a task unit may be a cluster task, which refers to a specific computing or data processing operation, such as a data cleaning task, a data analysis task, etc.

[0050] In some embodiments, the task unit can also be an association unit between a data table / library and a corresponding task, which means that a specific task and the data table / library it processes are considered as a whole. For example, a task of writing data to a data table / library can be regarded as a task unit as a whole. This approach helps to have a more comprehensive understanding of the resource consumption during the task execution process.

[0051] It can be understood that, taking the first task unit in the cluster as an example, the first task unit will occupy and consume certain resources when executing tasks, which are collectively referred to as consumed resources or resource consumption in this application. In some embodiments, the resource information of consumed resources may include computing resources and storage resources, computing resources may include CPU, memory, network bandwidth, etc., which are used for task execution and data processing, and storage resources may include storage space of a hard disk or solid-state drive for storing data tables and intermediate results.

[0052] like Figure 5 and Figure 6 As shown, the present application can periodically collect the computing resources and storage resources consumed by each task unit in the cluster. For example, it can collect information such as storage space consumption of data tables and databases, quota information of databases, cluster task resource consumption, and data table resource consumption associated with cluster tasks every day.

[0053] In some embodiments, after collecting the resource information of the above resource consumption, the resource information can be counted and summarized in a longer period, for example, the above resource information can be summarized monthly to determine the total cost of the monthly resource consumption of the first task unit.

[0054] The cost of this application refers to the economic cost of resource consumption, such as the CPU usage cost and memory usage cost corresponding to the consumption of computing resources, the storage space cost and read / write operation cost corresponding to the consumption of storage resources, etc. The calculated total cost will eventually be converted into a specific amount, and the bill of the specific amount of the total cost can be sent to the relevant business department for verification and reimbursement.

[0055] In order to better implement the method of the embodiment of the present application, in a possible example, step 201 may include the following steps:

[0056] Determining computing resources and storage resources consumed by the first task unit according to the resource information;

[0057] The total cost of resources consumed by the first task unit is determined based on the computing resources and storage resources consumed by the first task unit, as well as a first cost unit price per unit of computing resources consumed and a second cost unit price per unit of storage resources consumed.

[0058] In a specific implementation, the computing resources consumed by the first task unit when processing tasks, such as CPU usage time, memory occupancy, etc., can be recorded, and the storage resources consumed by the first task unit during data storage and reading can also be synchronously recorded, such as the size of the data table, the number of read and write operations, etc.

[0059] After calculating the above resource information, the first cost unit price for each unit of computing resource consumed and the second cost unit price for each unit of storage resource consumed can be obtained. Among them, one unit can be a time unit or a quantity unit. For example, the first cost unit price can be 5 yuan per hour for each CPU occupied in task processing, 0.1 yuan per hour for each 1GB of memory occupied, and the second cost unit price can be 1 yuan per month for each 1GB of storage space occupied, and 0.05 yuan for each read and write operation performed.

[0060] It can be understood that after determining the first cost unit price and the second cost unit price, the total cost of the resources consumed by the first task unit within the preset period can be determined by combining the computing resources and storage resources consumed by the first task unit within the preset period, where the preset period can be weekly, monthly, etc.

[0061] In some embodiments, the total cost of resources consumed by the first task unit may be determined by:

[0062] combining the first cost unit price and the second cost unit price into a third cost unit price;

[0063] Determine the quantity of the third cost unit price according to the computing resources and storage resources consumed by the first task unit;

[0064] The total cost of resources consumed by the first task unit is determined according to the third cost unit price and the corresponding quantity.

[0065] The third cost unit price may also be referred to as the "computing unit (CU) unit price", which refers to the cost unit price after unifying the cost unit prices of different types of consumed resources. For example, the unit price of computing resources (first cost unit price) and the unit price of storage resources (second cost unit price) may be combined into a unified cost unit price (third cost unit price).

[0066] In some embodiments, the computing resources (such as CPU time, memory usage) and storage resources (such as data storage space, number of read and write operations) consumed by the first task unit during execution can be counted, and then the counted resource consumption can be converted into the number of the third cost unit price. For example, the number of computing units corresponding to the computing resource consumption and the storage resource consumption can be determined. For example, if the first task unit consumes computing resources and storage resources corresponding to 5 computing units, a total of 10 computing units are consumed, and the number corresponding to the third cost unit price is 10. Furthermore, the third cost unit price can be multiplied by the number of the third cost unit price consumed to obtain the total cost of the resources consumed by the first task unit.

[0067] In some embodiments, the third cost unit price is determined by:

[0068] Combining the first unit number of computing resources with the second unit number of storage resources to obtain a unit of computing unit;

[0069] The cost unit price of one unit of the computing unit is determined according to the first cost unit price and the second cost unit price as the third cost unit price.

[0070] In some embodiments, a first unit number and a second unit number can be predefined. For example, the first unit number can be one CPU hour, and the second unit number can be one hour of 4 GB of storage space. The computing resources of the first unit number and the storage resources of the second unit number can be merged to obtain a computing unit of one unit. For example, the merged computing unit can be one CPU hour and 4 GB of storage space hour. That is, every time the task unit consumes the computing resources of one CPU and 4 GB of storage space for one hour, it is counted as the consumption of resources of one computing unit.

[0071] It can be understood that after determining how many computing units resources the first task unit specifically consumes, the total cost of the resources consumed by the first task unit can be determined by combining the number of consumed computing units with the third cost unit price.

[0072] Through the above method, this application uniformly measures computing resources and storage resources as computing units CU, which can simplify the cost calculation process of resource consumption. It is no longer necessary to calculate the consumption costs of different types of resources separately. It only needs to be uniformly calculated based on computing units and the third cost unit price. It can comprehensively consider the consumption of computing resources and storage resources, avoid ignoring the correlation between computing resource consumption and storage resource consumption, and thus more accurately reflect the actual resource consumption and cost. It is helpful for global management and optimization of resource use and improve resource utilization.

[0073] It should also be noted that the unit quantity of computing resources and storage resources contained in the computing unit of the present application can be flexibly adjusted, and the computing method and unit price of the computing unit can be dynamically adjusted according to actual needs to adapt to changes in different environments and tasks, thereby improving the adaptability and scalability of the system.

[0074] Step 202: Determine a cost sharing ratio for sharing the total cost of resources consumed by the first task unit according to the data lineage relationship between the first task unit and each second task unit in the cluster.

[0075] Data lineage refers to the flow and dependency of data between task units. Through data lineage, the input and output of each task unit can be tracked to understand the dependencies between tasks. For example, the output data of task unit A is used by task unit B, and the output data of task unit B is used by task unit C. Then there is a data lineage relationship between task unit A and task unit B, and between task unit B and task unit C.

[0076] like Figure 6 As shown, similar to the resource consumption collection, the data lineage relationship generated between the first task unit and each second task unit in the cluster can be periodically collected, for example, the data lineage relationship can be collected daily.

[0077] In some embodiments, after collecting the above data lineage, the data lineage relationship can be counted and summarized according to a longer period, for example, the above collected data lineage relationship can be summarized monthly to determine the monthly data lineage relationship generated between the first task unit and each other second task unit.

[0078] like Figure 7 As shown, in different levels of the cluster, data can flow between task units at the same level and between task units at different levels. Data can also flow across levels between each task unit. For example, in task c, the data of data table E at the ODS layer is written into data table C, so that a data lineage relationship is established between data table E and data table C.

[0079] In order to better implement the method of the embodiment of the present application, in a possible example, step 202 may include the following steps:

[0080] Determine the blood relationship heat between the first task unit and each second task unit according to the number of times the first task unit and each second task unit generate data blood relationship;

[0081] The cost sharing ratio of the total cost of the first task unit to each second task unit is determined according to the blood relationship heat between the first task unit and each second task unit.

[0082] The second task unit is a task unit other than the first task unit in the cluster. In some embodiments, the number of times the first task unit generates a data lineage relationship with each second task unit can be determined. For example, if the data in data table A is written to data table B multiple times, for example, the data is written 5 times, then the number of times the data lineage relationship is generated between data table A and data table B is 5.

[0083] In some embodiments, the blood relationship heat between the two can be calculated based on the number of times a data blood relationship is generated between the first task unit and each second task unit. Among them, the blood relationship heat can be understood as the dependency strength or frequency between task units. The more times, the higher the blood relationship heat. For example, in the aforementioned example, the number of times a data blood relationship is generated between data table A and data table B is 5, then the corresponding blood relationship heat between data table A and data table B can also be counted as 5, or a mapping value can be calculated based on the number of times a data blood relationship is generated between the two by a mapping method.

[0084] In some embodiments, the blood relationship heat can be used to determine the proportion of the total cost of the first task unit to be allocated to each second task unit. Similar to the relationship between the number of blood relationships and blood relationship heat, the higher the blood relationship heat, the greater the cost allocation ratio.

[0085] In some embodiments, the cost sharing ratio may be further determined by:

[0086] Determine, according to the blood relationship heat of the first task unit, a first number of times the first task unit generates data blood relationship with all second task units, and a second number of times the first task unit generates data blood relationship with each second task unit;

[0087] The cost allocation ratio allocated to the second task unit is determined according to the ratio of the second number of times to the first number of times corresponding to each second task unit.

[0088] Just as an example, Figure 8As shown, assuming that data table E is the first task unit, data table F and data table C are the second task unit, data table E writes data e1 and e2 to data table F respectively, and a data lineage relationship of two times is generated between data table E and data table F. Correspondingly, data table E writes data e3 to data table C, and a data lineage relationship of one time is generated between data table E and data table C. That is, the second times of data table E of the first task unit and data table F and data table C of the second task unit are 2 and 1 respectively, and then the first time number is obtained as 3 by calculating the sum of all the second times numbers.

[0089] The cost sharing ratio represents the share of the total cost of the first task unit that each second task unit should bear. In some embodiments, the cost sharing ratio corresponding to each second task unit can be determined according to the ratio of the second number to the first number. For example, the cost sharing ratio corresponding to data table F is 2 / 3=66.6%, and the cost sharing ratio corresponding to data table C is 1 / 3=33.3%.

[0090] Step 203: Cost-sharing the total cost of resources consumed by the first task unit according to the cost-sharing ratio.

[0091] In some embodiments, after determining the cost allocation ratio of the first task unit to each second task unit, the total cost of the first task unit may be allocated to each second task unit.

[0092] For example, assuming that within a preset period, for example, in the current month, the total cost of resources consumed by the first task unit data table E is 100 yuan, and it has been determined that the cost sharing ratio of data table E to data table F is 2 / 3=66.6%, and the cost sharing ratio of data table C is 1 / 3=33.3%, then the cost to be shared by data table F can be solved as: the product of the total cost of data table E and the cost sharing ratio corresponding to data table F, that is, 100×66.6%=66.6 yuan. Correspondingly, the cost to be shared by data table C can be solved as: the product of the total cost of data table E and the cost sharing ratio corresponding to data table C, that is, 100×33.3%=33.3 yuan.

[0093] Through the above method, this application allocates the total cost of resources consumed by the first task unit according to the cost sharing ratio, which can ensure the rationality and fairness of the cost sharing, so that each second task unit can reasonably bear its corresponding cost, thereby accurately reflecting the resource usage and dependency relationship between tasks, and improving the accuracy and rationality of cluster cost sharing.

[0094] It should be understood that, although each step in the flowcharts involved in the above-mentioned embodiments is displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0095] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more data processing devices provided below can refer to the limitations on the data processing method above, and the specific limitations are not repeated here.

[0096] The data processing device provided in this embodiment may be integrated into a terminal device. Fig. 9 As shown, the data processing device may include: a cost determination module 301 , a ratio determination module 302 and a cost allocation module 303 .

[0097] The above-mentioned data processing device first determines the total cost of the resources consumed by the first task unit according to the resource information of the resources consumed by the first task unit in the cluster for processing tasks through the cost determination module 301, and can accurately count the cost to be shared of each task unit. The cost sharing ratio for sharing the total cost of the first task unit is determined by the ratio determination module 302 according to the data lineage relationship between the first task unit and each second task unit in the cluster, so that the cost sharing can more accurately reflect the actual resource usage and data flow of the task unit based on the dynamic data lineage relationship between each task unit, strengthen the correlation between cost sharing and data lineage, and make the cost sharing result more accurate and more in line with the actual situation. The total cost of the resources consumed by the first task unit is shared according to the cost sharing ratio through the cost sharing module 303, and then more accurate cost sharing is achieved through the above method, which can better reflect the cost of each task unit in resource use, help identify and optimize high-cost operations, and thus improve the overall resource utilization efficiency of the big data cluster.

[0098] In some embodiments, the cost determination module 301 is further specifically configured to:

[0099] Determining computing resources and storage resources consumed by the first task unit according to the resource information;

[0100] The total cost of resources consumed by the first task unit is determined based on the computing resources and storage resources consumed by the first task unit, as well as a first cost unit price per unit of computing resources consumed and a second cost unit price per unit of storage resources consumed.

[0101] In some embodiments, the cost determination module 301 is further specifically configured to:

[0102] combining the first cost unit price and the second cost unit price into a third cost unit price;

[0103] Determine the quantity of the third cost unit price according to the computing resources and storage resources consumed by the first task unit;

[0104] The total cost of resources consumed by the first task unit is determined according to the third cost unit price and the corresponding quantity.

[0105] In some embodiments, the cost determination module 301 is further specifically configured to:

[0106] Combining the first unit number of computing resources with the second unit number of storage resources to obtain a unit of computing unit;

[0107] The cost unit price of one unit of the computing unit is determined according to the first cost unit price and the second cost unit price as the third cost unit price.

[0108] In some embodiments, the ratio determination module 302 is further specifically configured to:

[0109] Determine the blood relationship heat between the first task unit and each second task unit according to the number of times the first task unit and each second task unit generate data blood relationship;

[0110] The cost sharing ratio of the total cost of the first task unit to each second task unit is determined according to the blood relationship heat between the first task unit and each second task unit.

[0111] In some embodiments, the ratio determination module 302 is further specifically configured to:

[0112] Determine, according to the blood relationship heat of the first task unit, a first number of times the first task unit generates data blood relationship with all second task units, and a second number of times the first task unit generates data blood relationship with each second task unit;

[0113] The cost allocation ratio allocated to the second task unit is determined according to the ratio of the second number of times to the first number of times corresponding to each second task unit.

[0114] By adopting the scheme of the embodiment of the present application, the total cost of the resources consumed by the first task unit is determined based on the resource information of the resources consumed by the first task unit in the cluster to process the task, and the cost to be shared of each task unit can be accurately counted. According to the data lineage relationship between the first task unit and each second task unit in the cluster, the cost sharing ratio for sharing the total cost of the first task unit is determined, so that based on the dynamic data lineage relationship between each task unit, the cost sharing can more accurately reflect the actual resource usage and data flow of the task unit, strengthen the correlation between cost sharing and data lineage, and make the cost sharing result more accurate and more in line with the actual situation. According to the cost sharing ratio, the total cost of the resources consumed by the first task unit is shared, and then more accurate cost sharing is achieved through the above method, which can better reflect the cost of each task unit in resource use, help identify and optimize high-cost operations, and thus improve the overall resource utilization efficiency of the big data cluster.

[0115] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0116] In one embodiment, the computer device is a terminal device, for example, its internal structure diagram can be as follows: Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data processing method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0117] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0119] Since the computer program stored in the computer-readable storage medium can execute any data processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0120] Based on the same inventive concept, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.

[0121] It should be noted that the resource consumption data, cost unit price data, data lineage relationship and other data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0122] Any reference to the memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Re RAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0123] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0124] In the above data processing device, computer-readable storage medium, computer equipment, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described data processing device, computer-readable storage medium, computer program product, computer equipment, and its corresponding units can refer to the description of the data processing method in the above embodiment, and will not be repeated here.

[0125] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above is a detailed introduction to a data processing method, device, computer equipment, computer-readable storage medium and computer program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the total cost of resources consumed by the first task unit according to the resource amounts of different types of resources consumed by the first task unit in the cluster for processing tasks and the unit prices of costs uniformly calculated for the different types of resources; Determining a cost sharing ratio for sharing a total cost of resources consumed by the first task unit according to a data lineage relationship between the first task unit and each second task unit in the cluster; The total cost of the resources consumed by the first task unit is apportioned according to the cost apportionment ratio.

2. The method according to claim 1, characterized in that The resources include computing resources and storage resources; The determining, according to the resource amounts of different types of resources consumed by the first task unit in the cluster for processing tasks and the unit prices of costs uniformly calculated for the different types of resources, the total cost of resources consumed by the first task unit includes: Dividing the consumption of the computing resources and the storage resources into unified units to obtain a computing unit for representing the consumption of unit resources; The total cost of resources consumed by the first task unit is determined based on the first number of computing units consuming resources by the first task unit, a first cost unit price per unit of computing resources consumed, and a second cost unit price per unit of storage resources consumed.

3. The method according to claim 2, characterized in that The determining the total cost of resources consumed by the first task unit according to the first number of computing units that consume resources by the first task unit, as well as the first cost unit price per unit of computing resources consumed and the second cost unit price per unit of storage resources consumed, comprises: According to the resource amounts of computing resources and storage resources included in the computing unit, combining the first cost unit price and the second cost unit price into a third cost unit price; The total cost of resources consumed by the first task unit is determined according to the third cost unit price and the first quantity.

4. The method according to claim 3, characterized in that The combining the first cost unit price and the second cost unit price into a third cost unit price according to the resource amount of the computing resources and the storage resources included in the computing unit includes: Determining a first number of units including a total number of computing resource units and a second number of units including a total number of storage resource units in the computing unit; The cost price of one unit of the computing unit is determined as the third cost price according to the first unit price of the first unit number and the second unit price of the second unit number.

5. The method according to claim 1, characterized in that The determining, according to the data lineage relationship between the first task unit and each second task unit in the cluster, a cost sharing ratio for sharing the total cost of resources consumed by the first task unit includes: Determine the blood relationship heat between the first task unit and each second task unit according to the number of times the first task unit and each second task unit generate data blood relationship; The cost sharing ratio of the total cost of the first task unit to each second task unit is determined according to the blood relationship heat between the first task unit and each second task unit.

6. The method according to claim 5, characterized in that The determining, according to the blood relationship heat between the first task unit and each second task unit, a cost allocation ratio of the total cost of the first task unit to each second task unit includes: Determine, according to the blood relationship heat of the first task unit, a first number of times the first task unit generates data blood relationship with all second task units, and a second number of times the first task unit generates data blood relationship with each second task unit; The cost allocation ratio allocated to the second task unit is determined according to the ratio of the second number of times to the first number of times corresponding to each second task unit.

7. A data processing device, characterized in that: The device comprises: A cost determination module, configured to determine a total cost of resources consumed by a first task unit in a cluster according to resource information of resources consumed by the first task unit in processing tasks; a ratio determination module, configured to determine a cost sharing ratio for sharing a total cost of resources consumed by the first task unit according to a data lineage relationship between the first task unit and each second task unit in the cluster; The cost allocation module is used to allocate the total cost of the resources consumed by the first task unit according to the cost allocation ratio.

8. A computer device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the data processing method according to any one of claims 1 to 6.

10. A computer software, characterized in that The computer software comprises a computer program, and the computer program is used by a processor to execute the steps of the data processing method according to any one of claims 1 to 6.

Images