Cost allocation method and device, equipment and storage medium

Abstract

The invention discloses a cost allocation method and device, equipment and a storage medium. The cost distribution method comprises the following steps: acquiring call records of payment data by each terminal object and cost generated by the payment data; calling a purchase table of which the record comprises payment data; establishing a blood relationship graph from each purchasing table to each terminal object based on the calling record of the payment data; the blood relationship graph is composed of multiple levels of associated path nodes, the purchase tables serve as primary path nodes in the blood relationship graph, and the terminal objects serve as ultimate path nodes in the blood relationship graph; calculating a path weight between the upper-level path nodes and the lower-level path nodes based on data attributes between the upper-level path nodes and the lower-level path nodes; and allocating the cost to each terminal object based on the blood relationship map and the path weight. Through the above mode, the cost allocation of the source end payment data to the terminal department can be accurately and efficiently carried out.

Description

Technical Field

[0001] The present application relates to the field of data analysis, and in particular to cost allocation methods, devices, equipment and storage media. Background Art

[0002] The cost allocation method of information consulting is mainly based on the specific data tables used by each user department, and is calculated by dividing the purchase amount by the number of user departments. However, this cost allocation method is unfair in use. For example, some key data tables may be frequently cited by multiple departments due to the uniqueness and high value of their content, resulting in higher resource consumption and dependence in actual use, but this high-frequency citation and additional resource occupation have not been reflected in the corresponding cost; at the same time, the existing allocation method also ignores the system resources occupied by the data table, such as storage space, processing power and network bandwidth. With the continuous increase in data volume and complexity, the consumption of these system resources is becoming increasingly significant. If this is not properly considered, it may lead to over-allocation and waste of system resources, which in turn affects the performance and stability of the overall data service. In addition, the current allocation method lacks a dynamic monitoring and adjustment mechanism for data usage, and cannot reflect the actual use of data resources by various departments in real time, resulting in lag and inaccuracy in cost allocation. Summary of the invention

[0003] The present application mainly provides a cost allocation method, apparatus, device and storage medium to solve the problem of inaccurate cost allocation.

[0004] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a cost allocation method, including: obtaining the call records of each terminal object for payment data and the cost generated by the payment data; the call records include the purchase table of the payment data; based on the call records of the payment data, a blood relationship map from each purchase table to each terminal object is established; the blood relationship map is composed of associated multi-level path nodes, each purchase table serves as a primary path node in the blood relationship map, and each terminal object serves as an ultimate path node in the blood relationship map; based on the data attributes between the upper and lower path nodes, the path weight between the upper and lower path nodes is calculated; based on the blood relationship map and the path weight, the cost is allocated to each terminal object.

[0005] In some embodiments, the call record based on the payment data is used to establish a blood relationship map from each of the purchase tables to each of the terminal objects, including: parsing the structured query language in the Hive log file through a syntax parser generator to extract the source table, target table and field mapping relationship in the structured query language; and constructing the blood relationship map based on the source table, the target table and the field mapping relationship.

[0006] In some embodiments, the path weight includes field-level dependency weight, data volume weight and query structure weight; the calculation of the path weight between upper and lower path nodes based on the data attributes between the upper and lower path nodes includes: based on the data attributes between the upper and lower path nodes, respectively calculating the field-level dependency weight, the data volume weight and the query structure weight between the upper and lower path nodes; calculating the weighted average of the field-level dependency weight, the data volume weight and the query structure weight as the path weight.

[0007] In some embodiments, the calculation of the field-level dependency weights between upper and lower path nodes includes: extracting the mapping relationship between the target table corresponding to the lower node in the blood relationship map and the source table corresponding to the upper node to obtain the field reference data of the target table to the source table; calculating the field-level dependency weights between upper and lower path nodes based on the referenced field weights in the field reference data and the total weight of the source table.

[0008] In some embodiments, the calculation of the data volume weight between the upper and lower path nodes includes: in the preset target operation of the structured query language, calculating the ratio of the source table data volume to the total data volume of the source table and the target table as the data volume weight between the upper and lower path nodes.

[0009] In some embodiments, the calculating the query structure weight between upper and lower level path nodes includes: matching the weight coefficient corresponding to the operation of the structured query language in a preset weight coefficient table as the query structure weight between upper and lower level path nodes.

[0010] In some embodiments, the cost includes procurement cost, storage cost and computing cost; the cost is allocated to each terminal object based on the blood relationship map and the path weight, including: calculating the path weight of each terminal object for the procurement table based on the path weight between the upper and lower path nodes; according to the path weight corresponding to each terminal object, allocating the sum of the procurement cost, storage cost and computing cost to each terminal object.

[0011] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide a cost allocation device, including: an acquisition module, used to obtain the call records of payment data of each terminal object and the cost generated by the payment data; the call records include the purchase table of the payment data; an establishment module, used to establish a blood relationship map from each purchase table to each terminal object based on the call records of the payment data; the blood relationship map is composed of associated multi-level path nodes, each purchase table serves as a primary path node in the blood relationship map, and each terminal object serves as an ultimate path node in the blood relationship map; a calculation module, used to calculate the path weight between upper and lower path nodes based on the data attributes between the upper and lower path nodes; an allocation module, used to allocate the cost to each terminal object based on the blood relationship map and the path weight.

[0012] The present application also provides a computer device, which includes: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the cost allocation method as described above.

[0013] The present application also provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the cost allocation method as described above is implemented.

[0014] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a cost allocation method, device, equipment and storage medium. By obtaining the call records of each terminal object for paid data and the cost generated by the paid data; the call records include the purchase table of the paid data; based on the call records of the paid data, a blood relationship map from each purchase table to each terminal object is established; the blood relationship map is composed of associated multi-level path nodes, each purchase table is used as a primary path node in the blood relationship map, and each terminal object is used as an ultimate path node in the blood relationship map, forming a complete data processing path, so as to reversely trace the initial node at the terminal node. Based on the data attributes between the upper and lower path nodes, the path weights between the upper and lower path nodes are calculated; based on the blood relationship map and the path weights, the cost is apportioned to each terminal object, so that the cost generated by the source end is apportioned according to the actual usage, ensuring that the cost allocation is efficient and in line with the actual usage of the paid data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which: Figure 1 It is a flow chart of an embodiment of a cost allocation method provided by the present application; Figure 2 Yes Figure 1 The flowchart of the embodiment of step 20 of the method is shown; Figure 3 Yes Figure 1 The flowchart of the embodiment of step 30 of the method is shown; Figure 4 Yes Figure 3 The flowchart of the embodiment of step 31 of the method shown; Figure 5 Yes Figure 1 The flowchart of the embodiment of the method step 40 is shown; Figure 6 Yes Figure 5 A schematic diagram of a blood relationship map of an embodiment of the method shown; Figure 7 It is a structural schematic diagram of an embodiment of a cost allocation device in an embodiment of the present invention; Figure 8 It is a structural diagram of an embodiment of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0018] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0019] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of a cost allocation method provided by the present application, and the cost allocation method includes: Step 10: Obtain the call records of each terminal object for the paid data and the cost generated by the paid data. The call records include the purchase table of the paid data.

[0020] When the terminal object calls the source paid data and the derivative data generated by the source paid data, corresponding cost will be incurred. These costs are accurately calculated based on the call frequency and data processing volume to ensure that each fee can be traced back to the specific terminal object and the corresponding paid data usage.

[0021] The call record includes a purchase list of paid data, which is used to track and verify the data usage behavior of each terminal object, ensure the transparency and rationality of fee allocation, and effectively avoid waste of resources and inflated costs.

[0022] Step 20: Establish a blood relationship map from each purchase table to each terminal object based on the call record of the payment data. The blood relationship map is composed of associated multi-level path nodes, each purchase table is used as a primary path node in the blood relationship map, and each terminal object is used as the ultimate path node in the blood relationship map.

[0023] In SQL statements, the source table is the table where the data comes from, and the target table is the table where the data is finally stored or output. By traversing the syntax tree, we can identify these tables. In SQL statements, the field mapping relationship describes the correspondence between the source table fields and the target table fields. For example, in a SELECT statement, the fields in the SELECT clause are usually mapped to the table fields in the FROM clause. Based on the identified source table, target table, and field mapping relationship, we can build a lineage graph. This graph records the dependencies between tables and the dependencies at the field level.

[0024] Get all call records related to payment data, including purchase tables, intermediate processing steps, and the use of terminal objects. Make sure each step in the call record has a clear identifier, such as purchase table ID, processing step ID, terminal object ID, etc.

[0025] Each purchase table is regarded as the primary path node of the blood relationship map, all intermediate processing steps are identified and defined as intermediate path nodes, and each terminal object is regarded as the ultimate path node of the blood relationship map.

[0026] Create nodes for each purchase table, intermediate processing step, and terminal object. Establish relationships between nodes based on call records. For example, purchase table A generates intermediate data C through processing step B, which is eventually used by terminal object D. Then the relationship chain is ABCD.

[0027] The blood relationship map records the calling paths and dependencies of each terminal object on the paid data. Specifically, the paid data purchase table undergoes multiple data processing to generate multi-level derived data, and the paths of each terminal object calling derived data or paid data constitute the blood relationship map.

[0028] Further, see Figure 2 , step 20 comprises: Step 21: Parse the structured query language in the Hive log file through the syntax parser generator to extract the source table, target table and field mapping relationship in the structured query language.

[0029] All structured query languages ​​(hereinafter referred to as SQL statements) are extracted from Hive log files, and the structured query languages ​​in Hive log files are parsed by the syntax parser generator.

[0030] Optionally, the SQL grammar parser generator may be ANTLR, SQLGlot, Apache Calcite, etc., for parsing SQL statements. ANTLR is used for illustration in this embodiment.

[0031] Use ANTLR to define the grammar rules of HiveSQL. The grammar rule file defines the structure of SQL statements, such as SELECT, FROM, JOIN, etc.

[0032] Use the parser generated by ANTLR to parse the SQL statement and generate an abstract syntax tree (AST). Traverse the AST to extract the source table (tables in the FROM clause and JOIN clause), the target table (table in INSERT INTO or CREATE TABLE), and the field mapping relationship (the correspondence between the fields in the SELECT clause and the fields in the source table). Store the parsing results as structured data for use in subsequent steps.

[0033] Step 22: Build a blood relationship map based on the source table, target table and field mapping relationship.

[0034] The blood relationship graph consists of nodes and edges, where nodes represent tables or fields, and edges represent the dependency relationships between tables and fields.

[0035] Create nodes for each source table and target table, and create edges from the source table field to the target table field based on the field mapping relationship to form a complete dependency chain. For example, if field X of source table A is mapped to field Y of target table B, add edge AX-BY to the graph to ensure that each data flow path is clearly visible. In this way, the complete flow process of data from the procurement table to the terminal object is fully displayed, which is convenient for traceability and management.

[0036] For example, there is the following SQL statement: “SQL INSERT INTO TABLE table1 SELECTa.name,b.salaryFROMemployeesaJOINsalariesbONa.id=b.employee_idWHEREa.department='Engineering'" ANTLR will generate a syntax tree. By traversing this syntax tree, we can extract: source table: employees, salaries, target table: table1, field mapping relationship: a.name→name, b.salary→salary.

[0037] Through the above steps, the blood relationship in this SQL statement is extracted, and the following is obtained: source table: employees, salaries, target table: table1, field mapping relationship: employees.name→name, salaries.salary→salary. Based on this mapping relationship, the system can accurately track the data flow, optimize query efficiency, and ensure data consistency.

[0038] Step 30: Calculate the path weights between the upper and lower level path nodes based on the data attributes between the upper and lower level path nodes.

[0039] The data attributes between the upper and lower path nodes include data volume, update frequency, importance, dependency, processing time, and data quality. The path product algorithm is used to quantify the cost transmission relationship to ensure that the weight of A→B→C accurately reflects the product of the weight of A to B and the weight of B to C, thereby accurately allocating storage costs. The path product algorithm is used to accurately calculate the weight ratio between each node to ensure the transparency of the cost transmission relationship of the data flow path.

[0040] Furthermore, the path weight includes field-level dependency weight, data volume weight, and query structure weight.

[0041] See also Figure 3 , step 30 comprises: Step 31: Based on the data attributes between the upper and lower level path nodes, the field level dependency weight, data volume weight and query structure weight between the upper and lower level path nodes are calculated respectively.

[0042] The field-level dependency weight reflects the direct dependency between fields, the data volume weight reflects the impact of data scale on cost, and the query structure weight considers query complexity. By combining these three types of weights, the cost transmission between nodes can be accurately quantified to ensure reasonable and transparent cost allocation.

[0043] Further, see Figure 4 , step 31 comprises: Step 311: Extract the mapping relationship between the target table corresponding to the lower-level node and the source table corresponding to the upper-level node in the blood relationship map to obtain the field reference data of the target table to the source table.

[0044] Parse SQL statements to extract the mapping relationship between the target table fields and the source table fields. Count the references of the target table fields to the source table fields.

[0045] Step 312: Calculate the field-level dependency weights between upper and lower level path nodes based on the referenced field weights in the field reference data and the total weight of the source table.

[0046] Assign higher weights to key fields such as primary key and price (for example, primary key weight is 2, common field weight is 1). Calculate according to the field-level dependency weight calculation formula "Σ(reference field weight) / source table total weight".

[0047] For example, the source table has 10 fields (including 1 primary key), the primary key weight is 2, the common field weight is 1, the target references 1 primary key + 2 common fields, and the field-level dependency weight is (2+1+1) / (2+9*1)=4 / 11≈0.36, ensuring that key data affects weight distribution.

[0048] Further, the data volume weight between the upper and lower path nodes is calculated, including: in the preset target operation of the structured query language, calculating the ratio of the source table data volume to the total data volume of the source table and the target table as the data volume weight between the upper and lower path nodes.

[0049] The data volume weight can reflect the impact of data scale on cost allocation. The number of rows in the table is obtained through Hive Metastore or the statistical system, and the contribution ratio of the source table data volume in the JOIN or UNION operation is calculated to ensure that the weight distribution accurately reflects the impact of data volume on blood relationship, thereby optimizing storage cost allocation. The calculation is performed according to the data volume weight calculation formula "source table data volume involved in calculation / total data volume".

[0050] For example, if the source table has 100,000 rows, 10,000 rows remain after field filtering, and the total data volume is 50,000 rows, then the data volume weight is 10,000 / 50,000=0.2, which accurately reflects the contribution of the source table data. Combined with the query structure weight, the cost transmission between nodes is comprehensively evaluated to ensure reasonable and transparent cost allocation.

[0051] Further, the query structure weight between the upper and lower path nodes is calculated, including: matching the weight coefficient corresponding to the operation of the structured query language in a preset weight coefficient table as the query structure weight between the upper and lower path nodes.

[0052] The query structure weight can reflect the impact of the complexity of the query operation on the cost.

[0053] In this embodiment, the preset weight coefficient table refers to Table 1:

[0054] Table 1 By matching the weight coefficient table, the weight of each operation type is determined. For example, the JOIN operation weight is 1.2, and the AGGREGATE operation weight is 1.5.

[0055] Step 32: Calculate the weighted average of the field-level dependency weight, data volume weight, and query structure weight as the path weight.

[0056] The path weight of each edge in the final blood relationship map is calculated according to the path weight calculation formula "(field-level weight + data volume weight + query structure weight) / 3".

[0057] Taking the SQL statement in the above example as an example, in order to calculate the weights of employees and salaries to table1, we need to calculate them separately according to the three dimensions of weight calculation (field-level dependency weight, data volume weight, and query structure weight), and then take the average value as the final weight.

[0058] Through field mapping, we can extract that the field name of table1 comes from employees.name. The field salary of table1 comes from salaries.salary. Assume that the employees table has 10 fields (including 1 primary key), the primary key weight is 2, and the weight of common fields is 1. Assume that the salaries table has 5 fields (including 1 primary key), the primary key weight is 2, and the weight of common fields is 1.

[0059] The field weights from employees to table1 include: Reference field: name (normal field, weight is 1).

[0060] Total weight of the source table: 1 primary key (weight 2) + 9 common fields (weight 1) = 11.

[0061] Field weight = 1 / 11≈0.09.

[0062] Salaries to table1 field weights: Reference field: salary (normal field, weight 1). Total source table weight: 1 primary key (weight 2) + 4 normal fields (weight 1) = 6. Field weight = 1 / 6≈0.17.

[0063] The final field-level weight of employees to table1 is 0.09; the field-level weight of salaries to table1 is 0.17.

[0064] Next, calculate the data volume weight. Data volume acquisition includes: Assume that the employees table has 100,000 rows and the salaries table has 80,000 rows. The selectivity of the filter condition a.department='Engineering' is 0.1, so the amount of data in the employees table involved in the calculation is 100,000 × 0.1 = 10,000 rows.

[0065] The salaries table is joined to the employees table via a JOIN operation. Assuming the selectivity of the JOIN condition a.id=b.employee_id is 1.0, the amount of data in the salaries table involved in the calculation is 80,000 rows.

[0066] The total data volume calculation includes: Total data volume = data volume of employees involved in the calculation + data volume of salaries involved in the calculation = 10,000 + 80,000 = 90,000 rows.

[0067] The data volume weight calculation includes: The weight of the amount of data from employees to table1 = 10,000 / 90,000 ≈ 0.11. The weight of the amount of data from salaries to table1 = 80,000 / 90,000 ≈ 0.89.

[0068] The final data volume weight calculation results are: the data volume weight from employees to table1 is 0.11, and the data volume weight from salaries to table1 is 0.89.

[0069] Next, we get the query structure weight. The SQL statement contains JOIN operations and WHERE filtering operations. The weight coefficient of the JOIN operation is 1.2. The weight coefficient of the WHERE filtering operation is 0.8. The query structure weight is calculated using the formula "JOIN weight coefficient × WHERE weight coefficient", which is 1.2 × 0.8 = 0.96.

[0070] Since the query structure weight is global, employees and salaries share the same query structure weight.

[0071] The final query structure weight results are: the query structure weight of employees to table1 is 0.96, and the query structure weight of salaries to table1 is 0.96.

[0072] According to the final weight calculation formula "(field-level weight + data volume weight + query structure weight) / 3", the final weight of employees to table1 = (0.09+0.11+0.96) / 3≈0.39; the final weight of salaries to table1 = (0.17+0.89+0.96) / 3≈0.67. The final result shows that the influence weight of the employees table on table1 is 0.39, while the influence weight of the salaries table is 0.67, indicating that the salaries table is more important in the query.

[0073] Step 40: Allocate the cost to each terminal object based on the blood relationship map and path weight.

[0074] Clean and standardize expense costs and generate cost metadata.

[0075] Furthermore, the cost includes procurement cost, storage cost and computing cost.

[0076] The purchase cost is the cost of the purchase table, and is allocated according to the original table storage cost principle.

[0077] The storage cost is divided into two parts: the original table and the derived table. The storage cost of the original table is allocated according to the proportion of procurement cost, while the storage cost of the derivative table is borne in full by the direct downstream to ensure accurate cost transmission.

[0078] The computing cost is obtained from the computing cost waterfall allocation results based on the computing resource usage of each department, ensuring that the actual computing costs borne by each department match its resource usage.

[0079] See also Figure 5 , step 40 further comprises: Step 41: Calculate the path weight of each terminal object for the purchase table based on the path weights between the upper and lower path nodes.

[0080] If the terminal object directly depends on the purchase table, the path weight is the weight of the direct path between the two. If the terminal object indirectly depends on the purchase table through multiple intermediate nodes, the weights of each path segment need to be calculated comprehensively. Common comprehensive calculation methods include: adding the weights of each path segment, multiplying the weights of each path segment, or taking the average weight of each path segment.

[0081] For each terminal object, all path weights to the purchase table are summed up to obtain the total path weight of the terminal object to the purchase table.

[0082] Step 42: Allocate the sum of the purchase cost, storage cost and computing cost to each terminal object according to the path weight corresponding to each terminal object.

[0083] The procurement cost allocation adopts a three-level transmission model, which traverses all valid paths from the procurement table to the terminal department, calculates the transmission weight of each path, and finally allocates the cost to the terminal object according to the weight ratio. This method ensures the accuracy and traceability of cost allocation.

[0084] For example, see Figure 6 , taking O1 procurement cost 20000 as an example, first traverse the valid path, including: Path 1: O1→M1→R1→D1, Path 2: O1→M1→R1→D2, Path 3: O1→M1→M2→R2→D2, Path 4: O1→M1→M2→R3→D3, Path 5: O1→M1→M2→R3→D4.

[0085] Next, the path weight is calculated, including: Path 1 weight: W_path1=(0.7÷1.0)×(0.6÷1.0)×(0.8÷1.0)=0.7×0.6×0.8=0.336; Path 2 weight: W_path2=(0.7÷1.0)×(0.6÷1.0)×(0.2÷1.0)=0.7×0.6×0.2=0.084; Path 3 weight: W_path3=(0.7÷1.0)×(0.4÷1.0)×(0.3÷1.0)×(1.0÷1.0) =0.7×0.4×0.3×1.0=0.084; path 4 weight: W_path4=(0.7÷1.0)×(0.4÷1.0)×(0.7÷1.0)×(0.6÷1.0)=0.7×0.4×0.7×0.6=0.1176; path 5 weight: W_path5=(0.7÷1.0)×(0.4÷1.0)×(0.7÷1.0)×(0.4÷1.0)=0.7×0.4×0.7×0.4=0.0784.

[0086] Then aggregate the department weights, including: D1 weight: W_D1=W_path1=0.336; D2 weight: W_D2=W_path2+W_path3=0.084+0.084=0.168; D3 weight: W_D3=W_path4=0.1176; D4 weight: W_D4=W_path5=0.0784.

[0087] Finally, the cost is allocated according to the weight ratio. The total weight is calculated to be 0.336+0.168+0.1176+0.0784=0.7.

[0088] The acquisition costs of each terminal object include: D1 acquisition cost = 20000×(0.336÷0.7) = 9600; D2 acquisition cost = 20000×(0.168÷0.7) = 4800; D3 acquisition cost = 20000×(0.1176÷0.7) = 3360; D4 acquisition cost = 20000×(0.0784÷0.7) = 2240.

[0089] According to the allocation results, a summary table of allocation results is generated as shown in Table 2:

[0090] Table 2 Finally, conservation verification is required: Total allocated cost = 9,600 + 4,800 + 3,360 + 2,240 = 20,000 Total allocated cost = 9,600 + 4,800 + 3,360 + 2,240 = 20,000, which completely matches the initial cost of 20,000, verifying the accuracy of the allocation process.

[0091] The calculation ideas for storage cost and computing cost can also refer to the calculation steps of procurement cost in the above example.

[0092] Continue with Figure 6 Taking the data lineage relationship in as an example, the calculation cost of model table M1 is set to 5000, and the calculation cost of each terminal object is calculated.

[0093] In the first-level node downward from M2, M2 obtains a proportion of 0.4 / (0.4+0.6)=40%, and R1 obtains a proportion of 0.6 / (0.4+0.6)=60%. The calculated cost distribution is M2 obtains = 5000×40%=2000, and R1 obtains = 5000×60%=3000.

[0094] Further allocating the cost of the M2 node, R2 gains = 2000 × 30% = 600, and R3 gains = 2000 × 70% = 1400.

[0095] At the same time, the cost of the R1 node is allocated, D1 obtains = 3000×80%=2400, and D2 obtains = 3000×20%=600.

[0096] Next, allocate the cost of the R2 node, D2 gains = 600 × 100% = 600.

[0097] The cost of allocating R3 node is: D3 gains = 1400 × 60% = 840, and D4 gains = 1400 × 40% = 560.

[0098] After summarizing, the final costs and sources allocated to each department are shown in Table 3:

[0099] Table 3 Finally, conservation verification is performed and the total allocated cost = 2400 (D1) + 1200 (D2) + 840 (D3) + 560 (D4) = 5000, which exactly matches the initial cost of 5000, ensuring that the allocation process is correct.

[0100] Optionally, based on the cost allocation of each department, a corresponding department cost summary table is generated to display the cost allocation details from the purchase table to each department.

[0101] Optionally, a kinship map is generated through a visualization tool for display.

[0102] Optionally, resource consumption data such as CPU, memory, and storage are collected from the YARN resource monitoring system and associated with task execution logs to provide task-level resource usage analysis.

[0103] Obtain task resource consumption data in real time through YARN's REST API, including CPU usage, memory usage, storage IO, etc. Associate resource consumption data with task execution logs to record resource usage for each task. Store the collected resource data in the resource indicator library to support subsequent analysis and query.

[0104] Specifically, the resource indicator library includes task ID, resource type (CPU, memory, storage), resource consumption, and timestamp.

[0105] Optionally, synchronize the cost price list from the procurement system, clean and standardize the cost data, generate cost metadata, and record the procurement cost and resource cost of each table.

[0106] Synchronize cost data from the procurement system regularly through data synchronization tools (such as Sqoop or Kafka). Clean the synchronized cost data, including deduplication, format conversion, missing value processing, etc. Standardize the cost data into a unified format and unit to facilitate subsequent calculation and analysis. Calculate the procurement cost and resource cost of each table based on resource consumption data and cost price list.

[0107] The above describes the cost allocation method in the embodiment of the present invention. The following describes the cost allocation device in the embodiment of the present invention. Figure 7 In one embodiment of the present invention, a cost allocation device includes: The acquisition module 401 is used to acquire the call record of each terminal object to the payment data and the cost generated by the payment data. The call record includes the purchase table of the payment data.

[0108] The establishment module 402 is used to establish a blood relationship map from each purchase table to each terminal object based on the call record of the payment data. The blood relationship map is composed of associated multi-level path nodes, each purchase table is used as a primary path node in the blood relationship map, and each terminal object is used as an ultimate path node in the blood relationship map.

[0109] The calculation module 403 is used to calculate the path weights between the upper and lower level path nodes based on the data attributes between the upper and lower level path nodes.

[0110] The allocation module 404 is used to allocate the cost to each terminal object based on the blood relationship map and the path weight.

[0111] above Figure 7The characteristic feature extraction device in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the computer device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0112] Figure 8 1 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. The computer device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 may be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the computer device 500.

[0113] The computer device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 8 The illustrated computer device structure does not constitute a limitation on the computer device, and may include more or fewer components than illustrated, or combine certain components, or arrange the components differently.

[0114] The present invention also provides a computer device, which includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the steps of the cost allocation method in the above embodiments.

[0115] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes the steps of the cost allocation method.

[0116] Different from the existing technology, this application quantifies the dependencies between tables accurately by analyzing the data transmission relationship between paid data and terminal objects, ensuring that the allocation of procurement costs, computing costs and storage costs is highly accurate and traceable. At the same time, it supports dynamic weight calculation, and can adjust the cost allocation ratio in real time according to multi-dimensional factors such as data volume, field dependency and query complexity to adapt to complex data flow scenarios. The cost calculation of this application covers the full life cycle management of procurement costs, computing costs and storage costs, provides a complete cost allocation path from data sources to terminal objects, and supports comprehensive cost analysis and optimization. This application also provides visualization tools to display the cost transmission path, enhance transparency, help users intuitively understand the cost allocation logic, and optimize resource allocation. Through automated cost allocation and report generation, manual intervention is reduced and efficiency is improved. The system has good scalability, adapts to data environments of different scales, and ensures the flexibility and adaptability of cost management.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0119] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A cost allocation method, characterized in that: include: Acquire the call record of each terminal object for the payment data and the cost generated by the payment data; the call record includes a purchase list of the payment data; Based on the call record of the payment data, a blood relationship map from each purchase table to each terminal object is established; the blood relationship map is composed of associated multi-level path nodes, each purchase table serves as a primary path node in the blood relationship map, and each terminal object serves as an ultimate path node in the blood relationship map; Based on the data attributes between the upper and lower level path nodes, the path weights between the upper and lower level path nodes are calculated; The cost is apportioned to each of the terminal objects based on the blood relationship map and the path weight.

2. The cost allocation method according to claim 1, characterized in that: The establishing of a blood relationship map from each of the purchase tables to each of the terminal objects based on the call record of the payment data includes: Parsing the structured query language in the Hive log file through a syntax parser generator to extract the source table, target table and field mapping relationship in the structured query language; The blood relationship map is constructed based on the source table, the target table and the field mapping relationship.

3. The cost allocation method according to claim 2, characterized in that: The path weights include field-level dependency weights, data volume weights, and query structure weights; The calculating the path weights between the upper and lower level path nodes based on the data attributes between the upper and lower level path nodes includes: Based on the data attributes between the upper and lower level path nodes, respectively calculating the field level dependency weight, the data volume weight and the query structure weight between the upper and lower level path nodes; A weighted average of the field-level dependency weight, the data volume weight, and the query structure weight is calculated as the path weight.

4. The cost allocation method according to claim 3, characterized in that: The calculating the field-level dependency weight between the upper and lower level path nodes includes: Extracting the mapping relationship between the target table corresponding to the lower-level node and the source table corresponding to the upper-level node in the blood relationship map to obtain the field reference data of the target table to the source table; The field-level dependency weight between upper and lower level path nodes is calculated based on the referenced field weight in the field reference data and the total weight of the source table.

5. The cost allocation method according to claim 3, characterized in that: The calculating the data volume weight between the upper and lower level path nodes includes: In the preset target operation of the structured query language, the ratio of the source table data volume to the total data volume of the source table and the target table is calculated as the data volume weight between the upper and lower level path nodes.

6. The cost allocation method according to claim 1, characterized in that: The calculating the query structure weight between the upper and lower level path nodes includes: The weight coefficient corresponding to the operation of matching the structured query language in the preset weight coefficient table is used as the query structure weight between the upper and lower path nodes.

7. The cost allocation method according to claim 3, characterized in that: The cost includes procurement cost, storage cost and computing cost; Allocating the cost to each of the terminal objects based on the blood relationship graph and the path weight includes: Calculating the path weight of each terminal object for the purchase table based on the path weights between the upper and lower level path nodes; According to the path weight corresponding to each terminal object, the sum of the procurement cost, the storage cost and the computing cost is allocated to each terminal object.

8. A cost allocation device, characterized in that: include: An acquisition module, used to acquire the call record of each terminal object for the payment data and the cost generated by the payment data; the call record includes a purchase table of the payment data; An establishing module, for establishing a blood relationship map from each of the purchase tables to each of the terminal objects based on the call record of the payment data; the blood relationship map is composed of associated multi-level path nodes, each of the purchase tables serves as a primary path node in the blood relationship map, and each of the terminal objects serves as an ultimate path node in the blood relationship map; A calculation module, used to calculate the path weights between the upper and lower path nodes based on the data attributes between the upper and lower path nodes; An allocation module is used to allocate the cost to each of the terminal objects based on the blood relationship map and the path weight.

9. A computer device, characterized in that: The computer device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the computer device to perform the cost allocation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the cost allocation method according to any one of claims 1 to 7 is implemented.

Images