Method and device for data table analysis and processing

Abstract

The embodiment of the present application provides a method and device for data table analysis and processing, the data table includes a general data table of the data public layer, and an external data table of the non-data public layer, and the method includes: for the Calculate the processing cost data of the general data table of the data public layer; determine the general data table on which the external data table of the non-data common layer depends; calculate the use cost of the external data table according to the processing cost data of the general data table data, so that when evaluating the cost of each general-purpose data table in the data public layer, the current data table's own storage and computing consumption are no longer considered in isolation, but several upstream data of the data table will be considered comprehensively Tables and brother data tables, so as to be able to reasonably and accurately evaluate the processing cost of general data tables, so as to reflect the advantages and disadvantages of the data model construction of the data public layer, and provide decision support for the optimization and operation of the data public layer model.

Description

technical field [0001] The present application relates to the technical field of big data processing, in particular to a data table analysis processing method and a data table analysis processing device. Background technique [0002] The advent of the era of big data has highlighted the need for massive data storage, calculation, and processing, and the association and service between data is particularly important. These massive data are generally stored in cloud computing clusters in a structured or semi-structured form, such as Hadoop, ODPS, etc. The relationship between massive data is organized and reflected through the data tables stored in the cloud computing cluster, and mutual access, circulation and exchange are formed between different companies and different business departments within the same company, so as to truly Give full play to the value of data in the era of big data. [0003] Among the thousands of data tables in the cloud computing environment, some ...

AI Technical Summary

Problems solved by technology

[0005] In the prior art, the processing cost of the data table is only measured by the computing hardware resources (such as CPU consumption, memory consumption) and storage resources (storage medium consumption) consumed in the data processing process, that is, only isolated Analyze the storage consumption and calculation consumption generated during the processing of the current data table

For the cost of using the data table, the data processing cost of the used table is evenly distributed to each user of this data table, which is obviously not fair and reasonable.

This will inevitably lead to the problem that neither the measurement of data processing cost nor the measurement of data usage cost in the existing technology is accurate enough, which will seriously affect the judgment of data validity in the cloud computing environment, resulting in excessively high data costs, and too many unnecessary resource consumption

Images