Data sheet analyzing and processing method and device

Abstract

The invention provides a data sheet analyzing and processing method and device. The data sheets comprise a common data sheet on a data common layer and an external data sheet on a non data common layer. The data sheet analyzing and processing method comprises calculating the processing cost data of a common data sheet on a data common layer; determining a common data sheet which the external data sheet of the non data common layer depends on; and calculating the usage cost data of the external data sheet on the basis of the processing cost data of the common data sheet. Thus, when the cost of each common data sheet on the data common layer is evaluated, self storage and computational consumption of the current data sheet are not individually considered, several upstream data sheets and brother data sheets of the data sheet are comprehensively considered, and therefore the processing cost of the common data sheet can be evaluated reasonably and accurately, the quality of the data model construction of the data common layer can be reflected, and decision support can be provided for optimization and operation of the data common layer model.

Description

technical field [0001] The present application relates to the technical field of big data processing, in particular to a data table analysis processing method and a data table analysis processing device. Background technique [0002] The advent of the era of big data has highlighted the need for massive data storage, calculation, and processing, and the association and service between data is particularly important. These massive data are generally stored in cloud computing clusters in a structured or semi-structured form, such as Hadoop, ODPS, etc. The relationship between massive data is organized and reflected through the data tables stored in the cloud computing cluster, and mutual access, circulation and exchange are formed between different companies and different business departments within the same company, so as to truly Give full play to the value of data in the era of big data. [0003] Among the thousands of data tables in the cloud computing environment, some ...

AI Technical Summary

Problems solved by technology

[0005] In the prior art, the processing cost of the data table is only measured by the computing hardware resources (such as CPU consumption, memory consumption) and storage resources (storage medium consumption) consumed in the data processing process, that is, only isolated Analyze the storage consumption and calculation consumption generated during the processing of the current data table

For the cost of using the data table, the data processing cost of the used table is evenly distributed to each user of this data table, which is obviously not fair and reasonable.

This will inevitably lead to the problem that neither the measurement of data processing cost nor the measurement of data usage cost in the existing technology is accurate enough, which will seriously affect the judgment of data validity in the cloud computing environment, resulting in excessively high data costs, and too many unnecessary resource consumption

Images