Method and device for data table analysis and processing

An analysis processing and data table technology, applied in the field of big data processing, can solve the problems of resource consumption, high data cost, and inaccurate measurement of data usage cost, etc.

CN106991101BActive Publication Date: 2021-02-02ZHEJIANG TMALL TECH CO LTD
5 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Publication Date
2021-02-02

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The embodiment of the present application provides a method and device for data table analysis and processing, the data table includes a general data table of the data public layer, and an external data table of the non-data public layer, and the method includes: for the Calculate the processing cost data of the general data table of the data public layer; determine the general data table on which the external data table of the non-data common layer depends; calculate the use cost of the external data table according to the processing cost data of the general data table data, so that when evaluating the cost of each general-purpose data table in the data public layer, the current data table's own storage and computing consumption are no longer considered in isolation, but several upstream data of the data table will be considered comprehensively Tables and brother data tables, so as to be able to reasonably and accurately evaluate the processing cost of general data tables, so as to reflect the advantages and disadvantages of the data model construction of the data public layer, and provide decision support for the optimization and operation of the data public layer model.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present application relates to the technical field of big data processing, in particular to a data table analysis processing method and a data table analysis processing device. Background technique

[0002] The advent of the era of big data has highlighted the need for massive data storage, calculation, and processing, and the association and service between data is particularly important. These massive data are generally stored in cloud computing clusters in a structured or semi-structured form, such as Hadoop, ODPS, etc. The relationship between massive data is organized and reflected through the data tables stored in the cloud computing cluster, and mutual access, circulation and exchange are formed between different companies and different business departments within the same company, so as to truly Give full play to the value of data in the era of big data.

[0003] Among the thousands of data tables in the cloud computing environment, some ...

Examples

Embodiment Construction

[0224] In order to make the above objects, features and advantages of the present application more obvious and comprehensible, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0225] In the prior art, the processing cost of the data table is only measured by computing hardware resources (such as CPU consumption and memory consumption) and storage resources (storage medium consumption) consumed in the data processing process. However, for the generation of a data table, the data in it may come from the upstream N data tables, that is, the generation of a data table needs to rely on the upstream N data tables. However, the existing cost measurement model only analyzes the storage consumption and computing consumption generated during the processing of a current data table in isolation, and does not take into account the dependencies between data tables, thus ignoring the scanning...