Method for classifying and evaluating ETL job scheduling resources in distributed environment

Abstract

The invention discloses a method for classifying and evaluating ETL job scheduling resources in a distributed environment. The method comprises the following steps: 1, determining an index system forevaluating the performance of an ETL server; 2, determining an ETL operation type; 3, clustering and analyzing the ETL server index data based on an index system to obtain an ETL server candidate setclassified corresponding to the ETL operation type; 4, establishing an index evaluation matrix, and calculating the information entropy of the index and the weight of the information entropy; 5, sorting the ETL server candidate sets of various types; and 6, calculating ETL jobs according to the step 2 to determine ETL job types, and then selecting an ETL server with a top sequence from the ETL server candidate set of the corresponding classification. According to the method, the clustering analysis method and the evaluation method based on the information entropy are adopted to evaluate the performance of the ETL server, the ETL jobs which do not enter the execution state are automatically matched, the idle resources of the ETL server can be fully utilized, and the computing resources aredynamically distributed in a quasi-real-time manner.

Description

technical field [0001] The invention belongs to the field of network technology, in particular to a method for classifying and evaluating ETL job scheduling resources in a distributed environment. Background technique [0002] The rapid development of information technology has also brought about the vigorous rise of data construction. The scale of data warehouses has gradually increased, and the system structure has become more complex. The most important link in the construction of data warehouses is ETL data introduction, including data extraction and conversion. , The process of loading accounts for 80% of the workload of the data warehouse construction process. The distributed data connection method is the mainstream technology of data connection at present, and the data connection operation can be distributed to relatively cheap computer clusters. Reasonable allocation of ETL jobs and full use of ETL server computing resources. In a medium-scale data warehouse in the ...

AI Technical Summary

Problems solved by technology

In a medium-scale data warehouse in the field of network operation and maintenance, in order to meet the needs of various data analysis, daily, weekly, monthly and other data statistics, the number of ETL connection tasks that are ready for data connection is generally maintained at more than 100 , The ETL server cluster is generally composed of about twelve ordinary computers or virtual machines. If the strategy of "who is idle, who handles it" is simply adopted, then the ETL server cluster will be a waste of resources, such as figure 1 shown

[0003] Under normal circumstances, the tasks of data connection mainly include three types: data extraction, data conversion, and data loading. The consumption of CPU, memory, and I / O operation resources is somewhat heavy. If an ETL server is engaged in ETL operations for a long time It takes up a lot of I / O operation resources, and the CPU computing resources are still idle, which is a great waste for ETL jobs that require CPU computing resources. Similarly, for a computer that is doing a lot of memory-occupied operations , I / O operation idleness is also a waste of resources for ETL jobs that require simple data replication between tables. If the idle resources of ETL jobs and ETL servers can be accurately matched, ETL tasks will be greatly improved. Execution efficiency, which requires us to first determine the type of ETL job, then accurately evaluate the service capacity of the idle resources of the ETL server, and finally accurately match the ETL job to be executed and the ETL server, such as figure 2 shown

Images