Data ETL construction method and system based on BS and CS architecture

Abstract

The embodiment of the invention provides a data ETL construction method and system based on BS and CS architectures. The method comprises the following steps: creating a database of a city brain; creating a new WEB application, calling a Kettle source code integrated in the WEB application in the WEB application, generating a Kettle database resource library, and storing the Kettle database resource library; configuring a database resource library to the pre-created database; when tasks in the database resource library are scheduled according to a preset task scheduling method, obtaining data stored in the database correspondingly configured with the database resource library, and performing cleaning, conversion and loading operation. According to the method, the influence of the system environment on the convenience of using the Kettle is reduced, and the monitoring on the Kettle calling log is supported, so that a developer can more accurately monitor the Kettle task, master the working state of each task in the Kettle tool at any time, and ensure the stable operation of the Kettle tool.

Description

technical field [0001] The invention relates to the technical field of data ETL system of city brain, in particular to a data ETL construction method and system based on BS and CS architecture. Background technique [0002] Information is an important resource of modern enterprises and the basis for enterprises to use scientific management and decision-making analysis. At present, most enterprises spend a lot of money and time to build OLTP business systems and office automation systems to record various related data of transaction processing. According to statistics, the amount of data will double every 2-3 years. These data contain huge commercial value, but what enterprises pay attention to usually only accounts for about 2%-4% of the total data amount. Therefore, enterprises still do not maximize the use of existing data resources, so that more time and money are wasted, and the best opportunity to make key business decisions is lost. Therefore, how enterprises convert...

AI Technical Summary

Problems solved by technology

However, due to data security issues, high server requirements, high message transmission overhead, and lack of personalization of the software, it is difficult to achieve special functional requirements in the traditional mode.

It is difficult to realize complex application functions

[0006] Although commonly used data ETL tools have their own advantages, their disadvantages are also prominent. The disadvantages of tools based on C / S architecture: 1. Narrow applicability, usually used in local area networks; 2. Fixed user group

Because the program needs to be installed before it can be used, it is not suitable for some unknown users; 3. The maintenance cost is high, and once an upgrade occurs, all client programs need to be changed; 4. The client needs to be installed independently, and the cost of distribution, installation and upgrade is relatively high High; 5. Poor compatibility. For different operating systems, programs usually need to be rewritten; 6. Development and maintenance costs are high, and technical personnel with a certain professional level are required to complete

Disadvantages of tools based on B / S architecture: 1. Weak functions, unable to realize complex interface functions; 2. On cross-browser, B / S architecture is not satisfactory; 3. Most operations need to be carried out with the WEB server through HTTP Message interaction, user operation response is slow; 4. It takes a lot of effort to achieve the level of C / S program performance; 5. Huge design costs are required in terms of speed and security, which is the biggest problem of B / S architecture; 6. The interaction between the client and the server is a request-response mode, which usually requires refreshing the page, which is not what customers are happy to see

Images