Data processing framework system

Abstract

A data processing framework system is applicable to a client device and includes a communication device, a storage device, and a processing device. The storage device stores a data processing framework platform, which includes a data processing framework and multiple plug-ins. The data processing framework includes multiple core instructions and multiple common modules. The plug-ins include multiple plug-in instructions. The processing device is electrically connected to the communication device and the storage device, and forms an electrical connection with the client device through the communication device, enabling the client device, operated by a user, to develop a corresponding data processing project using the core instructions, the common modules, and the plug-ins.

Claims

1. A data processing framework system, applicable to a user terminal device, comprising: a communication device for providing networking functionality; a storage device for storing a Data ETL Framework platform, the data processing framework platform including a data processing framework and multiple plugins, the data processing framework including multiple core commands and multiple shared units, the plugins including multiple plugin commands; and a processing device electrically connected to the communication device and the storage device, and forming an electrical connection with the user terminal device through the communication device, enabling the user terminal device, operated by a user, to develop a corresponding data processing project using the core commands, the shared units, and the plugins.

2. The data processing framework system as described in claim 1, wherein, When the user is a framework developer, the client device allows the user to develop the core of the entire framework. After the core development is completed and subsequent related plugins are developed, the data processing framework platform is formed. Alternatively, when the user is a command user, the client device allows the user to install the core data processing framework and the corresponding plugins according to their needs, and actually execute the corresponding command operations provided by the plugins to complete the corresponding data processing project.

3. The data processing framework system as described in claim 2, wherein, This data processing framework platform implements a plug-in architecture using instructions.

4. The data processing framework system as described in claim 3, wherein, The data processing framework platform uses Pluggy to implement the plugin architecture.

5. The data processing framework system as described in claim 1, wherein, When the user is a plugin developer, the client device implements the plugin according to the user's needs, thereby enabling the core framework of the data processing framework platform to collect each function (i.e. the plugin developed and implemented) in a hierarchical manner and provide it to subsequent users.

6. The data processing framework system as described in claim 1, wherein, The data processing framework platform also defines a consistent recording format for each shared module and each plugin, so that each shared module and each plugin can output a standardized log.

Images