Unlock instant, AI-driven research and patent intelligence for your innovation.

A data loading and cleaning engine, scheduling and storage system

A data loading and storage system technology, applied in the computer field, can solve problems such as functional docking deviation, ETL processing method is embarrassing, and processing speed is difficult to meet processing requirements, etc., to improve performance, improve manageability and usability, and be easy to promote Effect

Active Publication Date: 2019-11-01
广东省信息网络有限公司
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The functions of these traditional ETL systems have been developed relatively well, but when dealing with scenarios with large amounts of data, it is difficult to meet the processing requirements in terms of processing speed, and there are many deviations in function connection, which makes the traditional ETL processing methods embarrassing and heavy

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data loading and cleaning engine, scheduling and storage system
  • A data loading and cleaning engine, scheduling and storage system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0021] Such as Figure 1 to Figure 2 As shown, it is a data loading and cleaning engine, a scheduling and storage system of the present invention, including a data source, a data warehouse and a user display module. The data warehouse is connected with an ETL management module, and the ETL management module includes an ETL scheduling module, an ETL monitoring module, and a data The quality module and the ETL task module, the ETL scheduling module is used to control the operation of all ETL tasks, the ETL scheduling module is connected with the time setting module, each task can be set when to execute, so that each task can be executed at the specified time It runs automatically at all times, and the execution cycle of the task is very different. Some define the time interval (such as executing once every 3 minutes), and some define a certain time (such as starting to execute at 21:00 every Friday night) , for determining the time, it can be divided into many ways such as year,...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data loading cleaning engine, dispatching and storage system, which comprises a data source, a data warehouse and a user display module, wherein the data warehouse is connected with an ETL management module; the ETL management module comprises an ETL dispatching module, an ETL monitoring module, a data quality module and an ETL task module; the data warehouse comprises an interface file region, a detail data temporary storage region SSA, a detail data SOR, a data mart, a data summarizing module, a feedback module and a metadata storage MDR. The system provided by the invention has the advantages that the practicability is high; the data management is convenient and fast; the flexibility is high; the popularization is easy; the high-efficiency data processing is realized; the throughput is great; the dealing with the addition of more data sources can be realized; more analysis requirements are supported.

Description

technical field [0001] The invention belongs to the technical field of computers, and in particular relates to a data loading and cleaning engine, scheduling and storage system. Background technique [0002] With the rapid development of big data technology and the advancement of informatization, the amount of data accumulated by human society has exceeded the sum of the past 5,000 years, and the amount of massive data collection, storage, processing and dissemination is also increasing day by day. The realization of data sharing by enterprises can enable more people to use existing data resources more fully, and reduce duplication of labor and corresponding costs such as data collection and data collection. However, in the process of implementing data sharing, since the data provided by different users may come from different channels, the data content, data format and data quality vary widely, and sometimes even the data format cannot be converted or the data is lost after...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25
CPCG06F16/254
Inventor 孙永剑郑书礼裘鑫芳董磊
Owner 广东省信息网络有限公司