Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

ETL scheduling method and device

A scheduling method and job scheduling technology, applied in the field of data processing, can solve the problems of manual triggering or timing triggering, low processing efficiency, long development cycle, etc., and achieve the effect of solving the long release cycle, realizing dynamic correction, and improving reliability

Pending Publication Date: 2020-09-15
INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF4 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The first method has high labor costs; the development cycle is long, and every time a requirement changes or a problem is corrected, a new version needs to be formulated; it is difficult to meet the business needs of dynamic and high-frequency changes Requirements; the second method, in order to ensure the integrity of the data, it is necessary to determine whether the T-1 or T data is due (T represents a natural day), and then process, count and visualize the data; in the case of frequent statistics, Additional repetitive workload is added, and the processing efficiency is low; manual triggering or timing triggering is required, and the timeliness of data processing is low

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • ETL scheduling method and device
  • ETL scheduling method and device
  • ETL scheduling method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0049] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of the application will be clearly and completely described below in conjunction with the drawings in the embodiments of the application. Obviously, the described The embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0050] When applying existing data mining tools, the tool personnel cannot predict the batch date of ETL scheduling data. In order to ensure the integrity of the data, it is necessary to use manual triggering or timing triggering to ensure that T-1 or T data expires; manual The trigger method needs to wait for the execution result, the timeliness of data proce...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides an ETL scheduling method and device, and the ETL scheduling method comprises the steps: obtaining a target SQL statement list regularly, wherein the target SQL statement list comprises a plurality of SQL statements, and each SQL statement comprises a corresponding statement type identifier; if it is determined that the SQL statements belonging to the operation type contain the sub-queries according to the statement type identifiers of all the SQL statements, generating a blood relationship comparison relation table based on the identifiers of the source tables and the identifiers of the target tables of the SQL statements containing the sub-queries; and sending the blood relationship comparison relation table and the target SQL statement list to a target scheduling server, so that the target scheduling server executes target scheduling operation according to the mapping relationship table, the target data platform information, the blood relationship comparison relation table and the target SQL statement list. The ETL scheduling method can improve the ETL scheduling accuracy, flexibility and efficiency, can realize dynamic correction of the data model and visual display of the scheduling result, and can improve the data processing reliability.

Description

technical field [0001] The present application relates to the technical field of data processing, in particular to an ETL scheduling method and device. Background technique [0002] Traditional ETL (Extraction-Transformation-Loading) scheduling refers to cleaning and integrating scattered data and data from heterogeneous data sources and then loading them into big data platforms, data platforms, and data marts, which can be used for online analysis and processing. and data mining to provide decision support and data support. The ETL scheduling process is very complicated, not only the sequence and dependencies of each step need to be considered, but also the sequence and dependencies between different jobs need to be considered. [0003] In the fields of precision marketing, product innovation, risk control, market analysis, and resource allocation, raw data can be processed through ETL scheduling, which can provide data support for company management decision-makers, risk ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/25G06F16/242
CPCG06F16/254G06F16/2433Y02D10/00
Inventor 沈梦婷孙丽娜张文翰朱姣姣
Owner INDUSTRIAL AND COMMERCIAL BANK OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com