Method and device for processing dirty data in data exchange operation

A data exchange and processing method technology, applied in the computer field, can solve problems affecting the efficiency of data exchange operations, etc., and achieve the effect of facilitating processing and troubleshooting, realizing continuous progress, and improving processing efficiency

Pending Publication Date: 2020-04-03
HANGZHOU DT DREAM TECH
View PDF6 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In the existing technology, the ETL exchange system can extract and exchange data from the source database in a single-threaded or multi-threaded concurrent manner. Once dirty data occurs during the data exchange process, such as the destination field length does not match the destination database. In other cases, the data exchange job will be interrupted, and dirty data needs to be found from the massive data in the source database before the data exchange job can be started, which affects the efficiency of the data exchange job

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for processing dirty data in data exchange operation
  • Method and device for processing dirty data in data exchange operation
  • Method and device for processing dirty data in data exchange operation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0062] Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present application as recited in the appended claims.

[0063] The terminology used in this application is for the purpose of describing particular embodiments only, and is not intended to limit the application. As used in this application and the appended claims, the singular forms "a", "the", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a method and a device for processing dirty data in data exchange operation. The method comprises the following steps: receiving dirty data information uploaded by any one or more working nodes in a working node set; adding the dirty data information into a dirty data sample set, wherein the dirty data sample set comprises the dirty data information received from the workingnode set; and analyzing the dirty data sample set to obtain a detection rule for a destination database, the detection rule being used for being issued to the working node set, so that the working nodes in the working node set predict dirty data information in the data information to be provided to the destination database according to the detection rule. By means of the technical scheme, the dirty data information can be rapidly checked, even if dirty data occurs, interruption of data exchange operation cannot be caused, continuous operation of data exchange operation of the working nodes canbe achieved, the dirty data processing efficiency in the data exchange process can be improved, and the data exchange operation efficiency is effectively improved.

Description

technical field [0001] The present disclosure relates to the field of computers, and in particular to a method and device for processing dirty data in a data exchange operation. Background technique [0002] ETL (Extract-Transform-Load, data warehouse technology) is used to describe the process of extracting, transforming, and loading data from the source to the destination. Usually, in order to improve the switching capability of the ETL switching system and support more data source access, the ETL switching system is usually deployed in the form of a cluster, and the control node manages one or more working nodes in a unified manner. The control node can The exchange job is assigned to one or more work nodes, and the work nodes can be expanded horizontally to perform specific data exchange work. The work nodes are connected to the source database and the destination database. [0003] In the existing technology, the ETL exchange system can extract and exchange data from t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25G06F16/27G06F16/215G06F16/28
CPCG06F16/254G06F16/27G06F16/215G06F16/283
Inventor 陈振洲
Owner HANGZHOU DT DREAM TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap