Data processing method, apparatus and system

Abstract

The invention discloses a data processing method, apparatus and system to avoid the problem of excessively frequent IO reading-writing operations on a disk or a memory caused by the problem of dependency on deadlock during big data analysis processing based on an ETL technology. The method comprises the steps of determining nodes, which meet the condition, in all nodes comprised in an ETL system; for each determined node which meets the condition, selecting part of non-blocking nodes from all non-blocking nodes existent on a transmission path passed by a non-blocking data source received by the node, and modifying the selected non-blocking nodes to blocking nodes; and / or storing the non-blocking data source received by the node in the local side of the node. By adopting the method, the problem of dependency on the deadlock state due to the data analysis processing based on the ETL technology can be solved with relatively low performance loss, so that the problem of greatly reduced big data analysis performance due to the excessively frequent IO reading-writing operations on the disk or the memory can be avoided.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data processing method, device and system. Background technique [0002] Data extraction-transform-load (Extract-Transform-load, ETL) is used to implement the process of extracting, transforming, and loading the data to be analyzed from the source to the destination. ETL is more commonly used in data warehouses. As an important part of building a data warehouse, users extract the required data from the data source, after data cleaning, and finally load the data into the data warehouse according to the predefined data warehouse model. [0003] The system based on ETL technology includes three types of nodes for data extraction, data conversion and data loading. Each node is used to complete different functions, and each node is connected by a connection. Represents the specific data flow direction, and nodes with different functions are logical nodes used to com...

AI Technical Summary

Problems solved by technology

[0009] Embodiments of the present invention provide a data processing method, device, and system, so as to avoid the problem of dependency deadlock when analyzing and processing big data based on ETL technology, resulting in too frequent IO read and write operations on the disk or memory, resulting in large The problem that the performance of data analysis is greatly reduced

Images