A data processing method, device and system
A data processing device and data processing technology, applied in the field of data processing, can solve problems such as frequent read and write operations, dependent deadlocks, and performance degradation of big data analysis
Active Publication Date: 2019-08-20
HUAWEI TECH CO LTD
View PDF7 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0009] Embodiments of the present invention provide a data processing method, device, and system, so as to avoid the problem of dependency deadlock when analyzing and processing big data based on ETL technology, resulting in too frequent IO read and write operations on the disk or memory, resulting in large The problem that the performance of data analysis is greatly reduced
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
[0051] The embodiment of the present invention provides a data processing method and device to avoid the dependency deadlock problem in the prior art when analyzing and processing big data based on ETL technology, which causes too much IO read and write operations on disk or memory. Frequently, the performance of big data analysis is greatly reduced. Wherein, the method and the device are based on the same inventive concept, and since the princ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
Disclosed are a data processing method, apparatus, and system, so as to avoid the problem of excessively frequent IO read-write operations on a magnetic disk or a memory caused by deadlock dependency that appears during big-data analysis processing based on the ETL technology. The method comprises: determining nodes meeting a condition from all nodes comprised in an ETL system; for each determined node meeting the condition, selecting partial non-blocking nodes from all non-blocking nodes existing on a transmission path on which a non-blocking data source received by the node passes through, and modifying the selected non-blocking nodes into blocking nodes; and / or storing the non-blocking data source received by the node in the node locally. Therefore, by using the method in the present invention, the deadlock dependency state that appears during data analysis processing based on the ETL technology can be avoided with relatively low performance loss, so that the problem that big-data analysis performance greatly decreases due to excessively frequent IO read-write operations on a magnetic disk or a memory can be resolved.
Description
technical field [0001] The present invention relates to the technical field of data processing, in particular to a data processing method, device and system. Background technique [0002] Data extraction-transform-load (Extract-Transform-load, ETL) is used to implement the process of extracting, transforming, and loading the data to be analyzed from the source to the destination. ETL is more commonly used in data warehouses. As an important part of building a data warehouse, users extract the required data from the data source, after data cleaning, and finally load the data into the data warehouse according to the predefined data warehouse model. [0003] The system based on ETL technology includes three types of nodes for data extraction, data conversion and data loading. Each node is used to complete different functions, and each node is connected by a connection. Represents the specific data flow direction, and nodes with different functions are logical nodes used to com...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25
Inventor 朱玉麒
Owner HUAWEI TECH CO LTD