Connection query system and method for distributed data warehouse
A distributed data and connection query technology, applied in the database field, can solve problems such as low resource utilization, achieve the effects of improving efficiency, reducing transmission, and improving resource utilization
Inactive Publication Date: 2014-03-12
NEC (CHINA) CO LTD
View PDF6 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Therefore, in a large-scale distributed environment, resource utilization will be relatively low
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0041] In the following, the principle and implementation of the present invention will become apparent by describing specific embodiments of the present invention in conjunction with the accompanying drawings. It should be noted that the present invention should not be limited to the specific examples described below. Also, detailed descriptions of well-known elements are omitted for brevity.
[0042] image 3 A block diagram of a shard join query system for a distributed data warehouse according to an embodiment of the present invention is shown. As an example, in image 3 1 master node 50, 3 map worker nodes 60, and 2 reducer worker nodes 70 are shown in , and like numbers in the figures denote like elements. However, it should be understood that the present invention can be applied to a distributed system including any number of mapping working nodes and any number of reducing working nodes. Generally speaking, the number of mapping working nodes is greater than the nu...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a connection query system and a connection query method for a distributed data warehouse. The system comprises a master node, mapping work nodes and reduction work nodes, wherein the master node calculates a fragment size according to the size of a data table and system performance, allocates a data block to the mapping work node based on the calculated fragment size, and formulates fragmentation mapping rules and summarization rules in the mapping work node; the mapping work node maps a query keyword in the data block to a corresponding fragment number according to the fragmentation mapping rules, and transmits data with the same fragment number to a specified reduction work node according to the summarization rules; and the reduction work node receives the data from the mapping work node, combines the data with the same fragment number and establishes connection according to the query keyword to obtain a final connection query result. By the system and the method, data transmission in a distributed system is reduced, the data volume and program complexity of the reduction work node are decreased, and the performance of the distributed data warehouse is improved.
Description
technical field [0001] The invention relates to database technology, in particular to a connection query system and method for distributed data warehouses. Background technique [0002] With the rapid development of information technology, the storage, retrieval and analysis of massive data has become very critical. The data warehouse came into being, and its usual definition is: a collection of subject-oriented, integrated, stable, and time-varying data used to support management decisions. The data warehouse has two levels of meaning. One is that it is used to support decision-making and is oriented to analytical data processing; the other is that it is composed of multi-source heterogeneous data, which is reorganized according to themes after integration and includes historical data. Large capacity, high performance, high availability, scalability, manageability, and on-demand services have become key indicators for measuring today's data warehouses and distributed file ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
Inventor 伍涛胡卫松刘晓炜齐红威
Owner NEC (CHINA) CO LTD