Unlock instant, AI-driven research and patent intelligence for your innovation.

Processing connection query method and device

A connection query and connection key technology, applied in the direction of electrical digital data processing, special data processing applications, digital data processing parts, etc., can solve the problems of time consumption, long time, low connection query efficiency, etc., to reduce time and improve efficiency effect

Active Publication Date: 2018-03-09
HUAWEI TECH CO LTD
View PDF4 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, in the process of implementing the above method, when the key values ​​of the data in the data block are scattered, the process of bucketing the data in the map stage requires a lot of calculations, which takes a long time, and because each data There are multiple buckets of data in the block, and a large amount of network connection overhead and data transmission overhead will be generated during the shuffle process. figure 1 , the data in each data block needs to be transmitted to three different reduce nodes, and the transmission process takes a certain amount of time, which eventually leads to low efficiency of connection query

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Processing connection query method and device
  • Processing connection query method and device
  • Processing connection query method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0032] In order to improve the efficiency of connection query, an embodiment of the present invention provides a method for processing connection query, which can be specifically applied to a clustered big data analysis system, such as figure 2 As shown, the system includes Client (client), Metastore (metadata storage unit), a master node (master), multiple work nodes (workers) and DFS (Distribute File System, distributed file system).

[0033] Among them, th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a processing connection query method and device and relates to the technical field of communication. The problem of low connection query efficiency can be solved. The processing connection query method comprises the steps that a frequent table combination is determined, wherein the table combination is a table combination having the occurrence frequency greater a preset value in a historical query record and comprises connecting keys and tables connected through the connecting keys; cluster indexes are created according to the connecting key information in the frequent table combination, riffling operation is conducted on cluster columns in the cluster indexes, records having the same index column values are stored in at least one data block to form a table cluster corresponding to the frequent table combination. The processing connection query method is suitable for table connection query.

Description

technical field [0001] The invention relates to the field of communication technology, in particular to a method and device for processing connection queries. Background technique [0002] The rapid development of network technology has led to a sharp increase in the amount of data. In order to efficiently process large-scale data, a distributed computing framework based on MapReduce (map-reduce) can be used for query and analysis tasks of big data. However, due to the When performing query and analysis tasks in a distributed computing framework, complex programs need to be written for each task. For complex queries such as OLAP (On-Line Analytical Processing, Online Analytical Processing), the implementation process is more complicated and the ease of use is lower. Low. In contrast, SQL (Structured Query Language, Structured Query Language) is relatively easy to use, so SQL is usually applied to a distributed computing framework based on MapReduce for big data query analys...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/245G06F16/2255G06F16/2282G06F16/2393G06F16/00G06F16/285G06F16/24544G06F7/36
Inventor 王振华
Owner HUAWEI TECH CO LTD