Unlock instant, AI-driven research and patent intelligence for your innovation.

A method for realizing a unified interface of multiple big data computing frameworks

A computing framework and big data technology, applied in the direction of creating/generating source code, etc., can solve the problems of full cluster load, unbalanced utilization of small clusters, and low utilization of resources.

Active Publication Date: 2019-03-29
珠海巧工科技有限公司
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, because different types of jobs / services require different amounts of resources, the utilization of these small clusters is usually very uneven. Some clusters are fully loaded and resources are tight, while others are idle for a long time and resource utilization is extremely low.
In addition, since different computing frameworks have different calling APIs, it is necessary to develop calling programs for each computing framework separately, resulting in extremely low development efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method for realizing a unified interface of multiple big data computing frameworks

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0006] Step 1: Implement a metadata management module, use the database to save the "technical metadata" and "business metadata" in hadoop for users and task analysis controllers to call.

[0007] "Business metadata" describes the data in the data warehouse from a business perspective. It provides a semantic layer between the user and the actual system, so that business personnel who do not understand computer technology can also "read" the data warehouse. The data. Users can access "business metadata" to know what business data is available;

[0008] "Technical metadata" is data that describes the technical details of a data warehouse that are used to develop, manage, and maintain the data warehouse. The system program (task analysis controller) can call "technical metadata" to know where the data is stored and which computing framework can be used to operate.

[0009] Step 2: Implement an interface layer based on the JDBC standard

[0010] It provides external interface s...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

A uniform hadoop calculation frame interface is established to have access to multiple calculation frames of hadoop through the uniform interface, a user has access to data in the frames such as Hive, HBase, spark and Impala through the JDBC interface in a transparent mode, the method for the uniform interface of the multiple hadoop calculation frames is achieved, and the user can have access to the data in the frames such as Hive, HBase, spark and Impala through the JDBC interface in the transparent mode.

Description

technical field [0001] Establish a unified Hadoop computing framework interface to access multiple computing frameworks of Hadoop with a unified interface. Users can transparently access data in frameworks such as Hive, HBase, spark, and Impala through the JDBC interface. Background technique [0002] In the era of big data, in order to store and process massive data, large-scale server clusters are required. Generally speaking, many types of applications and services are running on these clusters, such as offline jobs, streaming jobs, and iterative jobs. etc. Traditionally, each type of job or service corresponds to a separate cluster to avoid interfering with each other. In this way, the cluster is divided into a large number of small clusters, some running Hadoop, some running Spark, and so on. However, due to the different amount of resources required by different types of jobs / services, the utilization of these small clusters is usually very uneven, some clusters are ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F8/30
Inventor 柴满徐健王国辉
Owner 珠海巧工科技有限公司