A method and system for storing data in hbase
A technology of data and data files, applied in the field of big data, can solve the problems of high time and labor costs, limited input and reading speed, slow speed, etc., to improve data storage efficiency, improve operation and work speed, reduce The effect of operating errors
Active Publication Date: 2021-02-09
广州市申迪计算机系统有限公司
View PDF6 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Among them, the first method is the most commonly used method for storing data to HBase, but it is only suitable for a small amount of data, and it is difficult to input a large amount of data (such as 10GB of data). Not only is the speed very slow, but it also consumes a lot of HBase of cluster resources
The second method is a common method for large file entry, but because MapReduce is based on disk computing, the speed of data file entry and reading is limited by the physical characteristics of the disk; moreover, it needs to be specific for different types of data. Programming development, resulting in higher time and labor costs
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
[0024] refer to figure 1 , a system for storing data in HBase, at least including a first terminal 100 and a server 300 . The first terminal 100 may also be connected to the second terminal 200 . The first terminal 100 is used to send a file preview request to the second terminal 200 or the server 300, and generate a data preview interface in the browser 140 by reading part of the pre-stored data files; and is used to receive user settings through the browser...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to the technical field of big data, in particular to a method and system for storing data into HBase. Wherein, the method includes: the first terminal sends a file preview request to the second terminal or the server, and generates a data preview in the browser by reading part of the pre-stored data files; the first terminal receives the pre-stored data files set by the user through the browser The format parameter, in response to the user's parameter submission request, the format parameter is packaged into a preset format parameter, and the preset format parameter is sent to the server; the server-side Spark module converts the HDFS module according to the preset format parameter The pre-stored data files are stored in the HBase module. The invention enables users to input various data files into HBase through zero coding, effectively improves data storage efficiency, and reduces the possibility of operation errors; and can quickly import data files of GB level and above into HBase, Effectively improve the running and working speed.
Description
technical field [0001] The invention relates to the technical field of big data, in particular to a method and system for storing data into HBase. Background technique [0002] With the popularity of mobile Internet, Internet of Things, cloud computing, and smart cities, people have entered the era of big data, and Apache Hadoop has become the driving force behind the development of the big data industry. Hadoop provides a file system interface and the realization of multiple distributed file systems, wherein the more important ones are the Hadoop distributed file system (HDFS, Hadoop Distributed Filesystem) and the HBase module (330). HDFS is suitable for deployment on low-cost hardware, has high throughput, and can store massive data through calculation. In addition, HBase is built on Hadoop and is a distributed, column-oriented and scalable massive data storage database. Similarly, it can also be deployed on a large number of cheap machines to provide a distributed data...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/27G06F16/182G06F16/172
Inventor 覃立辉
Owner 广州市申迪计算机系统有限公司