Unlock instant, AI-driven research and patent intelligence for your innovation.

A data storage method, system, device and storage medium based on big data

A data storage and big data technology, applied in the field of big data, can solve the problem of insufficient number of columns in the BigTable storage model, and achieve the effect of eliminating overhead

Active Publication Date: 2021-10-01
GUANGDONG POWER GRID CO LTD +1
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

For example, in a prefecture-level city of 7.2 million, the number of smart meters has reached 1.5 million. If a column is assigned to each measuring device and the data of each column is stored in the same data list, the connection between columns can be eliminated ( JOIN) calculation overhead, and then meet the requirements of efficient storage and high-performance analysis of massive data, but the table width of the BigTable storage model is far lower than the number of measurement devices, that is, the number of columns of the BigTable storage model is still insufficient

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data storage method, system, device and storage medium based on big data
  • A data storage method, system, device and storage medium based on big data
  • A data storage method, system, device and storage medium based on big data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] The core of the present invention is to provide a data storage method based on big data, which can store massive data with any number of columns, that is, realizes unlimited expansion of the width of the data storage list.

[0042] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0043]The applicant found that BigTable adopts the Key-value implementation mechanism, and the design supports one-dimensional expansion, that is, the data storage table can be cut ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data storage method based on big data: according to business parameters and hardware parameters of each computing cluster, plan the number of data columns stored for each computing cluster; According to the number of data columns stored in each computing cluster, the SuperTable is divided vertically into multiple target list Cabinets; routing information is constructed so that any column in the SuperTable can be read and written Locate to the corresponding computing cluster during operation. By applying the method provided by the present invention, massive data with any number of columns can be stored, and the unlimited expansion of the width of the data storage list is realized. The invention also discloses a big data-based data storage system, equipment and storage medium with corresponding technical effects.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a data storage method, system, device and storage medium based on big data. Background technique [0002] With the rapid development of today's society, with the advent of the cloud era, big data has attracted more and more attention. In more and more fields, big data has been widely used. [0003] A notable feature of big data is that the amount of data is very large. For example, the data of the Internet of Things is a kind of massive and explosively growing big data. For these massive data, in order to save physical storage space and improve data analysis efficiency, columnar storage in relational databases is usually used. [0004] For example, Vertica, as a representative columnar database storage model, has a performance improvement of 10 to 50 times compared to traditional row-based database engines, but it currently supports a table width of up to 1600 columns. ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/27
Inventor 郭文鑫王海柱赵瑞锋卢建刚徐展强邓大为
Owner GUANGDONG POWER GRID CO LTD