Data transmission method and device based on cloud storage and computer equipment

A data transmission method and a technology of a data transmission device, which are applied in the field of cloud storage and can solve problems such as low data writing efficiency and the impact of HBase cluster data processing efficiency

Pending Publication Date: 2020-05-01
PING AN TECH (SHENZHEN) CO LTD
View PDF2 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The embodiment of the present invention provides a data transmission method, device, computer equipment, and storage medium based on cloud storage, aiming to solve the problem that in the prior art, when data is written into HB

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data transmission method and device based on cloud storage and computer equipment
  • Data transmission method and device based on cloud storage and computer equipment
  • Data transmission method and device based on cloud storage and computer equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0033] It should be understood that when used in this specification and the appended claims, the terms "comprising" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude one or Presence or addition of multiple other features, integers, steps, operations, elements, components and / or collections thereof.

[0034] It should also be understood that the terminology used ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data transmission method and device based on cloud storage, computer equipment and a storage medium. The method comprises the following steps: receiving full data uploaded bya Hive database, and storing the full data; obtaining the number of pre-partitions in an HBase database; partitioning the total data according to the number of the pre-partitions and the row keys ofthe data in the total data to obtain corresponding partition data; performing ascending sorting on each part of partition data according to the column and row keys in sequence to obtain correspondingsorted partition data; and sending the sorted partition data to a partition server corresponding to the Hbase database for storage. According to the method, the sorting process is completed at the cloud before the total data is written into the Hbase database, so that the efficiency of writing the data into the Hbase database is improved.

Description

technical field [0001] The present invention relates to the technical field of cloud storage, in particular to a data transmission method, device, computer equipment and storage medium based on cloud storage. Background technique [0002] At present, when data in the Hive database (Hive is a data warehouse tool that can map structured data files into a database table) is written into HBase (HBase is a distributed, column-oriented open source database), Generally, offline batch writing or streaming writing is used, but the above two methods both use put when writing data into HBase (put is one of the data insertion methods in HBase), and insert through the put command The data is sorted and inserted at the same time, which affects the data processing efficiency of the HBase cluster and leads to low data writing efficiency. Contents of the invention [0003] The embodiment of the present invention provides a data transmission method, device, computer equipment, and storage ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/22G06F16/2455G06F16/27G06F16/28
CPCG06F16/221G06F16/24554G06F16/278G06F16/283
Inventor 邓煜
Owner PING AN TECH (SHENZHEN) CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap