Data transmission method and device based on Sqoop, computer readable storage medium and equipment

A data transmission method and technology of a data transmission device, which are applied in the field of data transmission, can solve the problems of slow speed and affect the operation efficiency of the overall system, and achieve the effect of increasing the degree of parallelism and improving the import speed.

Inactive Publication Date: 2019-07-16
AEROSPACE INFORMATION
View PDF4 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

But by default, the speed of importing using Sqoop is very slow, which seriously affects the operating efficiency of the overall system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data transmission method and device based on Sqoop, computer readable storage medium and equipment
  • Data transmission method and device based on Sqoop, computer readable storage medium and equipment
  • Data transmission method and device based on Sqoop, computer readable storage medium and equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0013] Implementing any technical solution of the embodiments of the present invention does not necessarily need to achieve all the above advantages at the same time.

[0014] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only some of the embodiments of the present invention, but not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the embodiments of the present invention shall fall within the protection scope of the embodiments of the present invention.

[0015] The specific implementation of the embodiments of the present invention will be further described below in conjunctio...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a data transmission method and device based on Sqoop, a computer readable storage medium and equipment. The data transmission method based on the Sqoop comprises the following steps: in a process of importing data of an Oracle database into a Hive database by utilizing the Sqoop, setting a parallelism parameter-m in the Sqloop to be greater than 1; judgingwhether a source table of the Oracle database has primary keys used for uniformly segmenting columns or not; if the main key does not exist, utilizing rownum in an Oracle database to generate a rownumcolumn through a sub-query statement, and performing data segmentation on the rownum; and enabling the Sqoop to import a single query into statements, generating a plurality of corresponding query statements according to a segmentation result, uniformly distributing the query statements to each machine node for operation, and finally performing a merging operation. According to the embodiment ofthe invention, the data importing speed is greatly improved.

Description

technical field [0001] The embodiments of the present application relate to the field of data transmission, and in particular to a Sqoop-based data transmission method, device, computer-readable storage medium, and equipment. Background technique [0002] With the continuous advancement of the digitalization process, tax authorities need to make full use of information technology to strengthen the follow-up monitoring and management of value-added tax invoices, track and analyze the use of taxpayers' invoices and tax declarations while doing a good job in tax payment services and improving tax handling efficiency. . [0003] The current difficulty is that there is a large amount of value-added tax invoice information every day, especially with the advancement of business tax reform, the number of value-added tax invoices has increased sharply. Traditional relational databases have been difficult to meet the analysis function of massive data. At this time, it is necessary t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/28G06F16/2453G06F16/2455
CPCG06F16/24532G06F16/24556G06F16/284
Inventor 果然范钢吴伟刚潘竞旭张玉魁宋颖
Owner AEROSPACE INFORMATION
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap