Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing equipment, data processing method and computer readable storage medium

A data processing device and data processing technology, applied in the field of data processing, can solve the problems that data processing does not support deduplication operations, complex data updating, and high maintenance costs

Active Publication Date: 2017-09-26
NUBIA TECHNOLOGY CO LTD
View PDF4 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The technical problem to be solved by the present invention is how to solve the problem that data processing using HIVE components in the prior art does not support deduplication operations, which leads to complex data update and high maintenance costs; in view of this technical problem, a data processing method is provided The device is characterized in that the data processing device includes a processor, a memory and a communication bus:

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing equipment, data processing method and computer readable storage medium
  • Data processing equipment, data processing method and computer readable storage medium
  • Data processing equipment, data processing method and computer readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0058] refer to figure 1 , figure 1 It is a schematic composition diagram of a data processing device based on Hadoop (distributed system infrastructure) provided by the first embodiment of the present invention.

[0059] The data processing device in this embodiment includes:

[0060] Confirmation module 101, for confirming deployed HIVE (data warehouse infrastructure) and HBASE (Haydup database);

[0061] The receiving module 102 is configured to receive the first control command corresponding to the creation and modification table sent by the client;

[0062] An analysis module 103, configured to analyze the first control command, determine whether deduplication is required, and generate a determination result;

[0063] The processing module 104 is configured to correspondingly create a table in HIVE and / or HBASE according to the judgment result.

[0064] This embodiment involves units such as server, client, and HIVE components, HBASE components, etc., wherein the mode...

no. 2 example

[0081] refer to image 3 , image 3 It is a flow chart of the Hadoop-based data processing method provided by the second embodiment of the present invention.

[0082] The data processing method in this embodiment includes:

[0083] S301. Confirm the deployed HIVE and HBASE;

[0084] S302. Receive the first control command corresponding to the creation and modification table sent by the client;

[0085] S303. Analyzing the first control command, judging whether deduplication is required, and generating a judging result;

[0086] S304, correspondingly create a table in HIVE and / or HBASE according to the judgment result.

[0087] In S301, the deployed HIVE and HBASE are confirmed. These two databases can be integrated into one component, which is generally deployed on the server. This component includes server and client respectively. The server mainly completes the calls with HBASE and HIVE structures, and completes functions such as creation, deletion, and execution of S...

no. 3 example

[0100] refer to Figure 4 , Figure 4 It is a schematic diagram of the composition of a data processing device provided in the third embodiment of the present invention. The data processing device in this embodiment includes:

[0101] Processor 401, memory 402 and communication bus 403;

[0102] The communication bus 403 is used to realize connection and communication between the processor 401 and the memory 402;

[0103] The processor 401 is used to execute the data processing program stored in the memory 402, to realize:

[0104] Confirmation of the deployed data warehouse infrastructure and Haidup database;

[0105] receiving the first control command corresponding to the modification creation table sent by the client;

[0106] Analyzing the first control command, judging whether deduplication is required, and generating a judging result;

[0107] Correspondingly create tables in the data warehouse infrastructure and / or Haidup database according to the judgment results...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides data processing equipment, a data processing method and a computer readable storage medium. The data processing method comprises the steps of determining an already-deployed data warehouse basic framework and a Hadoop database, receiving a first control instruction corresponding to a create modifier list sent by a client side, analyzing the first control instruction, judging whether duplication-removal is needed or not to generate a judgment result, and correspondingly creating a table in the data warehouse basic framework and / or the Hadoop database according to the judgment result. According to the data processing equipment, the data processing method and the computer readable storage medium, through the application of the Hadoop database, and by judging whether duplication-removal is needed or not during creating, convenience is brought to data renewal of an entity table in the data warehouse basic framework, and the maintenance cost is lowered.

Description

technical field [0001] The present invention relates to the field of data processing, and more specifically, to a data processing device, method and computer-readable storage medium. Background technique [0002] In big data processing, data update is the most common type of data processing. HIVE (Data Warehouse Infrastructure), as the most commonly used component for processing data, is widely used because it supports SQL (Structured Query Language) or SQL-like . However, because HIVE itself only processes data, it does not support deduplication operations on data, so it is impossible to update data. As a result, databases that simply apply HIVE cannot effectively complete the update operation of complex structured data, and the processing flow is complicated. Maintenance costs are high. Contents of the invention [0003] The technical problem to be solved by the present invention is how to solve the problem that the data processing using HIVE components in the prior ar...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/211G06F16/2282G06F16/27
Inventor 齐伟
Owner NUBIA TECHNOLOGY CO LTD