Data flow type processing method based on data processing center

A technology of data processing and processing method, applied in the field of big data processing, can solve the problem of no data flow connected in series, and achieve the effect of convenient unified control and speed improvement

Active Publication Date: 2015-01-28
ASIAINFO TECH NANJING
View PDF4 Cites 20 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The application platform processes system data from Cloud Ladder, Feitian, HBase and OceanBase, but as far as the platform is conc

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data flow type processing method based on data processing center
  • Data flow type processing method based on data processing center
  • Data flow type processing method based on data processing center

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0038] For example, if the present invention is applied to a provincial economic sub-system of a telecommunications company, it is required to synchronize the GPRS traffic interface data from the MPP database (GP) to Hadoop, and to perform privacy processing on the MSISDN (mobile phone number) field, and to perform a null value check. , and operate on the CALL_DUR (call duration) field (add one to the field value).

[0039] The requirements for the above data processing tasks can be completed in the data processing center through the following steps:

[0040] The configuration data processing process is: table scan→GP data source extraction→pipeline flow→conversion calculation→pipeline→HDFS loading, this process is also a data flow;

[0041] Configure the data processing method in the data processing center, that is, configure it in the "conversion calculation" of the above process, perform privacy and null check methods on the mobile phone number field, and perform operations...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data flow type processing method based on a data processing center. Processing method steps including data verification, sorting, aggregation and grouping and the connecting relationship between two different processing method steps are predefined in the data processing center; the data processing center is further provided with a data source connecting port used for being connected with a data source, a data processing method and process definition connecting port used for being connected with a user interface, a target data output port and a processing process monitoring port used for being connected with a process monitoring unit. The data flow type processing method includes the steps of data obtaining, flow type processing configuration, data processing method configuration, target data source obtaining and the like, a user can configure a data processing method and define a data processing process through the user interface, and therefore a corresponding target data source is obtained. The data flow type processing method based on the data processing center adopts data flow type processing through a big data platform, can increase the big data processing speed, and expands the range of types of supportable data processing methods.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a data stream processing method based on a data processing center. Background technique [0002] Regardless of whether the era is characterized by massive data or big data, the huge scale, rapid growth, various types and different structures of data have become unavoidable practical problems. How to turn complex big data into "small" data that we can deal with effectively, that is, to build a clean and complete data set for a specific problem, this process becomes particularly important. [0003] Big data governance and analysis are very thorny issues in the process of big data processing, and how to achieve the timeliness, flexibility and accuracy of processing is particularly important. Currently in the Internet industry, flexibility and accuracy (even allowing partial data loss) are usually sacrificed in exchange for the timeliness of data processing, but in some ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/254
Inventor 黄雪东
Owner ASIAINFO TECH NANJING
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap