Big data quantity high performance processing implementing method based on parallel process of split mechanism

A high-performance processing and parallel processing technology, applied in electrical digital data processing, special data processing applications, instruments, etc., to achieve the effect of low investment, easy portability, and low performance dependence

Inactive Publication Date: 2009-08-19
LINKAGE SYST INTEGRATION
View PDF0 Cites 26 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to propose a method of parallel processing based on a split mechanism to realize high-pe

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data quantity high performance processing implementing method based on parallel process of split mechanism
  • Big data quantity high performance processing implementing method based on parallel process of split mechanism

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0022] A universal interface can be designated for convenient invocation of the original system, thereby achieving the function of replacing the processing in the original database, and achieving a seamless connection with the original system while significantly improving the execution efficiency of the system.

[0023] Several key technologies in the implementation process are as follows:

[0024] One-time reading: In order to achieve the purpose of reducing access to the massive data source tables, it is necessary to read all subsequent summary tables at one time. You can formulate a SQL statement that extracts a large number of source tables by first listing the dimensions and index fields required by each summary table, and then taking the union. No matter how many kinds of aggregations there are, only one access to massive data sources is required to minimize database pressure. In order to reduce access to the massive data source tables, it is necessary to read all the inform...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for realizing large data amount high-performance process, which is based on splitting mechanism parallel processing. A splitting rule is set for the mass data of telegraph tickets to equally split the mass data to be processed into a plurality of files; and the multi-thread and multi-CPU parallel process of a file processing system is adopted. The quick processing of the mass data is as follows: the parallel process of the file processing system is to simulate the database sql algorithm to carry out calculation; an SQL sentence for extracting a mass data source table is established through firstly spreading out the dimensionality and index field required by each collection table and secondly obtaining the unions and then the information required by all the following mass data collection tables is read over; the assembly storing is as follows: after the work for collecting the small files formed while equally splitting a plurality of files is finished, all the result files are combined into large files according to the target table types and then are loaded into the collection tables; and the work can be completed by the peculiar quick accessing instruction of the database.

Description

technical field [0001] The invention belongs to the category of application technology for data processing of massive databases of telecom operators, in particular to a method for realizing high-performance processing of large amount of data through parallel processing. Background technique [0002] Generally speaking, the business inventory data of telecom operators is often massive, especially the inventory data that needs to be aggregated and counted, and the number of records processed every day reaches tens of millions. The usual practice is to pass one or more complex SQL statements in the database and submit them to the database for completion. Such work takes up a lot of time and database resources. [0003] For example, for the daily inventory data generated every day, it is necessary to summarize the records of the daily inventory table according to the specified conditions, and then update them into the summary table. The update method is: if the summary table alr...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30H04Q3/00
Inventor 沈小军庞海东赵懿敏李捷曹晓华
Owner LINKAGE SYST INTEGRATION
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap