Distributed data analyzing and processing method and system

A technology of distributed data and processing methods, which is applied in the fields of data analysis and processing and databases. It can solve the problems of prolonging the generation cycle of statistical analysis results and inability to share the load of database hosts, so as to shorten data processing time, improve concurrent processing capabilities, and shorten generation time. cycle effect

Active Publication Date: 2012-05-30
CHINA MOBILE GROUP SICHUAN
View PDF7 Cites 24 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0011] The processing capacity of the database host has become the bottleneck of online analysis data processing, prolonging the generation cycle of statistical analysis results; at the same time, low-load computing resources (such as WEB servers) in the system have a long idle period of resources, but they cannot share Database host load

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data analyzing and processing method and system
  • Distributed data analyzing and processing method and system
  • Distributed data analyzing and processing method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] The use of distributed data processing technology needs to meet two conditions: one is to be able to extract distributable processing units, and the other is to be able to aggregate the results of distributed processing to obtain the final result.

[0042] The three steps of data acquisition, analysis and processing of most online analysis systems just meet these two conditions. First, the execution order of most OLAP database scripts does not have a strong linear dependency; therefore, most OLAP database scripts (collection, cleaning, analysis, processing, etc.) can be fine-grained split according to specific business rules to achieve distributed Execute, and finally summarize the results.

[0043] By establishing a remote database connection between the load-sharing database and the main database, the result data table of the main database can be mapped to a remote table in the load-sharing database, so the processing results can be sent back in the script with a simp...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a distributed data analyzing and processing method, comprising the following steps that: A, a load-sharing server transmits a request for obtaining an executable object script to a script server, and the script server judges whether the executable object scrip is found, if so, a step C is executed, otherwise, a step B is executed; B, the script server reads script source codes from a database host, and edits the script source codes to obtain the executable object script; C, the script server obtains the executable object script from the script server, calls an entrance method of a script object, and executes data acquisition, clearing, counting, analysis and processing types of function codes defined by the executable object script; and D, the load-sharing server transmits the processing result of executing the executable object script back to the database host through a remote database connection. The invention further provides a distributed data analyzing and processing system.

Description

technical field [0001] The invention relates to the field of data analysis and processing and database technology, in particular to a distributed data analysis and processing method and system. Background technique [0002] The existing online analytical processing (OLAP) process often adopts centralized data processing, and there is serious competition for database resources. Data collection, cleaning, analysis, and execution have become bottlenecks, prolonging the generation cycle of result data. [0003] figure 1 It is a simple schematic diagram of an existing online analytical processing (OLAP) type data processing system. The system includes a database host 101 and three Web servers, namely a Web server 102, a Web server 103 and a Web server 104. [0004] figure 2 Shown is the existing online analytical processing (OLAP) type data processing flow, including the following steps: [0005] Step 201: multiple web servers collect various raw data for online analysis and...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F9/50
Inventor 曾健陈刚梅松梁宇赵勇徐苛杰
Owner CHINA MOBILE GROUP SICHUAN
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap