Distributed preprocessing method and distributed preprocessing system

A preprocessing and distributed technology, applied in the field of distributed systems, can solve the problem of low preprocessing efficiency of distributed systems, and achieve the effect of improving preprocessing performance, ensuring atomicity, and ensuring consistency

Active Publication Date: 2013-07-03
CHINA MOBILE COMM GRP CO LTD
View PDF10 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] The purpose of the present invention is to propose a distributed preprocessing method and system for the low efficiency of distributed system preprocessing in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed preprocessing method and distributed preprocessing system
  • Distributed preprocessing method and distributed preprocessing system
  • Distributed preprocessing method and distributed preprocessing system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0053] The preferred embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described here are only used to illustrate and explain the present invention, and are not intended to limit the present invention.

[0054] The distributed preprocessing system of the present invention has designed the main control node PMaster, can dynamically and organically manage the preprocessing process of the Process node (i.e. the processing node), and guarantee the consistency and atomicity of the Process node preprocessing process; and, the system guarantees The intermediate results of multiple execution modules invoked by the Process node are processed in memory without temporary storage.

[0055] like figure 1 As shown, the distributed preprocessing system of the present invention includes a management module (ie PManager) and an execution module set (ie PModule). The management modul...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a distributed preprocessing method and a distributed preprocessing system. The distributed preprocessing method includes the following steps: a process node receives data sent from a source node; a management module configures types of multiple executive modules and an executive order according to different demands of the process node; and the process node invokes an executive module collection which is deployed by the process node and configured by the management module for carrying out preprocessing operation on the received data, wherein the executive module collection comprises the multiple executive modules. Through the distributed preprocessing method and the distributed preprocessing system, the preprocessing process of the process node can be managed dynamically and flexibly, the executive module collection is managed, and consistency and atomicity of the preprocessing process of the process node are guaranteed; and the distributed preprocessing system guarantees that preprocessing intermediate results of the multiple executive modules of the process node are processed in an internal memory and do not need to be stored temporarily, and therefore preprocessing performance is improved.

Description

technical field [0001] The present invention relates to a distributed system in the field of computers, in particular to a distributed preprocessing method and system. Background technique [0002] The logical architecture of the existing distributed data loading and preprocessing system includes: [0003] Source node (that is, source node): deployed on the data source side, encapsulates the data source to be loaded, and sends the data in the data source to the Dest node; [0004] Dest node (that is, the destination node): deployed on the storage side, receives data from the Source node, and imports it into the back-end storage; [0005] Process node: Before the Dest node stores the data, multiple Process nodes can be introduced to perform some preprocessing on the data, such as filtering and conversion. The Process node can physically share a node with the Dest node. [0006] At present, the preprocessing work of the distributed data loading and preprocessing system main...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
Inventor 钱岭高丹齐骥郭磊涛周大
Owner CHINA MOBILE COMM GRP CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap