Intelligent data service method based on distributed system

A technology of distributed systems and data services, applied in the field of intelligent data services based on distributed systems, can solve the problems of difficult data services, difficult to form intelligent data services, adding preprocessing, etc., to achieve the effect of reducing bandwidth requirements

Active Publication Date: 2013-02-20
JIANGNAN INST OF COMPUTING TECH
View PDF3 Cites 15 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Non-relational databases provide Key-Value (key-value) storage, but it is difficult to add preprocessing functions to data services and form intelligent data services

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Intelligent data service method based on distributed system
  • Intelligent data service method based on distributed system
  • Intelligent data service method based on distributed system

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0027] The invention is an intelligent data service method for data-intensive applications derived from a traditional distributed system.

[0028] specifically, figure 1 The architecture of an intelligent data service platform based on a distributed system according to the first embodiment of the present invention is schematically shown.

[0029] Such as figure 1 As shown, similar to most distributed file systems and distributed databases, the entire architecture is a typical Master / Slave architecture. E.g, figure 1 The intelligent data service platform architecture based on the distributed system shown includes a master node M and multiple slave nodes; specifically, figure 1 A case where n slave nodes are included in is shown, that is, the first slave node S1, the second slave node S2, the third slave node S3, ..., the nth slave node Sn.

[0030] Among them, the master node M includes a data preprocessing analysis engine M1 and a global metadata management module M2. Each slave nod...

no. 2 example

[0038] figure 2 It schematically shows a schematic diagram of a process of writing a file to a distributed system according to the second embodiment of the present invention. Among them, the process of writing files to the distributed system does not preprocess the data. Here, it is assumed that there are three slave nodes in the distributed system: the first slave node S1, the second slave node S2, and the third slave node S3, but obviously the number of slave nodes in the distributed system is not limited to three, but can Is any suitable number.

[0039] Specifically, such as figure 2 As shown, the process of writing files to the distributed system according to the second embodiment of the present invention includes:

[0040] The first writing step a1: the client pcm1 asks the master node M whether the file to be written exists in the distributed system.

[0041] Second writing step b1: If the file to be written exists in the distributed system, the master node M sends the fil...

no. 3 example

[0046] image 3 It schematically shows a schematic diagram of a process of reading a file from a distributed system according to the third embodiment of the present invention. Here, it is assumed that there are three slave nodes in the distributed system: the first slave node S1, the second slave node S2, and the third slave node S3, but obviously the number of slave nodes in the distributed system is not limited to three, but can Is any suitable number.

[0047] The process of reading files from the distributed system according to the third embodiment of the present invention includes:

[0048] The first reading step a2: the client pcm1 sends a data request to the master node M, which includes the file path and the required preprocessing.

[0049] Second reading step b2: The master node M analyzes the data request of the client pcm1, can determine the slave node where the required file is located and the required preprocessing program, directly preprocess the required file, and sen...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides an intelligent data service method based on a distributed system. A master node of the distributed system is used for managing global file namespace; and in processes of writing files into the distributed system and reading files from the distributed system, the master node of the distributed system is used for analyzing and processing requests of clients, selecting specific data preprocessing programs, distributing the programs to slave nodes of the distributed system for subsequent data preprocessing and transmission. According to the intelligent data service method based on the distributed system, prior distributed system storage space can be clustered rapidly in data intensive application environments; computing resources of the distributed system are fully used, so that data services can be provided intelligently according to requests of external computing devices; and part of data processing loads are further transferred from external computing devices to the distributed system, and accordingly, bandwidth requirements for providing data services for external computing devices are reduced.

Description

Technical field [0001] The present invention relates to the field of computing technology, and more specifically, the present invention relates to an intelligent data service method based on a distributed system. Background technique [0002] In data-intensive applications, large-scale data set processing is the core of the application, and I / O (input / output) bandwidth has become the main factor affecting its performance, which makes traditional computing and storage separated systems unsuitable for data-intensive applications Application, therefore, requires a new type of data storage and service mode to improve the performance of data transmission and processing. [0003] Data service generally refers to the storage, management and transmission of data, and the specific manifestations are different in different applications. In the context of data-intensive applications, data services mainly include two aspects, one is the storage technology of massive data, and the other is the...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 谢向辉臧春峰吴东郝子宇原昊钱磊张鲁飞胡苏太
Owner JIANGNAN INST OF COMPUTING TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap