Distributed data processing method and system

A technology of distributed data and processing methods, applied in the field of data processing, can solve problems such as poor scalability, and achieve the effect of scalability and adaptability

Active Publication Date: 2015-03-25
新方正控股发展有限责任公司 +1
View PDF4 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] For this reason, the technical problem to be solved by the present invention is to overcome the problem of poor scalability of the entire distributed file system caused by t

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data processing method and system
  • Distributed data processing method and system
  • Distributed data processing method and system

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0086] Example 1

[0087] As a distributed data processing method running on the server side according to the present invention, such as figure 2 As shown, including the following steps:

[0088] The server obtains the resource information of the client in real time.

[0089] The server determines a task that meets the processing capability of the client according to the resource information.

[0090] The server allocates and sends tasks that meet its processing capabilities for the client.

[0091] Wherein, the task includes the address of the data to be processed, the address of the plug-in library, and the type of processing instruction required to process the data to be processed. Plug-and-play plug-ins are stored in the plug-in library.

[0092] In the distributed data processing method operated by the server of the present invention, the distributed task contains the address of the plug-in needed to process the data to be processed. When the client processes the task, the data pr...

Example Embodiment

[0093] Example 2

[0094] As an embodiment of the present invention, on the basis of embodiment 1, the distributed data processing method running on the server side further includes the following steps of updating the plug-in library:

[0095] Store the new plug-in in the plug-in library.

[0096] The processing instruction type corresponding to the new plug-in is stored in the server.

[0097] The version corresponding to the new plug-in is stored in the server. Correspondingly, the task can also include plug-in version information, and the specific form of the task including plug-in version information is as follows: .

[0098] During the normal operation of the server, if a new type of data processing occurs, at this time, it is only necessary to update the plug-in library, and add the new type of data processing plug-in to the existing plug-in library. The task contains the address of the new plug-in. When the client processes the task, it only needs to call the new plug-in accord...

Example Embodiment

[0099] Example 3

[0100] As an embodiment of the present invention, on the basis of the above-mentioned embodiment 1 or embodiment 2, the server allocates and sends a task that meets the processing capability of the client to a client that meets the task processing conditions. It also includes the status of the data to be processed, and the status of the data to be processed further includes client information and tag information, where:

[0101] The client information is information of the client to which the data to be processed is allocated; the marking information is information that marks that the data to be processed is taken away.

[0102] When the server allocates a new task to fetch the data to be processed, those with the tag information indicate that the data to be processed has been taken away and cannot be taken away and distributed again; it is less effective or even avoids the same data to be processed being assigned to The occurrence of resource waste caused by repe...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a distributed data processing method and system. The method comprises the following steps that: a server allocates and sends a task which is consistent with the processing capacity of a client which satisfies a processing condition to the client; after receiving the task, the client processes the task to obtain a plug-in address and the address of data to be processed; the client loads a plug-in according to the plug-in address, and extracts the data to be processed according to the address of the data to be processed; and the client processes the data to be processed by using the plug-in to obtain processed data. When processing of a new type of data to be processed is added, the client processes the data to be processed in a way of calling the plug-in, so that only one plug-in for processing the new type of data to be processed needs to be developed. Moreover, the plug-in can be called by different clients by being put into a plug-in library without upgrading of the clients on a large scale, so that the method and the system have the advantage of high adaptability to processing of new types of data to be processed.

Description

technical field [0001] The present invention relates to a data processing method and system, in particular to a distributed data processing method and system run by a client and a server. Background technique [0002] The so-called distributed data processing is a computer data processing method that connects multiple computers in different locations or with different functions or with different data through a communication network, and completes information processing tasks in a coordinated manner under unified management and control. Distributed data processing is introduced in the splittable file data processing process, that is, the data processed by a single computer is processed concurrently and cooperatively by multiple computers, which can provide a strong technical guarantee for a large number of file data processing. When processing a large number of data files, especially when frequent image compression, file format conversion, etc. consume computer performance, t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L29/08
Inventor 林厚超黄科
Owner 新方正控股发展有限责任公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products