A distributed data processing coordination method and system
A technology of distributed data and coordination methods, applied in the direction of electrical digital data processing, special data processing applications, program startup/switching, etc., can solve the problems that the system cannot be notified and the data cannot be processed.
Active Publication Date: 2018-04-27
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF3 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0006] Take the modulus according to the data, and distribute different data to different instances after taking the modulus, which solves the single point problem of the system. However, the existing problem is: when an instance crashes, the corresponding system cannot be notified, and the will result in the data not being processed
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0023] The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.
[0024] Such as figure 1 Shown is a work flow diagram of a distributed data processing coordination method of the present invention, including:
[0025] Step S101, the task scheduler classifies the tasks in the task scheduling database, associates batches with each task, associates one or more tasks with the same batch, and associates each batch with a processor;
[0026] Step S102, the processor acquires associated batches from the task scheduling database;
[0027] Step S103, the processor processes the tasks associated with the associated batch;
[0028] Step S104, the task scheduler regularly checks the online status of the processors, takes the processors whose online status is offline as offline processors, and uses the processors whose online status is online as online processors, and obtains the The associated batch is used ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a distributed data processing coordination method and system. The method includes the steps that tasks in a task scheduling database are classified by a task scheduler, the batches of the tasks are associated, one or more tasks of the same batch are associated, and each batch is associated with one processor; the processors acquire the associated batches from the task scheduling database; the processors process the associated tasks of the associated batches; the task scheduler inspects the online state of the processors regularly, the processors with the offline state are made to serve as offline processors, the processors with the online state are made to serve as online processors, the batches associated with the offline processors are obtained and serve as batches to be allocated, and the batches to be allocated are associated with one online processor again. According to the method and system, when the processors break down and are offline, the tasks are reallocated by the task scheduler, and therefore the distributed processors can process the tasks in a coordination mode.
Description
technical field [0001] The present invention relates to the technical field related to distributed data processing, in particular to a distributed data processing coordination method and system. Background technique [0002] Today's Internet is developing quite rapidly, especially with the arrival of the era of big data. How to quickly process these data is a major challenge for all companies. The following methods are commonly used, [0003] 1. Using single-instance multi-thread processing, [0004] The disadvantage of this is that there is a single point of risk. Once the instance crashes, all data will not be processed. The processing of large data is relatively slow and cannot be processed in a timely manner. It is basically not used now. [0005] Second, use multiple instances [0006] Take the modulus according to the data, and distribute different data to different instances after taking the modulus, which solves the single point problem of the system. However, the ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30G06F9/48
CPCG06F9/4881
Inventor 凡红恩
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD