A distributed data task processing method and device

A distributed data and data processing technology, which is applied in electrical digital data processing, multi-programming devices, program startup/switching, etc., can solve the problem of task interruption and cannot quickly resume execution, avoid repeated calls and improve computing efficiency. Effect

Inactive Publication Date: 2019-03-08
NORTHEAST GASOLINEEUM UNIV
View PDF3 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the embodiments of the present invention is to provide a distributed data task processing method and device to solve the problem that the task interruption cannot be quickly resumed during the existing data distributed computing

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A distributed data task processing method and device
  • A distributed data task processing method and device
  • A distributed data task processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0033] See figure 1 , in order to solve the problem that task interruption cannot be quickly resumed during existing data distributed computing, a distributed data task processing method disclosed in this embodiment is applied to a scheduling server.

[0034] Among them, the scheduling server is used to allocate tasks to appropriate data processing nodes for data processing, and re-match abnormal tasks or abnormal nodes through effective allocation logic during the allocation process.

[0035] Specifically, the distributed data task processing method includes:

[0036] Step 101 obtains one or more tasks currently to be executed, generates a task list for one or more tasks to be executed, and generates corresponding task ID information for each task to be executed, so that the task can be found when assigning tasks and executing tasks. Track the corresponding task. When the acquired task is a single task, the single task is split into subtasks so as to be distributed to multi...

Embodiment 2

[0043] See figure 2 , in order to solve the problem that task interruption cannot be quickly resumed during existing data distributed computing, a distributed data task processing method disclosed in this embodiment is applied to a scheduling server, and the method includes:

[0044] Step 201 obtains one or more tasks currently to be executed, generates a task list for one or more tasks to be executed, and generates corresponding task ID information for each task to be executed, so that the tasks can be found when assigning tasks and executing tasks. Track the corresponding task. When obtaining one or more tasks currently to be executed, judge the priority level of the tasks to be executed, including obtaining at least one of the task type of the to-be-executed task, estimated execution time, estimated occupied resource size, and queued waiting time of the task Task information, performing weight comparison on the task information, and obtaining the highest priority task for e...

Embodiment 3

[0051] In order to further ensure the security of data transmission during distributed task calculation, a distributed data task processing method disclosed in this embodiment, under the premise of embodiment 1 or embodiment 2, the task list will One or more tasks to be executed are sent to one or more available data processing nodes for data processing, a security protocol between each task to be executed and the data processing node is established, and each task to be executed is sent to the data processing node .

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a distributed data task processing method and device. The method is applied to a scheduling server. The method comprises the following steps of obtaining a task to be executed at present, generating a task list for the task to be executed, and generating corresponding task ID information for each task to be executed; acquiring currently available data processing nodes, generating a processing node list by the data processing nodes, and generating corresponding node ID information for each data processing node; sending the tasks to be executed in the task list to the available data processing nodes for data processing; in the course of the task, obtaining the execution status of the currently executed task in real time, if the task has been interrupted, returning the interrupt information including the task ID information and the node ID information to the scheduling server, and using the scheduling server to reallocate the interrupt informationto the new data processing node to process the task. When the task is interrupted, the task can not be recovered and executed quickly at the data distributed calculation, thereby improving the computing efficiency.

Description

technical field [0001] The invention relates to the technical field of distributed intelligence, in particular to a distributed data task processing method and device. Background technique [0002] Although big data technology includes a series of complex technologies such as storage, calculation and analysis, distributed data processing has always been its core. Distributed task data processing is to use distributed computing technology to process task data. With the rapid expansion of data volume, traditional centralized data processing has gradually reappeared to meet the needs of the market. Compared with centralized data processing, distributed data processing distributes the huge computing tasks that were originally concentrated on a single node to distributed data in a load-balanced manner. Computers in a network perform processing in parallel. Although the distributed system is composed of several independent computers, when solving a specific problem, each part can...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48
CPCG06F9/485G06F9/4881
Inventor 张可佳李春生胡亚楠田枫刘志刚杜睿山
Owner NORTHEAST GASOLINEEUM UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap