Unlock instant, AI-driven research and patent intelligence for your innovation.

Distributed data calculation-based task allocation method and task allocation apparatus

A technology of distributed data and task allocation, applied in computing, resource allocation, multi-program devices, etc., can solve problems such as increasing time-consuming and IO redundancy, and achieve the effect of reducing IO redundancy and time-consuming

Active Publication Date: 2017-02-22
HANGZHOU HIKVISION SYST TECH
View PDF7 Cites 20 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this way, data needs to go through the network repeatedly, resulting in IO redundancy and increasing time-consuming

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data calculation-based task allocation method and task allocation apparatus
  • Distributed data calculation-based task allocation method and task allocation apparatus
  • Distributed data calculation-based task allocation method and task allocation apparatus

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0020] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are exemplary only, and are not intended to limit the scope of the present invention. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present invention.

[0021] Please refer to figure 1 , which is a method flowchart of the first embodiment of a distributed data computing task allocation method provided in the specific embodiments of the present invention. The task allocation method in this embodiment is mainly used for parallel computing of a large amount of data in a distributed database to improve computing efficiency. As shown in the figure, the task allocation method...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed data calculation-based task allocation method and task allocation apparatus. The task allocation method comprises the steps of receiving storage parameters of calculated target data in distributed data; mapping data slices of the target data to a resilient distributed dataset according to the storage parameters, wherein each data slice corresponds to a partition of the resilient distributed dataset; and specifying the partitions to storage nodes to generate calculation tasks for performing calculation. The calculation tasks are allocated to the storage nodes corresponding to the data through data storage information in a distributed database, and only local memory data needs to be called in the calculation process, so that IO redundancy and time consumption caused by multi-time data forwarding are reduced.

Description

technical field [0001] The invention relates to the technical field of video monitoring, and in particular to a task allocation method and a task allocation device for distributed data computing. Background technique [0002] At present, there are two main ways to initialize the data of elastic distributed datasets in Spark: directly obtain data from the collection and store it in RDD (Resilient Distributed Datasets, elastic distributed datasets); read local or distributed file system ( HDFS, S3, etc.) text files, sequence files, etc. For the data in HBase, the data is mainly pulled through the HBase client, converted and processed, saved in the RDD, distributed to multiple slices, and then distributed computing is performed through the RDD operator. In this way, data needs to go through the network repeatedly, resulting in IO redundancy and increasing time consumption. Contents of the invention [0003] The object of the present invention is to provide a task allocation...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/50H04N7/18
CPCG06F9/5066G06F16/27G06F9/5016G06F9/5077
Inventor 刘志辉
Owner HANGZHOU HIKVISION SYST TECH