Distributed data analysis task scheduling system

A distributed data and data analysis technology, applied in structured data retrieval, electronic digital data processing, database distribution/replication, etc., can solve problems such as data loss, high node resource consumption, and inability to meet the needs of analysis task scheduling.

A distributed data and data analysis technology, applied in structured data retrieval, electronic digital data processing, database distribution/replication, etc., can solve problems such as data loss, high node resource consumption, and inability to meet the needs of analysis task scheduling.

CN107766147AInactive Publication Date: 2018-03-06SHANGHAI BAOSIGHT SOFTWARE CO LTD
7 Cites 13 Cited by

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data analysis task scheduling system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] The present invention will be described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0052] The architecture of the distributed data analysis task scheduling system is shown in the attached figure figure 1 , mainly consists of the following modules:

[0053] Distributed data storage service module: The distributed storage service is stored through the Nosql database HBase, and the data retrieval is realized through the distributed search engine Solr, which meets the characteristics of large capacity, high reliability, high performance, data copy security, and strong dynamic expansion ca...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a distributed data analysis task scheduling system. The system comprises a distributed data storage service module, a resource-based distributed task scheduling engine module, adistributed message queue module, a distributed application coordination service module and an automatic execution engine module. The invention provides a distributed task scheduling framework for performing data analysis by utilizing an R language; distributed resource management is realized by utilizing a resource management platform, so that distributed scheduling execution of R language analysis is realized; and by utilizing an automatic scheduling engine, automatic calling of a data fragmentation task is realized, so that the automatic analysis demand of data tracking in an industrial process is met.

Description

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
06 Mar 2018
Publication
CN107766147A