Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method to divide big data transmission tasks based on task scheduling

A task scheduling and data transmission technology, applied in the computer field, can solve the problems of high system operation risk, long execution time, abnormal resources, etc., and achieve the effect of improving system operation efficiency, improving efficiency and reducing system operation risk.

Inactive Publication Date: 2018-03-27
USTC SINOVATE SOFTWARE
View PDF4 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Using open source ETL tools to realize large data transmission tasks not only takes a long time to execute, but also causes abnormal situations due to resource occupation all the time, resulting in low data transmission efficiency and high system operation risks
In addition, the use of open source ETL tools to implement large data transmission tasks may not be able to meet some special data type requirements, and the support is not perfect, so it cannot fully meet the needs of application scenarios

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method to divide big data transmission tasks based on task scheduling
  • Method to divide big data transmission tasks based on task scheduling

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] A method for implementing large data transmission task segmentation based on task scheduling in the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0034] like figure 1 As shown, the steps of a method for realizing large data transmission task segmentation based on task scheduling in the present invention are as follows:

[0035] step 1. Configure the data source, the specific implementation method is as follows:

[0036] (1) Configure the data source in the data source configuration, including the data source and target source;

[0037] (2) The transfer task will connect to the data source to read data according to the configured data source and transfer it to the target data source;

[0038] (3) The configuration content of the data source is different according to the data source type, for example, the present invention can use relational database or big data HBASE, such as figure 2 As s...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method to divide big data transmission tasks based on task scheduling. The method comprises: configuring data sources; configuring big data transmission tasks; starting the transmission tasks, and dividing each task into multiple subtasks based on task scheduling so as to carry out data transmission. The method has the advantages that long execution time of a single taskcan be avoided, resources are never used for long time, data transmission efficiency can be effectively improved, system running risk can be relieved, system running efficiency can be improved, special data type requirements can be satisfied, and application scenario needs are fully met.

Description

Technical field: [0001] The invention relates to the technical field of computers, and specifically relates to a method for realizing large data transmission task segmentation based on task scheduling. Background technique: [0002] In the existing enterprise integration platforms, there is no task segmentation method for big data transmission tasks based on task scheduling, and open source ETL tools are generally used for transmission. Using open-source ETL tools to implement large data transmission tasks not only takes a long time to execute, but also causes abnormal situations due to resource occupation, resulting in low data transmission efficiency and high system operation risks. In addition, the use of open source ETL tools to implement big data transmission tasks may not be able to meet some special data type requirements, and the support is not perfect, so it cannot fully meet the needs of application scenarios. Invention content: [0003] The technical problem to...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/48
CPCG06F9/4806G06F9/485
Inventor 冯强中裴孝贞王震
Owner USTC SINOVATE SOFTWARE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com