Data conversion task scheduling method

A scheduling method and data technology, applied in the direction of program startup/switching, resource allocation, multi-program installation, etc., can solve the problems of large CPU and memory consumption, resource cost consumption, platform data impact, etc., to improve completion efficiency, avoid The effect of memory overflow and improving usage efficiency

Pending Publication Date: 2022-04-15
LIANYI TECHNOLOGY CO LTD
View PDF5 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] While they bring convenience to data conversion work, they also bring resource cost consumption. For example, in a data exchange tool developed using java, it takes 4 times to record from the database and put it in the program, using objects (ordinary processing method of ORM) About 10 times the amount of memory space required for KV storage like HashMap, which will lead to a large amount of CPU consumption when an execution error is found when processing large data scale data, or when a program calculates a million-level data volume And memory, in the case of limited resources, high occupancy can easily lead to system crashes, memory overflow, etc., and the associated platform data will be greatly affected

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data conversion task scheduling method
  • Data conversion task scheduling method
  • Data conversion task scheduling method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Apparently, the described embodiments are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0020] Such as figure 1 In the illustrated embodiment, the scheduling method of the data conversion task includes monitoring the memory usage of the server, monitoring the memory usage of the server, and obtaining the free memory value of the server. Specifically, the future memory situation of the server is estimated by the memory data consumed by the currently active data process execution of the server, and a scheduler is used to record the memory used by the serv...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a scheduling method for a data conversion task, which comprises the following steps of: monitoring the memory use condition of a server, and calculating to obtain a data extraction memory occupancy value, an incremental extraction memory occupancy value and an incremental data write-in memory occupancy value according to data source information and the data conversion task; according to the calculation result, whether the free memory of the server meets the execution condition of the data conversion task or not is judged, and if yes, the data conversion task is operated. According to the scheme, the risk of memory overflow is avoided, the use efficiency of server memory resources is improved, and the task completion efficiency is improved.

Description

technical field [0001] The invention relates to the field of memory resource scheduling, in particular to a scheduling method for data conversion tasks. Background technique [0002] At present, there are a variety of ETL tools on the market, which are extraction (extract), transformation (transform), and loading (load). ETL is used to extract data from distributed and heterogeneous data sources such as relational data and flat data files. After reaching the temporary middle layer, it is cleaned, converted, integrated, and finally loaded into the data warehouse or data mart. Today, similar products that are widely used include data conversion tools such as Kettle, ODI, DataPipeline, and RestCloud ETL. [0003] While they bring convenience to data conversion work, they also bring resource cost consumption. For example, in a data exchange tool developed using java, it takes 4 times to record from the database and put it in the program, using objects (ordinary processing method...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F9/50
Inventor 刘欣荣任刚舒畅肖炯恩
Owner LIANYI TECHNOLOGY CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap