City-wide big data scheduling system

A scheduling system and big data technology, applied in database management systems, data processing applications, structured data retrieval, etc., can solve the problems that the alarm system does not support, only supports IP addresses, DingTalk and logs cannot be located, etc.

Pending Publication Date: 2022-04-12
江苏银承网络科技股份有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to provide a large data scheduling system in the same city to alleviate the technical problems that existing escheduler scheduling systems only support IP addresses between nodes, the alarm system does not support DingTalk, and logs cannot be located to specific operating users

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • City-wide big data scheduling system
  • City-wide big data scheduling system
  • City-wide big data scheduling system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0042] At present, the escheduler of the mainstream scheduling system is dedicated to solving the intricate dependencies in the data processing process, so that the scheduling system can be used out of the box in the data processing process, and can be selected as a unified scheduling system for big data clusters. However, the escheduler scheduling system only supports IP addresses between nodes, the alarm system does not support DingTalk, and the logs cannot be located to specific operating users.

[0043] Based on this...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

According to the intra-city big data scheduling system provided by the invention, secondary development and new operation log service are carried out on the basis of an escheduler scheduling system, and specific operation users are positioned through operation logs, so that the troubleshooting efficiency of problems is improved; a temporary node of a hostname registration Zookeeper module is newly added, so that the problem that reconfiguration is needed due to IP change caused by restarting of an execution machine is avoided; a workflow state monitoring module is newly added to monitor system workflows and send alarm information to remind a user when important workflows are forgotten to be online, so that scheduling tasks in the workflows are ensured to be carried out orderly; finally, a resource utilization rate alarm module is added, when the resource utilization rate reaches a threshold value, alarm information is sent to remind a user to adjust, the stability of the system is guaranteed, and the alarm information is sent to a user nailing group through integrated nailing alarm.

Description

technical field [0001] The invention relates to the technical field of computer big data, in particular to an intra-city big data scheduling system. Background technique [0002] With the continuous increase of the company's business and the continuous development of big data technology, the company's business system needs to conduct business analysis on more report data, followed by the increase of offline tasks and real-time tasks, so a unified scheduling system is required to Tasks are scheduled. Currently, the mainstream scheduling systems include Oozie, Azkaban, and escheduler. However, Oozie has problems such as the need to write a large number of XML language configurations during the configuration workflow process, low community activity, and high code complexity. Azkaban has problems such as manually completing fixed-format files when creating jobs, and information cannot be stored persistently in memory during task execution. Once the storage fails, all workflows...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06Q10/10G06Q10/06G06F16/25
Inventor 孟恒
Owner 江苏银承网络科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap