Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Spark application monitoring method, system and device and storage medium

A technology of intelligent monitoring and monitoring data, applied in the computer field, can solve the problems of no dynamic self-healing, unfavorable rapid positioning of technicians, and inability to monitor SparkThriftServer, so as to achieve the effect of realizing intelligence

Pending Publication Date: 2020-08-25
深圳市卡数科技有限公司
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, this method has obvious shortcomings and shortcomings. First, it cannot effectively monitor the Spark Thrift Server application, because the tasks of the Spark application are resident in the Hadoop Yarn task pool, and cannot be realized simply by polling the Hadoop Yarn task pool. Monitoring of Spark Thrift Server applications; second, there is no dynamic self-healing solution, only restarting to solve the problem; third, there is no analysis of abnormal logs, which is not conducive to rapid positioning of problems by technicians

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Spark application monitoring method, system and device and storage medium
  • Spark application monitoring method, system and device and storage medium
  • Spark application monitoring method, system and device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0045] Such as figure 1 As shown, Embodiment 1 of the present invention provides a Spark application monitoring method, the method comprising:

[0046] S110. Add the pre-written intelligent monitoring code to the Spark application when starting the Spark application.

[0047]In this embodiment, the developer needs to pre-write the intelligent monitoring code, which can be JavaAgent code. When starting the Spark application, the jar package specified by the Java Agent code is written into the Spark application, and the preparation for the Spark application monitoring is completed. Work, and start monitoring according to the monitoring items defined by the developer without intrusion into the source code of the Spark application.

[0048] S120. When the Spark application is running, monitor the Spark application according to the intelligent monitoring code to obtain monitoring data, so as to determine whether the Spark application is abnormal.

[0049] S130. If the Spark appli...

Embodiment 2

[0053] Such as figure 2 As shown, Embodiment 2 of the present invention provides a Spark application monitoring method. Embodiment 2 of the present invention is a further explanation on the basis of Embodiment 1 of the present invention. The method includes:

[0054] S200. Determine whether to enable intelligent monitoring.

[0055] In this embodiment, if you want to monitor the Spark application, you first need to determine whether to enable the intelligent monitoring set by the developer. For different application scenarios, for example, if the Spark application that needs to be monitored is more important, then you need to enable the intelligent monitoring. Execute step S210-step S214. For example, if the current system computing resources are insufficient or there is a running time limit, it is not necessary to enable intelligent monitoring, and execute step S220-step S225.

[0056] S210. If it is necessary to enable the intelligent monitoring, add a pre-written intellig...

Embodiment 3

[0075] Such as image 3 As shown, Embodiment 3 of the present invention provides a Spark application monitoring system 100. The Spark application monitoring system 100 provided in Embodiment 3 of the present invention can execute the Spark application monitoring method provided in any embodiment of the present invention, and has the execution method corresponding functional modules and beneficial effects. The Spark application monitoring system 100 includes a code adding module 200 , an application monitoring module 300 and an email sending module 400 .

[0076] Specifically, the code adding module 200 is used to add pre-written intelligent monitoring codes to the Spark application when starting the Spark application; the application monitoring module 300 is used to monitor the Spark application according to the intelligent monitoring code when the Spark application is running. Application obtains monitoring data, to judge whether abnormal situation occurs in described Spark ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a Spark application monitoring method, system and device and a storage medium. The Spark application monitoring method comprises the steps of adding a pre-written intelligent monitoring code to a Spark application when the Spark application is started; when the Spark application runs, monitoring the Spark application according to the intelligent monitoringcode to obtain monitoring data so as to judge whether the Spark application is abnormal or not; and if the Spark application has an abnormal condition, recording the abnormal condition and sending anabnormal reason and a solution of the abnormal condition to a preset account through a mail. According to the embodiment of the invention, the intelligence of Spark application monitoring is realized.

Description

technical field [0001] Embodiments of the present invention relate to computer technology, and in particular to a Spark application monitoring method, system, device and storage medium. Background technique [0002] Spark is a real-time offline computing framework for big data. Hadoop is a distributed system infrastructure. Spark is an open source cluster computing environment similar to Hadoop, but Spark enables memory distributed data sets. In addition to providing interactive queries, It also optimizes iterative workloads. Although Spark was created to support iterative jobs on distributed datasets, it is actually complementary to Hadoop and can run in parallel on the Hadoop file system. [0003] The existing Spark application monitoring solution is mainly to train the tasks in the Hadoop Yarn task pool regularly, and then analyze the status of the target task through the set template, and then judge whether to restart according to the status obtained by the analysis, an...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/30
CPCG06F11/3051
Inventor 刘明
Owner 深圳市卡数科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com