A Spark application program control method and device

An application and parameter technology, applied in the field of communication, can solve problems such as reducing data delay, increasing application applicability, data loss, etc., to achieve the effect of improving availability, avoiding data loss, and ensuring acceptability

Active Publication Date: 2019-04-16
KANGCHENG INVESTMENT CHINA
View PDF6 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a Spark application program control method and control device to solve the problem in the prior art that the Spark application program is prone to data loss after interruption, so as to reduce data delay and increase the applicability of the application program

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Spark application program control method and device
  • A Spark application program control method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0038] The specific implementation of the Spark application program control method and control control provided by the present invention will be described in detail below in conjunction with the accompanying drawings.

[0039] This specific embodiment provides a kind of Spark application program control method, appended figure 1 It is a flow chart of the Spark application program control method in the specific embodiment of the present invention. Such as figure 1 As shown, the Spark application control method provided in this specific embodiment includes the following steps:

[0040] Step S11, judging whether the historical state parameters of the Spark application are valid, and if so, adding the historical state parameters to the running environment of the Spark application. Wherein, the historical state parameters refer to the state parameters saved in the process of running the Spark application before the current running of the Spark application. The historical state p...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the technical field of communication, in particular to a Spark application program control method and device. The control method for the Spark application program comprises the following steps: judging whether a historical state parameter of the Spark application program is effective or not, and if yes, adding the historical state parameter into an operation environment ofthe Spark application program; The Spark application program is operated according to the acquired historical state parameters; Judging whether the Spark application program operates abnormally or not, and if yes, keeping the current state parameter of the Spark application program; And storing the current state parameters in an administrator to update the historical state parameters. According to the method, the running accessibility of the Spark application program is ensured, the problems of data loss, data repeated consumption and the like caused by discontinuous states are avoided, and the usability of the program is improved.

Description

technical field [0001] The present invention relates to the field of communication technology, in particular to a Spark application program control method and a control device. Background technique [0002] Spark Streaming decomposes streaming computing into a series of short batch jobs. The batch processing engine here is Spark, that is, the input data of Spark Streaming is divided into pieces of data (Discretized Stream) according to the batch size (such as 1 second), and each piece of data is converted into RDD (Resilient Distributed Dataset) in Spark, and then The Transformation operation on DStream in Spark Streaming is changed to the Transformation operation on RDD in Spark, and the RDD is converted into an intermediate result and stored in memory. The entire streaming computing can superimpose the intermediate results according to the needs of the business, or store them in external devices. Kafka is a distributed publish-subscribe messaging system. It was original...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F11/14
CPCG06F11/1438
Inventor 文燕军卢春洋吴文钧
Owner KANGCHENG INVESTMENT CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap