A Spark application program control method and device

An application and parameter technology, applied in the field of communication, can solve problems such as reducing data delay, increasing application applicability, data loss, etc., to achieve the effect of improving availability, avoiding data loss, and ensuring acceptability
CN109634784AActive Publication Date: 2019-04-16KANGCHENG INVESTMENT CHINA
6 Cites 1 Cited by

Patent Information

Authority / Receiving Office
CN ยท China
Patent Type
Applications(China)
Current Assignee / Owner
KANGCHENG INVESTMENT CHINA
Publication Date
2019-04-16

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
Patent Text Reader

Abstract

The invention relates to the technical field of communication, in particular to a Spark application program control method and device. The control method for the Spark application program comprises the following steps: judging whether a historical state parameter of the Spark application program is effective or not, and if yes, adding the historical state parameter into an operation environment ofthe Spark application program; The Spark application program is operated according to the acquired historical state parameters; Judging whether the Spark application program operates abnormally or not, and if yes, keeping the current state parameter of the Spark application program; And storing the current state parameters in an administrator to update the historical state parameters. According to the method, the running accessibility of the Spark application program is ensured, the problems of data loss, data repeated consumption and the like caused by discontinuous states are avoided, and the usability of the program is improved.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present invention relates to the field of communication technology, in particular to a Spark application program control method and a control device. Background technique

[0002] Spark Streaming decomposes streaming computing into a series of short batch jobs. The batch processing engine here is Spark, that is, the input data of Spark Streaming is divided into pieces of data (Discretized Stream) according to the batch size (such as 1 second), and each piece of data is converted into RDD (Resilient Distributed Dataset) in Spark, and then The Transformation operation on DStream in Spark Streaming is changed to the Transformation operation on RDD in Spark, and the RDD is converted into an intermediate result and stored in memory. The entire streaming computing can superimpose the intermediate results according to the needs of the business, or store them in external devices. Kafka is a distributed publish-subscribe messaging system. It was original...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More