Data processing method and device based on Spark Streaming, computer equipment and storage medium

A data processing and data storage technology, applied in the direction of electrical digital data processing, special data processing applications, computing, etc., can solve the problems of SparkStreaming throughput decline, processing request ability decline and other problems, to improve throughput, reduce the number of requests, improve The effect of stability

Active Publication Date: 2020-03-24
RUN TECH CO LTD BEIJING
View PDF2 Cites 14 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a data processing method, device, computer equipment, and storage medium based on Spark Streaming, the purpose of which is to solve the problem of throughput drop when Spark Streaming performs data processing due to the drop of NameNode's ability to process requests

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device based on Spark Streaming, computer equipment and storage medium
  • Data processing method and device based on Spark Streaming, computer equipment and storage medium
  • Data processing method and device based on Spark Streaming, computer equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0025] figure 1 It is a flow chart of the data processing method based on Spark Streaming provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation where Spark Streaming is used for data processing for a long time, and the method can be executed by the data processing device based on Spark Streaming provided by the embodiment of the present invention, and the device can be implemented by hardware and / or software, Generally, it can be integrated into a computer device, such as a personal computer (Personal Computer, PC). Such as figure 1 As shown, it specifically includes the following steps:

[0026] S11. Obtain at least two first to-be-executed tasks of the current batch, and merge them into a second to-be-executed task, wherein the first to-be-executed tasks are encapsulated in an elastic distributed data set RDD.

[0027] Spark is a fast and general-purpose computing engine designed for large-scale data processing. It is an open-so...

Embodiment 2

[0051] figure 2 It is a flow chart of the data processing method based on Spark Streaming provided by Embodiment 2 of the present invention. The technical solution of this embodiment is further refined on the basis of the above-mentioned technical solution. In this embodiment, the situation of selecting the distributed publish-subscribe message system Kafka for the data source is further explained, and the technical solution is refined by using the unique structure of Kafka . Such as figure 2 As shown, it specifically includes the following steps:

[0052] S21. Obtain at least two first to-be-executed tasks of the current batch.

[0053] For the current batch, the first task to be executed, etc. in this embodiment, reference may be made to the description of the foregoing embodiments.

[0054] S22. Obtain partition information and offset information of the RDD to which the first task to be executed belongs in the distributed publish-subscribe message system.

[0055] Th...

Embodiment 3

[0075] image 3 A schematic structural diagram of a data processing device based on Spark Streaming provided by Embodiment 3 of the present invention. The device can be realized by means of hardware and / or software, and generally can be integrated into computer equipment, such as a personal computer (Personal Computer, PC). Such as image 3 As shown, the device includes:

[0076] A task merging module 31, configured to obtain at least two first to-be-executed tasks of the current batch and merge them into a second to-be-executed task, wherein the first to-be-executed task is encapsulated in the elastic distributed data set RDD;

[0077] The task execution module 32 is configured to execute the second to-be-executed task and store the execution result data in the distributed file system.

[0078] In the technical solution provided by the embodiment of the present invention, the acquired tasks of the current batch are combined and then executed, and the execution result data ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a data processing method and device based on Spark Streaming, computer equipment and a storage medium. The data processing method comprises the steps that atleast two first to-be-executed tasks of the current batch are acquired and combined into a second to-be-executed task, and the first to-be-executed tasks are packaged in an elastic distributed data set RDD; and the second to-be-executed task is executed and execution result data is stored to the distributed file system. The technical scheme solves the problem that the throughput is reduced when the Spark Streaming carries out data processing due to the fact that the request processing capacity of the NameNode is reduced, and reduces the number of files generated by the task, that is, the number of requests sent to the NameNode is reduced, so that the stability of the request processing capability of the NameNode is improved, and the throughput during Spark Streaming data processing is improved.

Description

technical field [0001] Embodiments of the present invention relate to the technical field of big data real-time computing, and in particular to a data processing method, device, computer equipment and storage medium based on Spark Streaming. Background technique [0002] Spark Streaming is a stream processing framework, which has the characteristics of quasi-real-time, micro-batch processing, scalability, high throughput and fault tolerance, and is widely used in large-scale data processing. In the actual data processing process, it is usually necessary to obtain the result data and store it in a distributed file storage system, such as Hadoop Distributed File System (Hadoop Distributed File System, HDFS). Since HDFS is open source, there are many applications in the case of using Spark Streaming for data processing. However, the application programming interface (Application Programming Interface, API) that comes with Spark Streaming usually outputs a wave of data for each...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F16/182
CPCG06F9/4881G06F16/182
Inventor 陶劲程强万月亮冯宇波
Owner RUN TECH CO LTD BEIJING
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap