Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data processing method, device, computer equipment and storage medium based on spark Streaming

A technology for data processing and data storage, applied in electrical digital data processing, special data processing applications, computing, etc., can solve problems such as reduced processing request capability and reduced SparkStreaming throughput, to improve throughput, reduce the number of requests, and reduce The effect of the number of files

Active Publication Date: 2022-04-08
RUN TECH CO LTD BEIJING
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a data processing method, device, computer equipment, and storage medium based on Spark Streaming, the purpose of which is to solve the problem of throughput drop when Spark Streaming performs data processing due to the drop of NameNode's ability to process requests

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, device, computer equipment and storage medium based on spark Streaming
  • Data processing method, device, computer equipment and storage medium based on spark Streaming
  • Data processing method, device, computer equipment and storage medium based on spark Streaming

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0025] figure 1 It is a flow chart of the data processing method based on Spark Streaming provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation where Spark Streaming is used for data processing for a long time, and the method can be executed by the data processing device based on Spark Streaming provided by the embodiment of the present invention, and the device can be implemented by hardware and / or software, Generally, it can be integrated into a computer device, such as a personal computer (Personal Computer, PC). Such as figure 1 As shown, it specifically includes the following steps:

[0026] S11. Obtain at least two first to-be-executed tasks of the current batch, and merge them into a second to-be-executed task, wherein the first to-be-executed tasks are encapsulated in an elastic distributed data set RDD.

[0027] Spark is a fast and general-purpose computing engine designed for large-scale data processing. It is an open-so...

Embodiment 2

[0051] figure 2 It is a flow chart of the data processing method based on Spark Streaming provided by Embodiment 2 of the present invention. The technical solution of this embodiment is further refined on the basis of the above-mentioned technical solution. In this embodiment, the situation of selecting the distributed publish-subscribe message system Kafka for the data source is further explained, and the technical solution is refined by using the unique structure of Kafka . Such as figure 2 As shown, it specifically includes the following steps:

[0052] S21. Obtain at least two first to-be-executed tasks of the current batch.

[0053] For the current batch, the first task to be executed, etc. in this embodiment, reference may be made to the description of the foregoing embodiments.

[0054] S22. Obtain partition information and offset information of the RDD to which the first task to be executed belongs in the distributed publish-subscribe message system.

[0055] Th...

Embodiment 3

[0075] image 3 A schematic structural diagram of a data processing device based on Spark Streaming provided by Embodiment 3 of the present invention. The device can be realized by means of hardware and / or software, and generally can be integrated into computer equipment, such as a personal computer (Personal Computer, PC). Such as image 3 As shown, the device includes:

[0076] A task merging module 31, configured to obtain at least two first to-be-executed tasks of the current batch and merge them into a second to-be-executed task, wherein the first to-be-executed task is encapsulated in the elastic distributed data set RDD;

[0077] The task execution module 32 is configured to execute the second to-be-executed task and store the execution result data in the distributed file system.

[0078] In the technical solution provided by the embodiment of the present invention, the acquired tasks of the current batch are combined and then executed, and the execution result data ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a data processing method, device, computer equipment and storage medium based on Spark Streaming. The method includes: obtaining at least two first tasks to be executed in the current batch, and merging them into a second task to be executed, wherein the first task to be executed is encapsulated in an elastic distributed data set RDD; executing the second task to be executed And store the execution result data in the distributed file system. The technical solution provided by the embodiment of the present invention solves the problem of decreased throughput of Spark Streaming during data processing due to the decreased processing request capability of the NameNode, reduces the number of files generated by the task, that is, reduces the number of requests sent to the NameNode The number, thereby improving the stability of NameNode's ability to process requests, thereby improving the throughput of Spark Streaming data processing.

Description

technical field [0001] Embodiments of the present invention relate to the technical field of big data real-time computing, and in particular to a data processing method, device, computer equipment and storage medium based on Spark Streaming. Background technique [0002] Spark Streaming is a stream processing framework, which has the characteristics of quasi-real-time, micro-batch processing, scalability, high throughput and fault tolerance, and is widely used in large-scale data processing. In the actual data processing process, it is usually necessary to obtain the result data and store it in a distributed file storage system, such as Hadoop Distributed File System (Hadoop Distributed File System, HDFS). Since HDFS is open source, there are many applications in the case of using Spark Streaming for data processing. However, the application programming interface (Application Programming Interface, API) that comes with Spark Streaming usually outputs a wave of data for each...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48G06F16/182
CPCG06F9/4881G06F16/182
Inventor 陶劲程强万月亮冯宇波
Owner RUN TECH CO LTD BEIJING
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com