A Spark data processing method and system based on Redis

A data processing system and data processing technology, applied in database management systems, electronic digital data processing, special data processing applications, etc., can solve the problem that the analysis speed of the Spark data processing framework cannot meet the requirements, so as to reduce complexity and overhead, The effect of improving performance and speeding up operation

Pending Publication Date: 2019-06-21
武汉烽火众智智慧之星科技有限公司
View PDF3 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The purpose of the present invention is to provide a Redis-based Spark data processing method and system, which is inte

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Spark data processing method and system based on Redis
  • A Spark data processing method and system based on Redis
  • A Spark data processing method and system based on Redis

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0028] like figure 1 As shown, an embodiment of the present invention provides a Redis-based Spark data processing method for business analysis, and the method includes the following steps:

[0029] S1, the Spark data processing framework obtains data from the distributed data source kafka, and converts the data into an elastic distributed data set RDD;

[0030] S2, the Spark data processing framework performs preliminary processing ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a Spark data processing method and system based on Redis, and the method comprises the following steps: S1, a Spark data processing framework obtains data from a distributed data source kafka, and converts the data into an elastic distributed data set RDD; S2, the Spark data processing framework performs preliminary processing on data in the elastic distributed data set RDDto generate a new distributed data set RDD, and stores the new distributed data set RDD into a Redis storage device; S3, the Redis storage device performs deeper processing on the distributed data setRDD generated after the preliminary processing to generate a new result data set, and provides an external fast access interface; And S4, the Spark data processing framework obtains a result data setgenerated after deeper processing through the access interface, performs complex logic processing according to business requirements, and writes a final result into the distributed data source kafka.The Spark data processing framework and the Redis storage device are combined, so that the performance of processing and analyzing tasks is greatly improved.

Description

technical field [0001] The invention relates to the field of computer data processing, in particular to a Redis-based Spark data processing method and system. Background technique [0002] Many companies increasingly need to analyze transactions as fast as the business transactions themselves, more and more decisions are being automated, and the analytics required to drive those decisions should be done in real time. Spark is an excellent general-purpose data processing framework that can achieve quasi-real-time data processing; Spark draws on open source algorithms and distributes processing tasks to computing node clusters, regardless of the type of data analysis that can be performed on a single platform In terms of performance, or in terms of the speed at which these tasks are performed, Spark's generation framework easily outperforms traditional frameworks. However, the current Spark data processing framework still cannot meet the requirements of business analysis spee...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/25G06F16/2458
Inventor 陈军侠杨犀
Owner 武汉烽火众智智慧之星科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap