Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and device for transmitting Spark analysis result

A technology for analysis results and transmission methods, applied in the field of big data, can solve problems such as the inability to improve the security and integrity of data transmission in Spark batch processing applications, to ensure fast and real-time performance, simple data transmission, and prevent damage and damage Effect

Active Publication Date: 2020-08-28
武汉美和易思数字科技有限公司
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the present invention proposes a transmission method and device for Spark analysis results, aiming to solve the problem that the prior art cannot improve the security and integrity of data transmission in Spark batch processing applications

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for transmitting Spark analysis result
  • Method and device for transmitting Spark analysis result
  • Method and device for transmitting Spark analysis result

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030]The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the implementation manners in the present invention, all other implementation manners obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of the present invention.

[0031] Such as figure 1 As shown, the device may include: a processor 1001 , such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 . Wherein, the communication bus 1002 is used to realize connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Key...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a method and a device for transmitting a Spark analysis result. The method comprises the following steps: acquiring data of a Dataset [A] type, and converting the data of the Dataset [A] type into data of a List [JsObject] type; creating JsObject data, storing the data of the List [JsObject] type into the JsObject data, and encrypting the JsObject data, so as to obtain the encrypted JsObject data; and sending the encrypted JsObject data through an HTTP POST request and waiting for the Spark data updating, according to the invention, the transmission of the Spark batch processing result is realized through the HTTP request, and the content to be sent is encrypted by utilizing the encrypted character string generated by the local rule, so that the safety and the stability of the Spark batch processing result are ensured.

Description

technical field [0001] The invention relates to the field of big data technology, in particular to a method and device for transmitting Spark analysis results. Background technique [0002] With the development of the big data era, more and more enterprises are using Spark to process and analyze batch or streaming data. HDFS, relational database and Kafka are three commonly used results storage methods. HDFS is suitable for the storage of Spark batch results, including intermediate results and final results. The final results are often aggregated statistics, so the amount of data is small. Since Spark batch programs are generally executed repeatedly at regular intervals, in order to obtain the latest final result data, users have to write a program to regularly read the result directory on HDFS. On the one hand, the writing, deployment, operation, and maintenance of the result reading program will increase part of the cost. On the other hand, the real-time performance of th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F21/60G06F9/54
CPCG06F9/546G06F21/602
Inventor 海克洪张成成
Owner 武汉美和易思数字科技有限公司
Features
  • Generate Ideas
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com