Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A method and device for transmitting spark analysis results

A technology for analyzing results and transmission methods, applied in the field of big data, can solve problems such as the inability to improve the security and integrity of data transmission in Spark batch processing applications, to ensure fast and real-time performance, improve security, and prevent damage and damage Effect

Active Publication Date: 2021-03-16
武汉美和易思数字科技有限公司
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the present invention proposes a transmission method and device for Spark analysis results, aiming to solve the problem that the prior art cannot improve the security and integrity of data transmission in Spark batch processing applications

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and device for transmitting spark analysis results
  • A method and device for transmitting spark analysis results
  • A method and device for transmitting spark analysis results

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030]The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0031] like figure 1 As shown, the device may include: a processor 1001 , such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 . Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), a...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a method and a device for transmitting a Spark analysis result. The method comprises the following steps: acquiring data of a Dataset [A] type, and converting the data of the Dataset [A] type into data of a List [JsObject] type; creating JsObject data, storing the data of the List [JsObject] type into the JsObject data, and encrypting the JsObject data, so as to obtain the encrypted JsObject data; and sending the encrypted JsObject data through an HTTP POST request and waiting for the Spark data updating, according to the invention, the transmission of the Spark batch processing result is realized through the HTTP request, and the content to be sent is encrypted by utilizing the encrypted character string generated by the local rule, so that the safety and the stability of the Spark batch processing result are ensured.

Description

technical field [0001] The invention relates to the technical field of big data, and in particular, to a method and device for transmitting Spark analysis results. Background technique [0002] With the development of the era of big data, more and more enterprises are using Spark to process and analyze batch or stream data. HDFS, relational database and Kafka are three common result storage methods. HDFS is suitable for the storage of Spark batch results, including intermediate results and final results. The final results are often aggregated statistics, so the amount of data is small. Since Spark batch programs are generally executed repeatedly, in order to obtain the latest final result data, users have to write a program to periodically read the result directory on HDFS. On the one hand, the writing, deployment, operation and maintenance of the result reading program will increase a part of the cost. On the other hand, the real-time performance of the result acquisition ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F21/60G06F9/54
CPCG06F9/546G06F21/602
Inventor 海克洪张成成
Owner 武汉美和易思数字科技有限公司
Features
  • Generate Ideas
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com