Data packaging method, electronic equipment and storage medium

A technology of data encapsulation and metadata, applied in the field of big data programming, can solve a lot of labor costs and other problems, and achieve the effect of reducing labor cost input

Pending Publication Date: 2020-12-08
CHINANETCENT TECH
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, due to the complexity of Spark development, when implementing big data Spark development, developers need to have a deep understanding of Spark principles and underlying technologies, such as broadcast variables (broadcast), RDD (Resiliennt Distributed Datasets, Elastic Distributed Datasets) operators etc., which requires a lot of manpower to train dedicated Spark developers

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data packaging method, electronic equipment and storage medium
  • Data packaging method, electronic equipment and storage medium
  • Data packaging method, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the following will describe each embodiment of the present invention in detail with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that, in each implementation manner of the present invention, many technical details are provided for readers to better understand the present application. However, even without these technical details and various changes and modifications based on the following implementation modes, the technical solution claimed in this application can also be realized. The division of the following implementations is for the convenience of description, and should not constitute any limitation to the specific implementations of the present invention, and the implementations can be combined and referenced to each other on the premise of no contradiction.

[0043] This embodiment rela...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data packaging method, electronic equipment and a storage medium. The method comprises the following steps of marking to-be-processed data as an elastic distributed data setcharacter string type object according to the metadata corresponding to the to-be-processed data, and filtering the to-be-packaged metadata from the metadata according to the preset filtering condition; converting the elastic distributed data set character string type object into an elastic distributed data set structured type object by taking a to-be-packaged element number as a filtering condition, and converting the to-be-packaged metadata and the elastic distributed data set structured type object into a Dataset object; and packaging obtained three objects into a data object in an RDD format which can be queried by using SQL statements, so that developers can realize the development of big data Spark only by using the SQL statements without deeply understanding the Spark principle andthe underlying technology, thereby effectively reducing the investment of labor cost.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of big data programming, and in particular to a data encapsulation method, electronic equipment and storage media. Background technique [0002] Apache Spark is a fast and general-purpose engine designed for distributed memory computing of large-scale distributed data. It is an open source Hadoop MapReduce-like general-purpose parallel framework provided by the AMP Lab at the University of California, Berkeley. Because Spark can save the intermediate output results of MapReduce Job in memory, it no longer needs to read and write HDFS (Hadoop Distributed File System, distributed file system). Therefore, Spark can be better suitable for data mining and machine learning that require iteration Algorithm of MapReduce. [0003] However, due to the complexity of Spark development, when implementing big data Spark development, developers need to have a deep understanding of Spark principl...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/242G06F16/2455G06F40/151
CPCG06F16/2433G06F16/2455G06F40/151Y02D10/00
Inventor 何通庆陈斌连庆仁
Owner CHINANETCENT TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap