Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A data processing method and device

A data processing and processing unit technology, applied in the field of big data, can solve problems such as bottlenecks, poor SparkonYARN cluster processing performance, and easy performance, and achieve the effect of improving processing performance, avoiding performance bottlenecks, and reducing hardware configuration requirements.

Active Publication Date: 2019-09-17
NEW H3C BIG DATA TECH CO LTD
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0016] The present invention provides a data processing method and device to solve the problem that the Spark onYARN cluster in the existing YARN-Cluster mode has poor processing performance and is prone to performance bottlenecks

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data processing method and device
  • A data processing method and device
  • A data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, some terms in Spark are briefly described below.

[0049] 1. Application: The process of combining multiple batch calculations can be physically expressed as a user-written program package + deployment configuration;

[0050] 2. RDD (Resilient Distributed Datasets, Resilient Distributed Datasets): A batch of data sets with the same source, same structure, and same purpose when spark executes distributed computing. This data set may be divided into multiple partitions (Partition) , distributed on different object nodes. When programming, the RDD object corresponds to this data set, and the RDD object is regarded as a basic unit of data operation. For example, performing a map (mapping) operation on an RDD object is actually equivalent to performing a map operation on each piece of data in each partition in the dataset;

[0051] 3. Partit...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data processing method and device. The method comprises the steps that after starting is completed, a resource allocation request is sent to a resource manager (RM) accordingto Spark Context initialization data saved in a shared memory layer; containers allocated as a response of the RM to the resource allocation request are received, and node managers (NMs) of nodes to which all the containers belong to are instructed to start executors in the allocated containers respectively; tasks are allocated to all the executors, the tasks executed by the executors are monitored, and data obtained by executing the tasks is saved to the shared memory layer. By applying the data processing method and device, the processing performance of a Spark on YARN cluster can be improved, the performance bottlenecks are avoided, and the requirement of hardware configuration of a server running a Spark Driver is reduced.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a data processing method and device. Background technique [0002] Spark is a distributed batch processing system and analysis mining engine based on the memory parallel computing framework, which provides efficient memory computing by distributing computing tasks of a large number of data sets to multiple computers. [0003] Spark application program (Application) runs on Spark Driver (driver). When Spark Drive starts, it creates a Spark Context (context), configures the Spark Context, and then initializes the Spark Context. [0004] After completing the Spark Context initialization, Spark Drive will convert the application into a task, and apply for resources from the cluster manager (such as YARN (Yet Another Resource Negotiator, another resource coordinator)) to run the Executor (executor) , assign tasks to each Executor according to the preset strategy. Among the...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/54G06F9/50
Inventor 赵英超
Owner NEW H3C BIG DATA TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com