Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method and device for multi-parallel processing frame

A data processing device and data processing technology, applied in the direction of electrical digital data processing, special data processing applications, multi-programming devices, etc., to achieve the effects of improving versatility, improving data processing capabilities, and reducing operational difficulty

Active Publication Date: 2017-12-15
HUNAN UNIV
View PDF2 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0013] In view of the diversity of existing data storage media, the complexity and polymorphism of operational data, the processing requirements of stream data and unstructured data, etc., the present invention provides a multi-parallel processing framework for solving the above problems Data processing method and device

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device for multi-parallel processing frame
  • Data processing method and device for multi-parallel processing frame
  • Data processing method and device for multi-parallel processing frame

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0059] This embodiment takes the application of the multi-parallel processing framework on the Spark platform as an example to elaborate a data processing method for the multi-parallel processing framework provided by the present invention. The flow chart of the data processing method is as follows figure 1 shown.

[0060] Step S1: Obtain configuration information of the usage environment.

[0061] It specifically includes: step S1.1: encapsulating the configuration information of the use environment and the environment acquisition method for obtaining the configuration information into a first class;

[0062] Step S1.2: Obtain the configuration information of the use environment through the environment acquisition method provided by the first class.

[0063] In this embodiment, the configuration information of the Spark platform, that is, the environment context SparkConf and JavaSparkContext are encapsulated into a class named CMContext, and the CMContext class is used to i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data processing method and device for a multi-parallel processing frame. The method includes the steps of S1, obtaining configuration information of a use environment; S2, selecting a corresponding data calling method and calling bottom layer storage data according to the configuration information of the use environment and the property of the bottom layer storage data, conducting data format converting on the obtained concentrated bottom layer storage data through a unified-format converting method to obtain a unified-format extract data set; S3, providing the unified-format extract data set for an upper layer application to use. By selecting the corresponding data calling method and calling the bottom layer storage data according to the configuration information of the use environment and the storage medium type of the bottom layer storage data and converting the bottom layer storage data into the unified-format extract data set through the unified-format converting method and providing the unified-format extract data set for the upper layer application to use, the expandability, universality and availability of the multi-parallel processing frame are improved.

Description

technical field [0001] The invention relates to the technical field of multi-parallel processing of big data, in particular to a data processing method and device for a multi-parallel processing framework. Background technique [0002] At present, the rapid development and application of big data technology has brought 4V challenges to existing technologies: scale (Volume), from terabyte (TB) level to petabyte (PB) level or even ten trillion billion Byte (ZB) level; Variety, a wide variety of data, including both traditional structured data and unstructured data such as text, video, pictures and audio, and the proportion of unstructured data is increasing rapidly; Value (Value), data value density is low, it is difficult to perform calculations such as predictive analysis, operational intelligence, decision support; Velocity (Velocity), the speed of big data processing is becoming more and more prominent, and timeliness is difficult to guarantee. Generally speaking, the dil...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30G06F9/54
CPCG06F9/545G06F9/547G06F16/116G06F16/182G06F16/2471G06F2209/544
Inventor 刘昆昆唐卓陈都
Owner HUNAN UNIV