Design and realization method for mass data processing system

A mass data processing and massive data technology, applied in the direction of electrical digital data processing, special data processing applications, structured data retrieval, etc., can solve the problems such as difficult to guarantee scalability, meet the needs of convenient transmission, improve processing efficiency, and ensure reliability sexual effect

Inactive Publication Date: 2016-04-06
INSPUR GROUP CO LTD
View PDF0 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In traditional data, the amount of data stored, processed and analyzed is relatively small, and relational databases and parallel warehouses can be used for processing, so it is difficult to guarantee scalability
In

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Design and realization method for mass data processing system
  • Design and realization method for mass data processing system
  • Design and realization method for mass data processing system

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0019] Example

[0020] Hereinafter, referring to the accompanying drawings, a specific example is used to describe the design and implementation process of a massive data processing system for the content of the present invention.

[0021] As described in the summary of the invention, the design and implementation of a mass data processing system in the present invention includes three modules, namely a mass data collection module, a mass data storage module and a mass data processing module. The frame diagram of a massive data processing system is attached figure 1 Shown.

[0022] First install the operating system Centos6.3 on the three-node server, and then component the three-node cluster environment. Follow the installation steps to build the Hadoop platform and start HDFS, MapReduce, HBase, and Flume components. Prepare the environment for the processing of massive data.

[0023] The schematic diagram of the massive data collection module is attached figure 2 As shown, using...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a design and realization method for a mass data processing system. The modes of distributed storage, distributed computing framework and rapid processing are adopted, so that data storage and processing efficiency is improved and CPU and network resources are saved. The method involves a mass data collection module, a mass data storage module and a mass data processing module. The structural design mainly considers the problems in mass data collection, data storage and data processing. In order to make a mass data processing process more convenient, efficient and secure, the modes of distributed storage, distributed computing framework and rapid processing are adopted and the design and realization of the mass data processing system are realized, so that data storage and processing efficiency is improved and CPU and network resources are saved.

Description

technical field [0001] The invention relates to a data processing system, in particular to a design and implementation method of a massive data processing system. Background technique [0002] With the rapid development of the Internet age, people's lives have undergone tremendous changes. People will use the Internet to work, study and live, and it is very common to use the convenience of the Internet to find learning materials, work materials, and purchase daily necessities. Modern society has entered the high-tech era. All office environments and communication are carried out through electronic products. The speed of data generation and sharing has also increased exponentially, resulting in a dramatic increase in the amount of data. However, due to the complexity and diversity of data sources and types, the amount of data is very large, which is very different from traditional data. In traditional data, the amount of data stored, processed and analyzed is relatively sma...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/2471G06F16/182G06F16/27
Inventor 孙志云郭美思
Owner INSPUR GROUP CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap