Unlock instant, AI-driven research and patent intelligence for your innovation.

A big data storage and parallel processing method

A parallel processing, big data technology, applied in electrical digital data processing, special data processing applications, database indexing, etc., can solve problems such as data impact, and achieve the effects of avoiding loss, improving security and reliability, and improving utilization.

Active Publication Date: 2019-12-20
武汉烽火普天信息技术有限公司
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In the existing technology, the data is concentrated in the same storage, and if a failure occurs, all the data will be affected. Therefore, we propose a large data storage and parallel processing method

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0020] The present invention will be further described below in combination with specific embodiments.

[0021] The present invention proposes a large data storage and parallel processing method, comprising the following steps:

[0022] S1: Establish a storage platform according to the needs, the specific steps are as follows:

[0023] A1. Establish a management server and multiple internal memories and form a cluster to process large data requests in parallel;

[0024] A2. Number and classify the multiple internal memories in A1, and record them in the management server, and when storing in the internal memory, perform data classification and storage. The storage platform is also equipped with signal denoising, data compression, Data analysis, data encryption and data decryption functions to ensure the normal operation of the storage platform;

[0025] A3. Establish multiple storage units and set them in different areas, and connect them in real time through wireless commun...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a storing and parallel processing method of big data. The method comprises the following steps that a storage platform is built according to requirements, mass data needs to be queried and analyzed, the data is subjected to parallel excavation treatment and transplanted to an Hadoop platform through a common machine learning algorithm, and parallelization is achieved; the fetched data after parallel processing is stored into the storage platform, processed by the storage platform, classified and stored into different storage units remotely. According to the storing and parallel processing method of big data, the fetched data can be classified and stored, the safety and reliability of the data can be effectively improved, database resources are sufficiently used, the use ratio of an existing concentrated database is improved, and a user can also backup the data, so that loss of the data is prevented.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to a large data storage and parallel processing method. Background technique [0002] With the rapid development of Internet technology, how to effectively supervise Internet information has become a major issue that needs to be solved urgently by network information departments at all levels, especially the continuous development and changes of self-media and new media technology, which poses severe challenges to online public opinion guidance and information control. To meet the challenges, we must constantly adapt to new changes in the situation and improve the technical means and working platform of the network information department. [0003] In recent years, with the deep integration of informatization and data storage and the rapid development of Internet of Things technology, given that high-speed optical fiber data networks and wireless transmission have been widely po...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/22G06F16/951G06F16/953G06F16/9535G06F16/2453
CPCG06F16/2219G06F16/24532G06F16/951
Inventor 杨绪升
Owner 武汉烽火普天信息技术有限公司