Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data dynamic processing method based on distributed storage

A distributed storage and data dynamic technology, applied in the field of storage data, can solve the problems of central data storage and the inability of the platform to meet the needs of customers in a short time, and achieve the effect of eliminating the barriers of data capacity

Pending Publication Date: 2020-09-01
上海鼎茂信息技术有限公司
View PDF0 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

3. With data indexing function, data search speed is fast; but its disadvantages are also obvious. Although this type of data processing platform supports distributed architecture deployment, data analysis nodes need to store data in the central data due to the relationship between indexers. in storage
However, for temporary data analysis, this type of platform cannot meet the needs of customers in a short time

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data dynamic processing method based on distributed storage
  • Data dynamic processing method based on distributed storage
  • Data dynamic processing method based on distributed storage

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0023] The present invention will be further described below in conjunction with accompanying drawing:

[0024] Such as Figure 1-3 Shown: A data processing method based on distributed storage that can support TB or even PB-level data capacity, but also has flexible and fast search capabilities, and can reuse data in the middle of data processing, has the following three parts Service architecture composition.

[0025] These three parts include: edge acquisition layer, data analysis layer, and interaction layer.

[0026] The edge collection layer is a distributed data collection cluster composed of edge collection nodes. Each edge collection node assumes two functional roles: 1. Data collection; 2. Data management. Mainly carry out the collection, storage, and forwarding of raw data.

[0027] Data collection: Each edge collection node supports multiple types of data collection, and supports various data sources to access data collection nodes through specific data transmis...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data dynamic processing method based on distributed storage. A system architecture comprises an edge data acquisition layer, a data analysis layer and an interaction layer. The method comprises the steps of 1, responding to a search request; 2, splitting the search request; 3, processing a search request; 4, generating a data request; 5, carrying out data primary search;6, forwarding data; 7, analyzing indexes; 8, a search analysis engine processing the data; 9, summarizing fragmentation search analysis results; and 10, visually displaying the interaction layer. A method for generating a data request according to a fragmentation rule, creating a data forwarding instance to realize dynamic forwarding of data from an edge collection node to an analysis node and dynamically loading an analysis index engine is introduced in a data processing process, the difficulty that dynamic search can be supported and the barrier of data capacity caused by central storage canbe eliminated is solved, and a reliable data processing method is provided for big data analysis.

Description

technical field [0001] The invention relates to the technical field of stored data, in particular to a method for dynamically processing data based on distributed storage. Background technique [0002] With the advent of the era of big data and 5G, analysis platforms based on time series data are increasingly being used in various fields of social production. With the changes brought about by 5G data transmission technology, more and more information data is included in the category of data processing, which makes the volume and types of data covered in the field of big data data analysis become larger and larger. . Due to the need to analyze the potential relationship between various types of large-scale data, the barriers to data pertinence have been repeatedly lowered. The existing data platform has risen to a new height in the support of data types. [0003] At present, various data processing platforms on the market have various types of data access functions, and su...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/182G06F16/13G06F16/14
CPCG06F16/134G06F16/156G06F16/182
Inventor 沈慧
Owner 上海鼎茂信息技术有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com