Supercharge Your Innovation With Domain-Expert AI Agents!

Data node-based parallel massive data processing method

A mass data processing and data node technology, applied in the field of data processing, can solve the problems of slow mass data processing and low development efficiency, and achieve the effects of reducing development and maintenance costs, saving resources, and avoiding data loss

Active Publication Date: 2014-05-07
STATE GRID SHANDONG ELECTRIC POWER
View PDF8 Cites 13 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is the problem of slow processing of massive data and low development efficiency in the prior art, thereby providing a method of using distributed data storage technology to realize massive data access services and forming a parallel massive data processing method based on data nodes

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data node-based parallel massive data processing method
  • Data node-based parallel massive data processing method
  • Data node-based parallel massive data processing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0030] The data node-based parallel massive data processing method of the present invention, such as figure 2 As shown, it includes the following steps:

[0031] Compress and encapsulate data information into a data object, which integrates data compression, storage and access, and is also the basic unit of data processing, that is, a data object is used to compress, store and access a monitoring of a certain measurement point Indicators, through the reading interface of the data object, all the historical data of the monitoring indicator can be obtained, and the value of the monitoring indicator at a certain time can be stored through the writing interface of the data object, and the data object stores the data information encapsulated An access interface is provided, and the data objects can classify and store massive data, and can perform parallel access to data objects in multiple data nodes.

[0032] Divide massive data into a series of data nodes according to actual bu...

Embodiment 2

[0037] On the basis of the data node-based parallel mass data processing method described in Embodiment 1, the bus is an ORB bus of CORBA, and ORB (Object Request Broker) is an object request agent. Based on the ORB mechanism, distributed, interoperable object structures and interoperable applications can be fully utilized. This advantage is obvious. The details of data communication are completely encapsulated, and users can use different development languages ​​and environments to build shared resource applications without knowing the details of the implementation of the interaction, which is portable.

[0038] The access interface is a GID interface, and the GID interface is an interface of the IEC61970 GID (Generic Interface Definition) standard, which provides a set of general interfaces for data access and exchange between systems, and solves the application of the system and the sharing of data resources Problems, especially for the data interaction between those closel...

Embodiment 3

[0040] On the basis of the above embodiments, each data object corresponds to a database file, and storage space is pre-allocated for each database file to prevent uncertain situations such as insufficient disk space during operation, and to prevent disk fragmentation. , to ensure the stability and efficiency of data access performance.

[0041] The distributed storage of database files is currently difficult for database systems to associate multiple databases, and mass data processing technology can configure database files for complex data requirements to realize the construction of distributed storage networks, which can conveniently store important data Backup to avoid data loss or uncheckability caused by server failure; the huge storage structure is only a simple CORBA interface to the outside world, making it easy for users to use; in addition, distributed deployment can enhance the availability of database services and prevent partial network paralysis The resulting d...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a data node-based parallel massive data processing method. In the method, massive data are stored into data objects of different data nodes, and a client end can perform parallel access by different data objects at the same time, so the access efficiency is improved, the large-capacity and high-performance data processing service is realized, a distribution database system is established, a distribution data storage technology is provided for realizing massive data access service, and the data node-based parallel massive data processing method is formed. The method is also based on the IEC61970GID standard and the CORBA standard, supports rich clustering operation and meets the function demands on data processing, so the development efficiency is improved, the sources are saved, and the development and maintenance costs are reduced.

Description

technical field [0001] The invention relates to a data node-based parallel mass data processing method, belonging to the technical field of data processing. Background technique [0002] With the rapid development of computer technology and modern communication technology, the power system automation software industry is setting off a wave of networking and componentization. From the process bus at the plant station side to the power enterprise integrated bus at the master station side, comprehensive networking and Internet, IT's popular common object request broker architecture (common object request broker architecture, referred to as CORBA) began to be used in power system automation solutions, networking enables power system automation software to work together in a distributed environment, this distributed The system is often a heterogeneous system composed of different hardware, different operating systems, different support environments or products from different manu...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30H04L29/06
CPCG06F16/24532G06F16/25
Inventor 张宏伟赵生传王华广崔潇于强牟磊郭英雷李延真蔡键王德东李长林陈明潘新民彭博肖庆维段蕴毅
Owner STATE GRID SHANDONG ELECTRIC POWER
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com