Real-time large data processing system and method based on distributed internal memory calculation

A memory computing and real-time processing technology, applied in the field of big data processing, can solve problems such as the lack of real-time processing methods for big data, and achieve the effect of improving usage efficiency

Inactive Publication Date: 2015-05-20
ECDATA INFORMATION TECHNOLOGY CO LTD
View PDF7 Cites 14 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Real-time computing has a lot of demand in the Internet, financial and other scenarios, bu

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time large data processing system and method based on distributed internal memory calculation
  • Real-time large data processing system and method based on distributed internal memory calculation
  • Real-time large data processing system and method based on distributed internal memory calculation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] Such as figure 1 As shown, the application scenario of the present invention includes users and big data platform manufacturers. Users provide real-time data sources, and data is continuously generated; data is continuously collected by means of data collection and continuously transmitted to the big data platform. Through the big data real-time processing system and method based on distributed memory computing, the big data platform forms a logical resource pool of multiple memory nodes 20 distributed in the system, and the resource scheduler 10 processes the memory resources according to the actual needs of tasks. Allocate, designate the corresponding memory nodes 20 in the resource pool as memory computing containers, each analyze and calculate the data recorded in it, and quickly obtain the corresponding calculation results; then combine the calculation results of each memory node 20 to get the full amount The calculation result of the data is sent to the user.

[...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a real-time large data processing system and method based on distributed internal memory calculation. The real-time large data processing method comprises the following steps: a resource pool is formed by respective internal memory resources of multiple distributed internal memory nodes; when a user originates a task, a client transfers the task and incremental data related to the task within a fixed time interval to a resource scheduler of the resource pool to be applied for execution; the resource scheduler can be used for carrying out deployment in the resource pool according to the internal memory resources needed by the task, corresponding internal memory nodes in the resource pool are appointed for carrying out calculation on the incremental data related to the task, obtained local results of respective calculation are stored in the resource pool; and local results, obtained by virtue of calculation of the appointed internal memory nodes, are calculated in a combination mode to obtain a global calculation result, and the global calculation result is sent to the user. The real-time large data processing system and method is capable of achieving centralized use of distributed internal memories, restoring data in the process of the internal memory calculation and achieving real-time data incremental calculation, is high in data access speed and can be used for providing a real-time analysis result for the user.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a real-time big data processing system and method based on distributed memory computing. Background technique [0002] In recent years, with the rapid development and popularization of computer and information technology, the scale of industrial application systems has expanded rapidly, and the data generated by industrial applications has grown explosively. Industry / enterprise big data that can easily reach hundreds of terabytes (terabytes) or even tens to hundreds of petabytes (petabytes) has far exceeded the processing capabilities of existing traditional computing technologies and information systems. Therefore, seeking Effective big data processing technologies, methods and means have become an urgent need in the real world. "Big data" has existed for a long time in the fields of physics, biology, environmental ecology, military, finance, communication and other industrie...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/48G06F9/50
Inventor 姚海
Owner ECDATA INFORMATION TECHNOLOGY CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap