Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

High performance data processing system based on memory computing

A data processing system and memory computing technology, applied in database management systems, electronic digital data processing, special data processing applications, etc., can solve the problems of slow efficiency of big data, etc.

Active Publication Date: 2019-04-09
MERIT DATA CO LTD
View PDF9 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is to provide a high-performance data processing system based on memory computing, to solve the problem of too slow big data efficiency in the process of data statistics, and to use memory computing technology to compress data into memory in a predefined way, Eliminate the I / O bottleneck in the statistical process, and finally realize the second-level statistical calculation of tens of millions to billions of data on ordinary single-node hardware

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • High performance data processing system based on memory computing
  • High performance data processing system based on memory computing
  • High performance data processing system based on memory computing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0024] In order to make the technical means, creative features, goals and effects achieved by the present invention easy to understand, the present invention will be further described below in conjunction with specific illustrations.

[0025] refer to figure 1 As shown, the high-performance data processing system based on memory computing includes: a data acquisition interface definition module, which automatically extracts relevant fields of the data table by selecting the data table, and defines the compression method by the user according to the field type and range;

[0026] A data real-time acquisition module, which extracts relevant data from the data source according to the metadata defined by the data acquisition interface definition module, wherein the metadata generally refers to the data defining the data format, and the data source refers to specific business data, generally referring to the relational database of the enterprise business system;

[0027] A data mem...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a high-performance data processing system based on memory calculation. The high-performance data processing system comprises a data collecting interface defining module, a data real-time collecting module, a data memory storage module and a data processing module. By means of the data collecting interface defining module, a data table is selected, related fields of the data table are automatically extracted, and a user defines a compression mode according to the type and range of the fields. The data real-time collecting module is used for extracting related data from a data source according to metadata defined by the data collecting interface defining module, wherein the metadata normally refer to data in a defined data format, and the data source refers to specific service data and normally a relation database of an enterprise service system. The data memory storage module is used for storing the data in a memory through a dictionary mode and a shaping, coding and compressing algorithm in a syllable code mode. The data processing module comprises an SQL analyzing module and is used for extracting records in statistics according to needs. According to the high-performance data processing system based on memory calculation, the data are stored in the memory in a byte mode through compression and reconstruction on the premise that the statistical precision and speed are not influenced.

Description

technical field [0001] The invention relates to a high-performance data processing system based on memory computing, which uses Java platform and Java NIO technology to realize high-speed statistics of big data. Background technique [0002] Existing statistical technologies are generally implemented through relational database storage procedures or SQL statistical procedures, which are limited by network I / O and disk I / O bottlenecks, and problems such as slow response and excessive database pressure will occur when large data are generated. [0003] For larger data, Hadoop map / reduce parallel computing technology is used. This technology requires huge hardware resources. Generally, 5 to 15 nodes or more can be applied (depending on hardware performance), and the statistical task startup time is at least 10 seconds. , which cannot meet the scenarios with high real-time performance requirements. Contents of the invention [0004] The technical problem to be solved by the p...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25G06F16/22G06F16/2453
CPCG06F16/2228G06F16/24534G06F16/25G06F16/254
Inventor 程宏亮卢耀宗郭联伟强劲张小平
Owner MERIT DATA CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products