Unlock instant, AI-driven research and patent intelligence for your innovation.

Big data aggregation computing method and device

A computing method and computing device technology, applied in the field of big data analysis, can solve problems such as low memory utilization

Active Publication Date: 2021-03-02
北京东方国信科技股份有限公司
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] In order to overcome the above-mentioned problem of low memory utilization in the existing big data aggregation computing or at least partially solve the above problems, embodiments of the present invention provide a method and device for big data aggregation computing

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data aggregation computing method and device
  • Big data aggregation computing method and device
  • Big data aggregation computing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0025] In one embodiment of the present invention, a large data aggregation calculation method is provided, figure 1 It is a schematic diagram of the overall flow of the large data aggregation calculation method provided by the embodiment of the present invention, the method includes: S101, read any row of raw data that has not been aggregated fro...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the present invention provides a large data aggregation calculation method and device, the method includes: reading any row of raw data that has not been aggregated and calculated from the memory, and according to the grouping field in the SQL statement that operates the original data, read The original data is grouped; if the key value of the group to which the original data belongs does not exist in the hash structure of the memory, then apply for memory to store the key value and the aggregation calculation result corresponding to the key value; if the memory is applied, Then store the key value in the hash structure, and store the aggregate calculation result obtained by the aggregate function calculation of the original data in the SQL statement in the memory of the application until all the data in the mdp data structure Raw data are aggregated and calculated. The embodiments of the present invention make full use of the memory and improve the computing efficiency.

Description

technical field [0001] Embodiments of the present invention belong to the technical field of big data analysis, and more specifically, relate to a big data aggregation calculation method and device. Background technique [0002] When doing big data analysis, it often involves aggregation calculations on the same data. Aggregation calculation refers to performing aggregate functions such as sum, count, max, and min on the value values ​​corresponding to the same key value based on one or more columns as key values. operation, output a single result row. [0003] Because the amount of raw data in big data analysis is too large, and the base of grouped key values ​​is also large, the remaining memory of the computer may not be enough to store all the data to be calculated. The existing technology may calculate the specified number of key values ​​each time. This method may cause the problem of not being able to apply for memory and cause the system to crash. It may also cause ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/50G06F16/22G06F16/28
CPCG06F9/5016
Inventor 杨鹏董隆超郭志弘
Owner 北京东方国信科技股份有限公司