Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Database-based big data incremental calculation processing method

A technology of incremental computing and processing methods, applied in the field of big data analysis, can solve the problems of lack of computing performance, high implementation cost, poor practicability, etc., and achieve the effect of saving computing resources, improving efficiency, and strong timing.

Pending Publication Date: 2021-10-01
NANJING BEIDOU INNOVATION & APPL TECH RES INST CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] At present, the existing incremental computing systems are mainly divided into two categories, one is InCoop, Inc-MR and other incremental computing systems based on the Hadoop computing framework, which lacks in computing performance and needs to be customized for the incremental computing framework The other is the incremental computing system dedicated to commercial companies such as Percolator and CBP, which has poor versatility and high implementation costs.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Database-based big data incremental calculation processing method
  • Database-based big data incremental calculation processing method
  • Database-based big data incremental calculation processing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0048] The present invention will be further described below in conjunction with the accompanying drawings.

[0049] Such as figure 1 As shown, the database-based large data incremental calculation processing method of the present invention comprises the following steps,

[0050] Step (A), select the type of incremental calculation operator, the incremental calculation operator includes Append type incremental operator, Update type incremental operator, and the Append type incremental operator includes typical Append type incremental operator Quantitative operators and multi-table associated Append incremental operators can be selected according to the needs of big data;

[0051] Step (B), execute the database read operator, for all data D of the input data source 1 Read and record the database execution statement as SQL 1 ;

[0052] Step (C), calculate the operator according to the selected increment, and use SQL 1 Calculate for the input and get the calculation result R...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a database-based big data incremental calculation processing method, and the method comprises the steps of obtaining an incremental calculation result R < delta > through only calculating newly-added data D < delta > in an incremental calculation process of incremental data; combining the input data source calculation result R1 and the incremental calculation result R delta, and carrying out related calculation to obtain a final updated data source calculation result R2. Redundant calculation is avoided, calculation resources are saved, and big data analysis efficiency is improved. The incremental data is characterized in that the time sequence is strong, and the data cannot change after being generated; for instance, for motor vehicle trajectory data, personnel activity trajectory and the like, after the track is recorded through the GPS, the data does not change any more, and a unique requirement for incremental data is that it contains an incremental identification field. The invention is suitable for the fields of change, such as motor vehicle track data and personnel movement tracks, and has a good application prospect.

Description

technical field [0001] The invention relates to the technical field of big data analysis, in particular to a database-based big data incremental calculation and processing method. Background technique [0002] With the rapid development of the Internet, all walks of life have accumulated a large amount of data assets. How to analyze and judge and correlate collisions in massive data, mine valuable information, and assist the development of industries is an urgent problem to be solved. [0003] The problems of big data applications generally have the characteristics of large total amount and fast growth rate. According to statistics, Google processes 2 trillion webpage searches every year, and the number of webpages is increasing at a rate of 300 to 500 per minute; the number of users of Facebook It is already close to 2.5 billion, and maintains a growth rate of about 40 million per quarter. In the face of such a huge amount of data, if the full data set is calculated every ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/23
CPCG06F16/23
Inventor 杜志强呙维
Owner NANJING BEIDOU INNOVATION & APPL TECH RES INST CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com