Real-time massive data index construction method and system

A technology for index establishment and mass data, applied in the field of data processing, can solve the problems of increased disk load, slow decompression, large data volume, etc., and achieve the effect of reducing CPU resource consumption, reducing disk load, and reducing read and write operations

Inactive Publication Date: 2011-10-05
TELEFON AB LM ERICSSON (PUBL)
View PDF2 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In the above-mentioned scheme, for each record, it needs to be processed multiple times from the establishment to the final "monthly index data", which is not only inefficient, but also needs to read the data from a small-grained data file for each processing , and then write it into a large-grained data file, which will undoubtedly consume too much CPU usage time and increase the load on the disk
In addition, when querying, multiple files need to be read, and the amount of data read is large, resulting in slow decompression, which cannot meet the needs of fast query

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time massive data index construction method and system
  • Real-time massive data index construction method and system
  • Real-time massive data index construction method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] In order to enable those skilled in the art to better understand the solution of the present invention, the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings and implementation manners.

[0023] The method and system for establishing a real-time massive data index in the embodiment of the present invention writes the data records generated in real time into the first memory in an appended manner; regularly writes the data stored in the first memory into the historical data file in an appended manner , and establish an index corresponding to the data written in the historical data file in the second memory.

[0024] Such as figure 1 As shown, it is a flowchart of a method for establishing a real-time massive data index in an embodiment of the present invention, including:

[0025] Step 101, write the data records generated in real time into the first memory in an appended manner.

[0026] The ab...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a data processing technology and discloses a real-time massive data index construction method and system. The method comprises the following steps: writing data records generated in real time into a first memory in an appending manner; and writing the data stored in the first memory into a historical data file periodically, and constructing an index of the historical datafile in a second memory. The method and system provided by the invention can be used for rapidly constructing the index with high efficiency so as to meet the requirement of quick query.

Description

technical field [0001] The invention relates to data processing technology, in particular to a method and system for establishing a real-time massive data index. Background technique [0002] In the information age, data processing often needs to find relevant content according to specified keywords. For example, through a search engine, as long as a keyword is entered, the content related to the keyword can be queried. For another example, when querying the database, all the data of the field can be obtained by querying a certain field. The query of the traditional database is done through the index built in the database. [0003] In some application environments, the amount of data that needs to be recorded changes in real time. For this situation, it is common practice to create indexes with different granularities. For example, create "minute index data" every minute, and collect all "minute Reprocess "index data", create "hourly index data", reprocess "hourly index da...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 钱善海温文全
Owner TELEFON AB LM ERICSSON (PUBL)
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap