Index read-write method based on compression merging and asynchronization updating

A reading and writing method and indexing technology, applied in special data processing applications, instruments, electronic digital data processing, etc., can solve the problems of poor performance, many times of writing to disk, less data, etc., to reduce the number of disk IO and time-consuming , the effect of increasing the number of comparisons

Inactive Publication Date: 2014-12-10
EISOO SOFTWARE
View PDF5 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This method effectively solves the IO bottleneck problem of reading the disk, but it also brings a new problem. Due to the large number of index categories, the data distributed to each storage area during batch writing will be

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Index read-write method based on compression merging and asynchronization updating

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is carried out on the premise of the technical solution of the present invention, and detailed implementation and specific operation process are given, but the protection scope of the present invention is not limited to the following embodiments.

[0033] Assuming that the number of records in the index is 1 billion, the index database divides the index into 470,329 storage areas by 470,329. When the number of index records in the memory reaches 5 million, a batch write operation is initiated. Assuming that all index cache files exclude incomplete index cache files and the total number of index cache files being written reaches n, the condition is considered to be satisfied and an asynchronous update can be initiated. Operation, in this embodiment, n=3. Compression and decompression use the fast compression algorithm snappy.

...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to an index read-write method based on compression merging and asynchronization updating. The method includes the steps that indexes are used for generating index cache files according to storage region serial numbers; incomplete index cache files or writing index cache files are filtered and removed; when the total number of the index cache files meets the asynchronization updating initiation condition, data of storage regions corresponding to the same storage region serial numbers in all the index cache files are read respectively, and the data are merged and sorted; data of storage regions corresponding to the current storage region serial numbers in an index database are read and decompressed; the data are merged, sorted and compressed, the executed data are covered and updated to the storage regions corresponding to the current storage region serial numbers in the index database, and metadata are updated; whether the ends of files of the index cache files are read is judged, if yes, the process is completed, if not, the storage region serial numbers are added with one, and the process is executed continuously. Compared with the prior art, the index read-write method based on compression merging and asynchronization updating has the advantages of being efficient, asynchronous, capable of reducing the data size for reading a magnetic disk every time and reducing the index comparison frequency and the like.

Description

technical field [0001] The invention relates to a data storage technology, in particular to an index reading and writing method based on compression and asynchronous update. Background technique [0002] With the continuous development of computer systems, there are more and more effective data in computer systems. When reading some valid data, the user needs to find the data he needs in all the valid data, and as the valid data increases, the search time will become longer. If there is no effective way to quickly find the data the user needs, the increase in effective data can only increase the burden on the user. [0003] The introduction of indexing technology solves this problem well. During the data writing process, some important information will be recorded to uniquely identify the data and record the storage location and size information of the data. We call this information an index. Indexes can help us quickly find the data we need, but as the amount of data con...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/2272G06F16/2315G06F16/2453G06F16/24552
Inventor 吴植民
Owner EISOO SOFTWARE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap