Data storage method and system for data sets

Abstract

Data storage method and system for data sets. The system and method store data in an intermediate format for storage, convert the intermediate data format into a production data format of two data volumes, and merge the two data volumes into one data volume.

Description

Technical Field

[0001] The present disclosure relates generally to code processing and data technology. Background Art

[0002] When single-key lookups need to be performed on massive collections of static and read-only data (trillions of records), traditional DBMSs (database management systems) are not only inadequate in response time, but also in the time required to synchronize data across servers or perform backups.

[0003] Given the simple task of a single-key read-only search over a large amount of static data, the internal mechanisms in a standard DBMS (including but not limited to concurrency control, access control, transaction processing, and logging) hinder performance. Summary of the Invention

[0004] The following simplified summary is provided to provide a basic understanding of some aspects of the various embodiments described in this disclosure. This summary is not an extensive overview of the disclosure. This summary is neither intended to identify key or important elements of the claimed invention nor to delineate the scope of the invention. The following summary serves only as a prelude to a more detailed description of the embodiments described in this disclosure, presenting some concepts used in these embodiments in a simplified form.

[0005] One objective of some embodiments described herein is to provide a system and method for eliminating the overhead incurred by standard DBMS (Database Management System) systems by a data processing method comprising the steps of storing data in an intermediate format for storage; converting the intermediate data format into a production data format of two data volumes; and merging the two data volumes into a single data volume.

[0006] Some embodiments described in the present disclosure also focus on a data system, which includes: a storage unit that stores data in an intermediate format for storage; a conversion unit that converts the intermediate data format into a production data format of two data volumes; and a merge unit that merges the two data volumes into one data volume.

[0007] The embodiments described herein may be based on storing data in an intermediate format for storage and converting the intermediate data format into a production data format for two data volumes.The embodiments described herein may also be based on merging two data volumes into one data volume.

[0008] A benefit of the embodiments described herein is that they can allow for significantly higher performance levels than using standard database management systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Exemplary and non-limiting embodiments of the present disclosure and their advantages are explained in more detail below with reference to the accompanying drawings, in which:

[0010] Figure 1 Sample data consisting of a passport number and a full name is shown.

[0011] Figure 2 Shown from Figure 1 Sample data with record keys added, these record keys are Figure 1 MD5 sum of the indicated passport.

[0012] Figure 3 Shown Figure 1 and Figure 2 The resulting data is stored in its original format.

[0013] Figure 4 The resulting data is shown grouped into production format (only banks of memory are shown, hex dump).

[0014] Figure 5 Shown for Figures 1 to 4 Flowchart of the data system for processing data. DETAILED DESCRIPTION

[0015] Embodiments described herein relate to systems and methods for eliminating the overhead incurred by standard database management systems (DBMSs). The method involves using cryptographic message digests to achieve a uniform distribution of hash values ​​(i.e., hash keys) and relaying a portion of the index search work to the file system itself. This means that a portion of the work is passed to the file system. Using a portion of the hash for the file name allows for a smaller storage volume than storing the full hash in the database. Furthermore, given that each file name requires space for the file's name, this space is essentially free. In other words, a portion of the hash can be used in the file system's file allocation table.

[0016] Preferably, the data set can be divided into data storage volumes, and a portion of the data keys can be stored in file and directory names, which delegates a portion of the index search task to the file system. Data updates are preferably performed in batches without interrupting service.

[0017] The disclosed data storage method may involve three phases: data acquisition to store data in an intermediate (raw format) format that ensures the fastest possible data recording. Data grouping to convert the acquired data into a production-ready format (production format) on which fast queries can be performed. When the intermediate format is processed and stored, the production format is implemented for the fastest possible lookup table (LUT). Data merging to merge at least two existing data volumes into a single volume.

[0018] The data storage method described in this patent application overcomes the shortcomings of prior art DBMS solutions and is used for high-performance querying of large data sets. The data sets described in this patent application are characterized by a single key and mutable data fields that can be updated without interrupting service. The data is stored in a portable format that allows code implementation in different computer environments and platforms.

[0019] In one exemplary embodiment, the purpose of the data collection mechanism is to provide the fastest possible storage of data for future processing and merging. Thus, data collection can be run on an unlimited number of machines, merged together at a later stage, and then moved to production.

[0020] The dataset consists of two fields: a data index and a data record. The data index is converted into a message digest using a selected encryption algorithm, with the goal of providing a uniform distribution of the index. Data records preferably do not have a fixed length, but in some embodiments may be limited to a predetermined maximum length.

[0021] In one exemplary embodiment, information from a data set is stored in flat files called data stores. These data stores are named after the first N (addressing) nibbles of the data index and can contain, for example, plain text information or information in binary form, with one row per data record. The row is the data index, followed by the entire data record.

[0022] Information can be collected in different situations and can be merged in its original format by simply concatenating the data stores together (e.g., by using colon-separated fields or representing everything in JSON format). This may result in duplicate information, but any duplicate information can be eliminated during data grouping.

[0023] In an exemplary embodiment, Figure 1 and Figure 2 As shown, given a simple passport number and full name data structure, the required key can be the passport number. The data index (passport number) can be converted into a message digest using the MD5 encryption algorithm.

[0024] Assume that the information will be stored in a database volume named people, which includes a data volume named passports with a two-nibble data bank address, such as Figure 3 As shown, Figure 2 The data in the Passports data volume can be recorded in raw format in two data banks. Using two nibbles will result in a maximum of 256 data banks for the Passports data volume.

[0025] In another exemplary embodiment, a data grouping process converts a data volume in a raw format into a production format. During this process, the data volume may be sorted, duplicate records may be eliminated, and the data volume may be converted into two files: an index file and a data file, with .index and .data file extensions used in this specification.

[0026] The data index can be converted from hexadecimal (ASCII) to binary and placed in the index file along with a fixed-length data pointer to the data record in the .data file. The nibble from the index present in the memory bank name can be removed from the index file. Given the smaller size of the index file, the index file can be copied to RAM to speed up seek times. The data file includes a concatenation of all data records. Each record preferably includes a header describing the record length and an optional integrity checksum.

[0027] In one exemplary embodiment, the data pointer is set to 32 bits, which enables a maximum data storage volume size of 4 Gb, the record length is a single byte, which enables a maximum data record size of 256 bytes, and the data checksum is set to a single byte.

[0028] Use from Figure 3 The original format of the data is as follows: The 0f memory bank includes two records stored in the 0f.data file. These two records are displayed in Figure 4 The data records are preceded by a two-byte header that specifies the record size and checksum. The two records are 8 bytes long, and the checksums are (hexadecimal) 97 and c6, respectively.

[0029] The storage volume index file of.index consists of the remaining 15 bytes from the record key (note that the first byte is in the file name), followed by a 32-bit (4-byte) pointer to the start of the record in the data file (bytes 0 and 10, respectively).

[0030] In another exemplary method, data merging may include joining a source data volume into a destination data volume.The merging may be performed on data volumes in their original format by simply concatenating the data storage volumes.

[0031] A binary merge of a production-format data volume can involve processing the data volumes within the data volume and merging them into a temporary data volume that is unified within the destination volume. Once the data volume is merged, it is moved to the destination data volume. This enables data updates without interrupting service.

[0032] The merging of two data banks may be performed by processing individual records on the banks and writing them sequentially to the merged bank by skipping duplicates.

[0033] Retrieval of a data record may include performing a search using a specific search algorithm such as an interpolation search or a logarithmic search in an index file of the corresponding data store, obtaining the data position based on a pointer, and obtaining the data length by subtracting the pointer from the pointer of the next record (or the data file length in the case of the last record).

[0034] Figure 5 A flow chart of a data system according to an exemplary and non-limiting embodiment of the present disclosure is shown. The data system may include: a storage unit 100 that stores data in an intermediate format for storage; and a conversion unit 102 that converts the intermediate data format into a production data format for at least two data volumes. This conversion may be performed, for example, using an encryption algorithm. The at least two data volumes may be, for example, a data index and a data record. In one exemplary embodiment, the data index may be located in binary form in an index file along with fixed-length data pointers to data records in a data record file. The data index may be converted from the intermediate format to the production format.

[0035] The data system also includes a merging unit 104 that merges at least two data volumes into a single data volume. In one exemplary embodiment, data records can be retrieved by searching a data index file, obtaining a data location based on a given pointer, and obtaining the data length by subtracting the value of the given pointer from the value of the pointer for the next data record, or, if the given pointer is the last pointer in the data volume, subtracting the given pointer value from the data volume length. Each data record volume may include a header describing the data record length and an optional integrity checksum.

[0036] The data systems (eg, units 100, 102, and 104) may be implemented, for example, by processor-based computers and microprocessor technology to use algorithms and implement methods according to the present invention.

[0037] The specific examples provided in the description given above should not be construed as limiting the scope and / or applicability of the appended claims.

Claims

1. A computer-implemented method comprising the following steps: storing data in an intermediate data format by collecting data in a data set including a data index and data records, and partitioning the data set and storing the data set in respective flat files, sorting the stored data to eliminate duplicate records, and converting the sorted stored data in the intermediate data format into a production data format comprising at least two data volumes including an index file and a data file, wherein the converting comprises storing a given data index in binary form together with a fixed-length pointer to a data record in the data file, merging the at least two data volumes into one data volume, and A data record is retrieved from the one data volume by performing a search on the index file, obtaining a data record position according to a given pointer, and obtaining a data record length by subtracting a value of the given pointer from a value of a pointer to a next sequential data record, or if the given pointer is the last pointer, obtaining the data record length by subtracting the value of the given pointer from a length of the data file.

2. The computer-implemented method of claim 1, wherein: The conversion is performed by using an encryption algorithm process.

3. The computer-implemented method of claim 1 , wherein: Each data file includes a header describing at least one of a data record length and an integrity checksum.

4. A data system, characterized in that: The data system includes: a storage unit that stores data in an intermediate data format by collecting data in a data set including a data index and data records, and dividing the data set and storing the data set in respective flat files; a conversion unit that sorts the stored data to eliminate duplicate records and converts the sorted stored data in the intermediate data format into a production data format comprising at least two data volumes including an index file and a data file, wherein the conversion comprises storing a given data index in binary form together with a fixed-length pointer to a data record in the data file, a merging unit, the merging unit merging the at least two data volumes into one data volume, and The data system is configured to retrieve data records from the one data volume by performing a search on the index file, obtaining a data record position according to a given pointer, and obtaining a data record length by subtracting a value of the given pointer from a value of a pointer to a next sequential data record, or if the given pointer is the last pointer, obtaining the data record length by subtracting the value of the given pointer from a length of the data file.

5. The data system according to claim 4, characterized in that The conversion is performed by using an encryption algorithm process.

6. The data system according to claim 4, characterized in that Each data file includes a header describing at least one of a length of the data record and an optional integrity checksum.

Images