Data deduplication for NoSQL databases
By blocking and sorting data in the NoSQL database, and using the key mapping mechanism to achieve data deduplication, the problem of low data deduplication efficiency in the NoSQL database is solved, and storage efficiency and backup performance are improved.
Patent Information
- Application Number
- CN202280094963.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-23
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to achieve efficient data deduplication in NoSQL databases, especially during backup and recovery, resulting in excessive replica size, low storage efficiency, and fixed-size data deduplication cannot capture similar but different data blocks. .
By blocking and sorting data in multiple nodes, the key mapping mechanism is used to continuously store data of the same key, realizing data deduplication, and integrating the data deduplication process during the backup process to optimize storage efficiency.
It improves the efficiency and storage density of data deduplication, reduces backup size, and enhances data recovery performance. It is suitable for multi-node storage environments of NoSQL databases.
Smart Images

Figure CN119072685A_ABST
Abstract
Description
Background Art
[0001] In some embodiments of the present invention, the present invention relates to a method and system for deduplication of data in a database, and more specifically but not limited to, to a method and system for deduplication of data in a NoSQL database. SQL or Structured Query Language is a standardized programming language used to manage relational databases and perform various operations on the data therein.
[0002] Data deduplication is a process that is typically used to eliminate or reduce excessive copies of data and reduce storage capacity requirements. Data deduplication can be run as a process when data is written to the storage system and / or as a background process after the data is written to disk to eliminate duplicates. Data deduplication and compression generally require trust because the storage performing the data deduplication must be able to access the data in order to find multiple identical data and be able to compress it.
[0003] Some block storage systems use fixed-size deduplication where storage is divided into 8kb aligned blocks. A signature is created for each block. If a block has the same signature, the data is not stored again, but a pointer to the stored block is kept. One disadvantage of fixed-size deduplication is that fixed-size deduplication does not capture similar blocks, only blocks that are identical to each other. For example, if the blocks are identical but shifted by 512 bytes, fixed-size deduplication will not capture one of the identical blocks. The same is true for byte shifts and insertion switches, fixed-size deduplication will not capture one of the identical blocks.
[0004] Typically, the locality of backup data deduplication is large, so once the backup system finds the reference point for data deduplication, it can find the rest of the data deduplication locations more quickly. Therefore, the backup system can optimize the data and perform inline variable data deduplication.
[0005] Variable data deduplication divides the data into blocks of variable size and checks whether the same data has been written to the system. If the same data has been written to the system, only a pointer to one copy of the data is kept.
[0006] NoSQL (originally meaning "non-SQL" or "non-relational") databases provide a mechanism for data storage and retrieval that is modeled differently from the tabular relationships used in relational databases. Such databases have been around since the late 1960s, but the name "NoSQL" was not coined until the early 2000s due to the needs of Web 2.0 companies. NoSQL databases are increasingly being used for big data and real-time network applications. NoSQL systems are sometimes also referred to as "not only SQL" to emphasize that NoSQL systems can support SQL-like query languages or coexist with SQL databases in polyglot persistence architectures. This approach can simplify design, allow for simpler "horizontal" scaling to clusters of machines (which is a problem with relational databases), provide finer control over availability, and limit the object-relational impedance mismatch.
[0007] NoSQL databases often maintain mirrors of data in multiple locations. In many of these NoSQL database systems, each mirror is identical. However, in modern clustered NoSQL databases such as Cassandra, data copies are not always exactly the same, which can lead to a significant decrease in the deduplication ratio.
[0008] Backups of database data can be performed directly by the database itself because the database data is stored in the database's native format, so the database has no problem locating and copying the data in the mirror on the backup storage device. The term "mirror" as used herein refers to an identical copy of virtual data, rather than a reversed copy (as might be the case with a mirror of a physical object). Alternatively, a general-purpose backup system can be used to back up the database data, which greatly increases the total backup size because general-purpose backup systems typically append new backups to the end of old backups without deleting previous backups.
[0009] Some databases offer application-aware deduplication systems, redundant copy detection, and backup and recovery mechanisms that work regardless of the database configuration.
[0010] Traditional database backup systems often append the backup to the end of the backup file, resulting in a copy that is larger than the original data.
[0011] Therefore, there needs to be integration, understanding, and consistency between the NoSQL database being backed up and the data deduplication store. Summary of the invention
[0012] A database configuration file provided to a database backup and data deduplication system may provide: a line format for each row of data in a database; a function used by the database to map each row of data in a database to a first storage node for storing database data and a second storage node for storing database data; and the number of copies of the row data to be backed up.
[0013] The database is typically stored in at least three different nodes, and each row of data is typically mirrored in at least two nodes. Each row of data in the multiple rows of data in the database is associated with a key, and the key is at least a function of a mapping to a first storage node and a second storage node. A block partitioning process performed by a processor divides each of the at least three nodes into blocks. The processor performs a sorting process on each block, wherein the sorting process sorts the row data in each block according to its associated key, so that each row of data in the multiple rows of data associated with the same key is stored continuously in each block, and the sorting process is used to create a mapping indicating the location of each row of data in the block before performing the sorting process on the row data.
[0014] According to one aspect of some embodiments of the present invention, there is provided a device, the device comprising: a processor; a database, wherein the database comprises data stored in at least three nodes, each node comprising at least one data storage device; multiple rows of data, wherein each row of data is stored in a first node among the at least three nodes, each row of data is mirrored in at least a second node among the at least three nodes, each row of data in the multiple rows of data is associated with a key, and the key is at least a function of a mapping to the first node and the second node; a blocking process, wherein the blocking process is executed by the processor, the blocking process divides each of the at least three nodes into blocks; a sorting process, wherein the sorting process is executed by the processor on each block, the sorting process sorts the row data in each block according to its associated key, so that each row of data in the multiple rows of data associated with the same key is stored continuously in each block, and the sorting process is used to create a mapping indicating the position of each row of data in the block before executing the sorting process on the row data.
[0015] According to an aspect of some embodiments of the present invention, a method is provided, the method comprising: storing multiple rows of data in a database in at least three nodes, wherein each node includes at least one data storage device; mirroring each row of data in the multiple rows of data in at least a first node among the at least three nodes and at least a second node among the at least three nodes, wherein each row of data in the multiple rows of data is associated with a key, and the key is at least a function of a mapping to the first node and the second node; dividing each of the at least three nodes into blocks; sorting the row data in each block according to its associated key, so that each row of data in the multiple rows of data associated with the same key is stored continuously in each block, wherein the sorting process is used to create a mapping indicating the position of each row of data in the block before executing the sorting process on the row data.
[0016] According to an aspect of some embodiments of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores data representing computer-executable software, wherein the software includes instructions for: storing multiple rows of data in a database in at least three nodes, wherein each node includes at least one data storage device; mirroring each row of data in the multiple rows of data in at least a first node among the at least three nodes and at least a second node among the at least three nodes, wherein each row of data in the multiple rows of data is associated with a key, and the key is at least a function of a mapping to the first node and the second node; dividing each of the at least three nodes into blocks; sorting the row data in each block according to its associated key, so that each row of data in the multiple rows of data associated with the same key is stored continuously in each block, wherein the sorting process is used to create a mapping indicating the position of each row of data in the block before executing the sorting process on the row data.
[0017] According to some embodiments of the present invention, a configuration file is also included, wherein the configuration file includes at least: a line format of each row of data in the database; a function of the database used to map each row of data in the database to the first node and the second node; and the number of copies of the row of data.
[0018] According to some embodiments of the present invention, the database comprises a NoSQL database.
[0019] According to some embodiments of the invention, data stored in each of the plurality of nodes is different from data in any other of the plurality of nodes.
[0020] According to some embodiments of the present invention, after the chunking process sorts the data in a chunk, the processor performs a data deduplication process on the data in the chunk.
[0021] According to some embodiments of the present invention, the size of each block is at least 4 megabytes (MB).
[0022] According to some embodiments of the present invention, a storage configuration file is also included, which includes at least: the line format of each row of data in the database; the function of the database used to map each row of data in the database to the first node and the second node; and the number of copies of the row of data.
[0023] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as those generally understood by those of ordinary skill in the art to which the invention belongs. Although methods and materials similar or equivalent to the methods and materials described herein can be used for the implementation or testing of embodiments of the invention, some exemplary methods and / or materials are described below. In the event of a conflict, this patent specification (including definitions) shall prevail. In addition, these materials, methods and examples are illustrative only and are not necessarily restrictive.
[0024] The implementation of the method and / or system provided by the embodiment of the present invention may involve performing or completing the selected task manually, automatically, or in combination of these two methods. In addition, according to the actual instruments and equipment of the embodiments of the method and / or system of the present invention, the operating system may be used to implement multiple selected tasks through hardware, software, firmware, or a combination of the three.
[0025] For example, the hardware for performing the selected tasks according to the embodiments of the present invention can be implemented as a chip or circuit. For software, the selected tasks according to the embodiments of the present invention can be implemented as a plurality of software instructions executed by a computer using any suitable operating system. In an exemplary embodiment of the present invention, one or more tasks according to the exemplary embodiments of the method and / or system described herein are performed by a data processor, for example, by a computing platform for executing a plurality of instructions. Optionally, the data processor includes a volatile memory and / or non-volatile memory for storing instructions and / or data, for example, a hard disk and / or removable media for storing instructions and / or data. Optionally, a network connection is also provided. Optionally, a display and / or a user input device such as a keyboard or a mouse is also provided. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Some embodiments of the present invention are described herein by way of example only in conjunction with the accompanying drawings. Now with specific reference to the accompanying drawings, it should be emphasized that the details shown are shown by way of example and for illustrative purposes only. Thus, it will be apparent to those skilled in the art how to implement the embodiments of the present invention based on the accompanying drawings.
[0027] In the attached picture:
[0028] Figure 1 It is a description of a system used to deduplicate database data;
[0029] Figure 2 yes Figure 1 Examples of applications of a system for deduplication of database data;
[0030] Figure 3 yes Figure 1 and Figure 2 A simplified flow chart of the method of operation of the system and device described in the embodiment of the present invention. DETAILED DESCRIPTION
[0031] In some embodiments of the present invention, the present invention relates to a system and method for deduplication of database data, and more specifically but not limited to, to a system and method for deduplication of database data in a NoSQL database.
[0032] In a system for deduplicating database data, it is generally believed that there is an understanding of the database system that performs the data deduplication operation. For example, there may be a configuration file in the system for deduplicating data, and the configuration file includes information about the format of the stored database data. The configuration file may also store information about how many copies of each data element in the database data are stored in the database. On the other hand, operations on data may be performed in storage, such as moving a block of data from one storage location to another storage location. Therefore, data deduplication can be achieved without integrating the data deduplication process into the database application.
[0033] If the data deduplication process is integrated into the database, there will be performance benefits (e.g., the data deduplication process may be more efficient than an external data deduplication process). However, such an integrated database data deduplication process may need to be specific to the particular database with which it is integrated, and generally cannot be used as a data deduplication process for data stored in a format optimized for a second database. In order to integrate the second data deduplication process into the second database, it is expected that a significant amount of additional programming time (and associated programming costs) will be required.
[0034] Database data (typically database row data) may be stored in multiple storage nodes such that each row of data is stored in one node and a mirror image of the row is stored in a second row. Each row of data may be stored with a key associated with the row of data, the key being at least a function of a mapping to at least each node when storing the data. A processor may perform a chunking process on each storage node, thereby storing the data stored in the storage node as data chunks. For example, a data chunk may be relatively large, and without limiting the generality of the foregoing, a data chunk may be 6MB, 5MB, 4MB, etc.
[0035] The processor may perform a sorting process on each block. The sorting process may sort the row data in each block according to the key associated with each row, so that each row of data in multiple rows of data associated with the same key is stored continuously in each block. The key associated with each row of data indicates on which node the row of data is stored. Since the database stores data in a known format, the creation of the key can include information in the key indicating which nodes store the row associated with the key. The sorting process may be used to create a map indicating the location of each row of data in the block before performing the sorting process on the row data.
[0036] Referring to the accompanying drawings, Figure 1 A system 100 for deduplicating database data is shown. The database 10 may include multiple rows of data 20A, 20B, 20C, ..., 20n (where n represents an integer other than 14 (corresponding to the position of the 14th letter n in the alphabet)). For example, the database may be a NoSQL database. Each row of data may be backed up and stored (e.g., "mirrored") in at least two of the multiple nodes 30A, 30B, 30C, 30D. For example, Figure 1 As shown, each row of data in rows 20A, 20B, 20C, ... 20n is depicted as having two arrows extending therefrom. One arrow is depicted with a solid line and one arrow is depicted with a dashed line. The arrow depicted with a solid line indicates that the row is stored in the node 30A, 30B, 30C, 30D at the end of the arrow. The arrow depicted with a dashed line indicates that the row is mirrored (e.g., backed up) in the node 30A, 30B, 30C, 30D at the end of the arrow. Therefore, for example, the row data identified as row 20A is backed up in node 30A and mirrored in node 30D; the row data identified as row 20B is backed up in node 30B and mirrored in node 30C; the row data identified as row 20C is backed up in node 30C and mirrored in node 30B; the row data identified as row 20n is backed up in node 30D and mirrored in node 30A. It should be understood that the data stored in each of the plurality of nodes 30A, 30B, 30C, 30D is not necessarily the same as the data in any other of the plurality of nodes 30A, 30B, 30C, 30D.
[0037] The processor 40 may execute the block process 50. The processor 40 may include a microprocessor, i.e., an integrated circuit (IC) containing the core functionality of a computer's central processing unit (CPU). The microprocessor may be a clock-driven, register-based, programmable multi-purpose silicon chip that takes binary data as input and provides output after processing it according to instructions stored in an associated memory 45.
[0038] The block process 50 is executed on the plurality of nodes 30A, 30B, 30C, 30D, as indicated by arrow 35. The block process 50 receives each of the plurality of nodes 30A, 30B, 30C, 30D as input and decomposes each of the nodes 30A, 30B, 30C, 30D into blocks 60A, ..., 60n (where n represents some integer other than 14 (corresponding to the position of the 14th letter n in the alphabet)). The output of the blocks 60A, ..., 60n is indicated by arrow 65. The size of the blocks 60A, ..., 60n is smaller than the size of the nodes in the plurality of nodes 30A, 30B, 30C, 30D. For example, each node can be decomposed into blocks of size 3MB, 4MB, 5MB, 6MB, etc.
[0039] Each block 60A, ..., 60n is sorted by a sorting process 70, as indicated by arrow 75. The sorting is performed based on a key, as described below. The sorting process 70 is executed by the processor 40. The sorting process 70 outputs a sorted block 60i, as indicated by arrow 76. The sorting process 70 sorts the row data in each block 60A, ..., 60n according to the key associated with the data.
[0040] The key associated with any given row of data is a function of the node on which the given row of data is stored. For example, if there are three nodes (node 1, node 2, node 3), and each row of data is stored on two of the three nodes, there are three possible keys: (1, 2), (1, 3), (2, 3). Each row of data can then be sorted according to its associated key so that data with the same key is stored consecutively. The mapping information for each row of data indicates the current location of the row of data in the block and the original location of the row of data. Therefore, when a block is read, the data can be sorted to restore its original storage configuration.
[0041] The processor 40 may perform a sorting process 70 on each block 60A, ..., 60n, wherein the sorting process 70 sorts the rows of data according to the associated key ( Figure 2240 in the example) sorts the row data in each block so that each row data in multiple rows of data associated with the same key is stored continuously in each block (hereinafter referred to as Figure 2 Before executing the sorting process on the row data, the sorting process 70 creates a map indicating the position of each row data in the data block, thereby maintaining the map to the original position of the row data so that the data can be located when the database needs to call the data.
[0042] The processor 40 may access a configuration file 80, which may be stored in a memory 80 or other location accessible to the processor 40, which may include a non-volatile memory. The configuration file 80 may include information that the processor may use in its operation, including but not limited to: the line format of each row of data in the database; the function used by the database to map each row of data in the database to a first node and a second node; the number of copies of the row of data. Block 60i (where i represents an integer other than 9 (corresponding to the position of the 9th letter i in the alphabet), and i is not necessarily equal to n) now includes the sorted data 80.
[0043] Reference now Figure 2 , Figure 2 yes Figure 1 An application example of a system for deduplication of database data. It should be understood that Figure 2 Describes Figure 1 The present invention is an operational example of an application of a system for deduplication of database data and is not intended to be limiting in any method or manner. Figure 2 Depicts the data in three nodes (node 0, node 1, and node 2). Node 0, node 1, and node 2 can be connected to Figure 1 The nodes in multiple nodes 30A, 30B, 30C, 30D are the same or similar. The data in node 0 is shown to have two types, where the "type" of data indicates which nodes the row is stored on. For example: for illustrative purposes, the first type 210 depicted as a dashed rectangle can be stored on node 0 and node 1; for illustrative purposes, the second type 220 depicted as a solid gray rectangle can be stored on node 0 and node 2. The data in node 1 is shown to have two types: the first type 210; the third type 230, for illustrative purposes, the third type depicted as a rectangle with vertical bars can be stored on node 1 and node 2. The data in node 2 is shown to have two types: the second type 220; the third type 230.
[0044] Reference above Figure 1The described sorting process 70 operates on the data in each of the three nodes (node 0, node 1, and node 2). After the sorting process 70 completes its operation, the data in node 0 is now arranged so that the data of the first type 210 is continuous with the other data of the first type 210. The data of the second type 220 is now continuous with the other data of the second type 220. After the sorting process 70 completes its operation, the data in node 1 is now arranged so that the data of the first type 210 is continuous with the other data of the first type 210. The data of the third type 230 is now continuous with the other data of the third type 230. After the sorting process 70 completes its operation, the data in node 2 is now arranged so that the data of the second type 220 is continuous with the other data of the second type 220. The data of the third type 230 is now continuous with the other data of the third type 230.
[0045] The data in node 0 is now prefixed with mapping information 240A. Mapping information 240A maps each data element in the node so that any given data element (e.g., third data element 210*) can be mapped from its current position in the node (i.e., third data element 210*) to its position before the operation of the sorting process 70 on the data in the node (i.e., fifth data element in the node). Each of the three nodes (node 0, node 1, node 3) is now prefixed with the node appropriate mapping information 240A, 240B, 240C. After the sorting process 70 sorts the data in a block 60A, ..., 60n, the processor 40 can perform a data deduplication process on the data 210, 220, 230 in the block.
[0046] Reference now Figure 3 , Figure 3 yes Figure 1 and Figure 2A simplified flow chart 300 of the method of operation of the system and apparatus described in . In step 310, multiple rows of data can be stored in at least three nodes in a database. Each of the at least three nodes may include at least one data storage device. In step 320, each row of data in the multiple rows of data can be mirrored in at least a first node of the at least three nodes and in at least a second node of the at least three nodes. Each row of data in the multiple rows of data is associated with a key (e.g., key 240A, 240B, 240C), and key 240A, 240B, 240C is a function of a mapping to the node storing the data. In step 330, each of the at least three nodes is divided into blocks by a block process 50. At step 340, each row of data in each block is sorted by the sorting process 70 according to its associated key (e.g., key 240A, 240B, 240C) so that each row of data 20A, 20B, 20C, ..., 20n in the multiple rows of data associated with the same key 240A, 240B, 240C is stored consecutively in each block. Before the sorting process is performed on the row of data, a map is created by the sorting process 70 so that the map indicates the location of each row of data in the block. Any blocks with the same key will be sorted in their original order.
[0047] It is expected that many related database data deduplication methods and systems and NoSQL databases will be developed during the patent validity period of this application, and the scope of the terms database data deduplication and NoSQL database is intended to a priori include all such new technologies.
[0048] The terms "including", "having" and variations thereof mean "including but not limited to".
[0049] The term "consisting of" means "including and limited to."
[0050] The term "consisting essentially of" means that the composition, method or structure may include additional ingredients, steps and / or parts, but only if the additional ingredients, steps and / or parts do not materially change the basic and novel characteristics of the claimed composition, method or structure.
[0051] Unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" used herein include plural references. For example, the term "a data element" or "at least one data element" may include a plurality of data elements.
[0052] In this application, various embodiments of the present invention may be presented in range format. It should be understood that the description of the range format is only for convenience and brevity and should not be interpreted as a fixed limitation on the scope of the present invention. Therefore, the description of the range should be considered to have specifically disclosed all possible sub-ranges and individual numerical values within the range. For example, the description of a range such as from 1 to 6 should be considered to have specifically disclosed sub-ranges such as 1 to 3, 1 to 4, 1 to 5, 2 to 4, 2 to 6, 3 to 6, and individual numbers within the range, such as 1, 2, 3, 4, 5, and 6. This applies regardless of how wide the range is.
[0053] When a numerical range is indicated herein, it is meant to include any listed numerals (fractional or integer) within the indicated range. The phrases "range between a first indicated number and a second indicated number" and "range from a first indicated number to a second indicated number" are used interchangeably herein to include the first indicated number and the second indicated number and all fractions and integers between the two.
[0054] It should be understood that, for clarity, certain features of the invention described in the context of different embodiments may also be provided in combination in a single embodiment. Conversely, for brevity, various features of the invention described in the context of a single embodiment may also be provided individually, in any suitable sub-combination, or in a form suitable for any other described embodiment of the invention. Certain features described in the context of various embodiments are not considered essential features of those embodiments, unless the embodiment is not operable without those elements.
[0055] Although the present invention has been described in conjunction with the specific embodiments of the present invention, it is obvious that many substitutions, modifications and changes will be apparent to those skilled in the art. Therefore, it is intended to cover all such substitutions, modifications and changes that fall within the spirit and broad scope of the appended claims.
[0056] The purpose of the applicant is that all publications, patents and patent applications mentioned in this specification are incorporated in this specification by reference in full, just as each individual publication, patent or patent application is specifically and individually pointed out when mentioning to be incorporated by reference in this article. In addition, the citation or identification of any reference in this application should not be construed as admitting that the reference is available as the prior art of the present invention. With regard to the use of section headings, section headings should not be understood as necessary limitations. In addition, any one or more priority documents of the application are incorporated in this application by reference in full.
Claims
1. A device, wherein: The device comprises: processor; A database, wherein the database includes data stored in at least three nodes, each node including at least one data storage device; a plurality of rows of data, wherein each row of data is stored in a first node of the at least three nodes, each row of data is mirrored in at least a second node of the at least three nodes, each row of data in the plurality of rows of data is associated with a key, the key being a function of at least a mapping to the first node and the second node; A block process, wherein the block process is executed by the processor, and the block process divides each of the at least three nodes into blocks; A sorting process, wherein the sorting process is executed by the processor on each block, the sorting process sorts the row data in each block according to its associated key, so that each row data of the multiple rows of data associated with the same key is stored continuously in each block, and the sorting process is used to create a map indicating the position of each row data in the block before executing the sorting process on the row data.
2. The device according to claim 1, wherein: The device further includes a configuration file, wherein the configuration file at least includes: The line format of each row of data in the database; The database is used to map each row of data in the database to a function of the first node and the second node; The number of copies of the row data.
3. The device according to any one of claims 1 to 2, wherein: The database includes a NoSQL database.
4. The device according to any one of claims 1 to 3, wherein: The data stored in each of the plurality of nodes is different from the data in any other of the plurality of nodes.
5. The device according to any one of claims 1 to 4, wherein: After the sorting process sorts the data in a block, the processor performs a data deduplication process on the data in the block.
6. The device according to any one of claims 1 to 5, wherein: Each block must be at least 4MB in size.
7. A method, wherein: The method comprises: storing a plurality of rows of data in a database in at least three nodes, wherein each node includes at least one data storage device; mirroring each row of the plurality of rows of data in at least a first node of the at least three nodes and at least a second node of the at least three nodes, wherein each row of data in the plurality of rows of data is associated with a key that is a function of at least a mapping to the first node and the second node; dividing each of the at least three nodes into chunks; The row data in each block are sorted according to their associated keys so that each row data of the multiple rows of data associated with the same key is stored consecutively in each block, wherein the sorting process is used to create a map indicating the location of each row data in the block before performing the sorting process on the row data.
8. The method according to claim 7, wherein: The method further comprises storing a configuration file, wherein the configuration file comprises at least: The line format of each row of data in the database; The database is used to map each row of data in the database to a function of the first node and the second node; The number of copies of the row data.
9. The method according to claim 7 or 8, wherein: The database includes a No SQL database.
10. The method according to any one of claims 7 to 9, wherein: The data stored in each of the plurality of nodes is different from the data in any other of the plurality of nodes.
11. The method according to any one of claims 7 to 10, wherein: After the sorting process sorts the data in a block, the processor performs a data deduplication process on the data in the block.
12. The method according to any one of claims 7 to 11, wherein: Each block must be at least 4MB in size.
13. A computer-readable storage medium, wherein: The computer-readable storage medium stores data representing computer-executable software, wherein the software includes instructions for: storing a plurality of rows of data in a database in at least three nodes, wherein each node includes at least one data storage device; mirroring each row of the plurality of rows of data in at least a first node of the at least three nodes and at least a second node of the at least three nodes, wherein each row of data in the plurality of rows of data is associated with a key that is a function of at least a mapping to the first node and the second node; dividing each of the at least three nodes into chunks; The row data in each block are sorted according to their associated keys so that each row data of the multiple rows of data associated with the same key is stored consecutively in each block, wherein the sorting process is used to create a map indicating the location of each row data in the block before performing the sorting process on the row data.
Citation Information
Patent Citations
Systems and methods for enabling deduplication of attachment files within a database
US9244936B1
Calculating differences between datasets having differing numbers of partitions
US9563697B1
