Method for organizing data fragments in distributed database and distributed database system

Abstract

The invention discloses a method for organizing data fragmentation in a distributed database and a distributed database system, and belongs to the technical field of data fragmentation organization, the method for organizing a Key based on a Range fragmentation scene by grouping and arranging time series data according to characteristics of the time series data and supporting a Range fragmentation mode comprises the steps that the organization format of the Key is a HashPoint / TimeStamp format, the specific calculation mode of the HashPoint is as follows: Hash% N, Hash represents a unique identifier of the equipment, and each acquisition equipment is different; n represents a specific remainder; the TimeStamp represents a time series data acquisition timestamp, and specifically, the time series data acquisition timestamp is a unix timestamp. According to the method, the hot spot problem of data can be avoided, it is guaranteed that the number of the active Range fragments is large enough, the situation that the number of data contained in the active Range fragments is too large is avoided, and the number of the non-active Range fragments is reduced as much as possible.

Description

Technical Field

[0001] The present invention relates to the technical field of data shard organization, and in particular to a method for organizing data shards in a distributed database and a distributed database system. Background Art

[0002] Currently, for distributed databases, there are data hotspot problems in distributed time series scenarios, and it is impossible to effectively distinguish whether the Range shards are active, so that data balance cannot be guaranteed, resulting in low data processing efficiency and unsatisfactory use results. Summary of the invention

[0003] The technical task of the present invention is to address the above shortcomings and provide a method for organizing data shards in a distributed database and a distributed database system, which can avoid data hotspot problems, ensure that the number of active Range shards is sufficient, and avoid the excessive amount of data contained in the active Range shards, thereby reducing the number of inactive Range shards as much as possible.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] The method of organizing data sharding in a distributed database is to group and arrange time series data according to its characteristics. Range sharding is supported. The organization methods of Key in the Range sharding scenario include:

[0006] The Key organization format is HashPoint / TimeStamp format.

[0007] The specific calculation method of HashPoint is: Hash%N, where Hash represents the unique identification of the device, which is usually the unique number of the timing acquisition device, and each acquisition device is different; N represents the specific remainder number;

[0008] TimeStamp represents the time stamp of time series data collection, specifically the Unix timestamp.

[0009] The hash algorithm in this scheme is not fixed and can be expanded.

[0010] According to the characteristics of time series data, the time series data is grouped and arranged. The specific definition of RangeKey is: / Hash (device unique identifier)%N / TimeStamp. In the distributed scenario, the query and write of data needs to obtain the corresponding node ID through the corresponding Range sharding information and find the corresponding data distribution information.

[0011] This method actually adopts the Range sharding strategy in the distributed database, where Range represents a continuous interval, and each Range is the smallest unit of distributed operation. However, a specific strategy is applied to RangeKey, that is, key unique information such as device name or ID is hashed, and a certain number is modulo to obtain a HashPoint. Since the HashPoint is the modulo after the unique identifier of the device is hashed, a fixed HashPoint value represents multiple device ranges, and because of the Hash calculation, the actual number of devices in the same HashPoint can be avoided as much as possible. There is a skew or hot spot.

[0012] Furthermore, the initial pre-distribution rule of Range in the Key organization case is: according to the number of nodes, each node is evenly split into N Range shards, and only the HashPoint value is split during the initial split.

[0013] In the initial pre-distribution stage, the modulus is taken according to a specific number, and the initial pre-sharding number is determined, and the Range is evenly distributed to each node. In this scheme, the initial pre-divided Rang range is only divided to the HashPoint level, not to a specific timestamp. For example, Range1 on node 1 is [0, 5), and the specific device data range included is all data of Hash (device unique identifier) ​​% 50 = 0, 1, 2, 3, 4.

[0014] Furthermore, the Split rule based on the organizational format of the Key is: in each Split, the vertical splitting of the HashPoint is prioritized, and then the horizontal splitting of the time dimension is performed.

[0015] When the amount of data included in each Range reaches the limit and needs to be split, the vertical split of HashPoint is prioritized. For example, if Range1 is [0, 5], when it reaches 5G, Range Split is triggered and vertical split is prioritized, splitting into Range1[0, X], RangeN[X, 5], where X is a specific number between 0 and 5. When splitting, priority is given to dividing the Range as evenly as possible to ensure that the sizes of the two Ranges after the split are as equal as possible.

[0016] Furthermore, after Range Split reaches HashPoint without any gap, it starts to split according to the time interval.

[0017] In the time series scenario, as time series data is continuously written, each write may cause the data volume corresponding to the Range to reach its limit, triggering the corresponding Range split (Split). After the Range Split to HashPoint has no gap, it starts to split according to the time interval. For example, when Range1 Split to the corresponding range [0, 1), it will continue to split according to the timestamp.

[0018] Furthermore, based on the Range sharding method, active Range shards and inactive Range shards are distinguished and cold and hot data processing is performed.

[0019] Since time series data has the characteristic of reporting data over time, all ranges will be continuously split, and the corresponding range write hotspots are concentrated on the last active shard range of the device. For example, for multiple range shard ranges with HashPoint of 0, the corresponding active range shard is usually RangeN"[0 / 1623549277,1), because this shard represents all intervals with HashPoint of 0 and corresponding data reporting Unix timestamp greater than 1623549277. The timestamps written by time series data are constantly increasing, so the active and inactive range shards can be effectively distinguished according to the sharding method and hot and cold data processing can be performed.

[0020] Furthermore, the Merge rule based on the organization format of the Key is: Merge the inactive Range according to the split time, and reduce the total data volume of the actual Range in the system.

[0021] Furthermore, according to the configured Merge time, periodic patrol checks whether all Ranges meet the Merge conditions. The conditions include:

[0022] 1) The range of the range split has been split into the corresponding timestamp range; the range that has not been split into a specific timestamp is not considered for merging to avoid the impact on the active range split as much as possible;

[0023] 2) Range has been split into specified time;

[0024] For example, if RangeN'[0 / 1655085277,0 / 1686621277) is split on June 3, then RangeN' will be merged into Range1 on June 13, and the range corresponding to Range 1 will be changed to [0, 0 / 1686621277);

[0025] Since the time range corresponding to this Range is the maximum time range, the Range of the last time range will not be merged;

[0026] Once a Range is merged, it will no longer be split according to the RangeSize.

[0027] The present invention also claims to protect a distributed database system, which groups and arranges time series data according to the characteristics of the time series data, supports Range sharding, and the organization format of Key in the Range sharding scenario is HashPoint / TimeStamp format.

[0028] The specific calculation method of HashPoint is: Hash%N, where Hash represents the unique identification of the device, which is usually the unique number of the timing acquisition device, and each acquisition device is different; N represents the specific remainder number;

[0029] TimeStamp represents the time stamp of time series data collection, specifically the Unix timestamp;

[0030] The system implements data sharding by using the above-mentioned method for organizing data sharding in a distributed database.

[0031] The present invention also claims a device for implementing the organization of data shards in a distributed database, comprising: at least one memory and at least one processor;

[0032] The at least one memory is used to store a machine-readable program;

[0033] The at least one processor is used to call the machine-readable program to implement the above method.

[0034] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which when executed by a processor can implement the above method.

[0035] Compared with the prior art, the method for organizing data fragmentation in a distributed database and the distributed database system of the present invention have the following beneficial effects:

[0036] 1. Data is randomly distributed to different ranges because it is hashed according to the device. Random distribution reduces the probability of data hot spots or skew.

[0037] 2. Initially, the number of Range shards is fixed according to the configured number, reducing the number of initial shards.

[0038] 3. Since the Key includes the dual dimensions of device and time, and Split is performed based on HashPoint first, in the time series scenario, since all device writes are written with new timestamps, the number of active shards is kept fixed as much as possible, and the corresponding data size of the Range is also predictable, which improves concurrency and controllability.

[0039] 4. Due to the support for the Merge of historical shards and the characteristics of time series data, the number of shards in the active range is greatly reduced without affecting user business as much as possible, and the active and inactive range shards are effectively distinguished. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a diagram showing initial pre-distribution of Range under Key organization provided by an embodiment of the present invention;

[0041] Figure 2 It is a diagram of a Range fragmentation strategy provided by an embodiment of the present invention;

[0042] Figure 3 It is a diagram of a Merge rule provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with specific embodiments.

[0044] The embodiment of the present invention provides a method for organizing data sharding in a distributed database, which groups and arranges time series data according to the characteristics of the time series data and supports the Range sharding mode. The Key organization method based on the Range sharding scenario includes:

[0045] The Key organization format is HashPoint / TimeStamp format.

[0046] The specific calculation method of HashPoint is: Hash%N, where Hash represents the unique identification of the device, which is usually the unique number of the timing acquisition device, and each acquisition device is different; N represents the specific remainder number;

[0047] TimeStamp represents the time stamp of time series data collection, specifically the Unix timestamp.

[0048] The hash algorithm in this scheme is not fixed and can be expanded.

[0049] According to the characteristics of time series data, the time series data is grouped and arranged. The specific definition of RangeKey is: / Hash (device unique identifier)%N / TimeStamp. In the distributed scenario, the query and write of data needs to obtain the corresponding node ID through the corresponding Range sharding information and find the corresponding data distribution information.

[0050] This method actually adopts the Range sharding strategy in the distributed database, where Range represents a continuous interval, and each Range is the smallest unit of distributed operation. However, a specific strategy is applied to RangeKey, that is, key unique information such as device name or ID is hashed, and a certain number is modulo to obtain a HashPoint. Since the HashPoint is the modulo after the unique identifier of the device is hashed, a fixed HashPoint value represents multiple device ranges, and because of the Hash calculation, the actual number of devices in the same HashPoint can be avoided as much as possible. There is a skew or hot spot.

[0051] The initial pre-distribution rule of Range in Key organization is: according to the number of nodes, each node is evenly split into N Range shards, and only the HashPoint value is split during the initial split.

[0052] like Figure 1 The figure shows the idea of ​​the initial pre-distribution stage of this method. In the initial pre-distribution stage, the modulus is taken according to a specific number, the number of initial pre-shards is determined, and the Range is evenly distributed to each node. In this scheme, the initial pre-divided Rang range is only divided into the HashPoint level, not into a specific timestamp, for example Figure 1 The Range1 on node 1 in is [0, 5), and the device data range specifically included is all data of Hash (device unique identifier) ​​% 50 = 0, 1, 2, 3, 4.

[0053] The Split rule based on the organizational format of the Key is: in each Split, the vertical splitting of the HashPoint is prioritized, and then the horizontal splitting of the time dimension is performed.

[0054] Figure 2 The sharding strategy corresponding to this method is shown. When the amount of data included in each Range reaches the limit and needs to be split, the vertical splitting of HashPoint should be prioritized, that is, Figure 2In the example, Range1 is [0, 5]. When the size reaches 5 GB, Range Split is triggered, and vertical splitting is prioritized, splitting into Range1[0, X], RangeN[X, 5], where X is a specific number between 0 and 5. When splitting, it is also prioritized to divide the range as evenly as possible to ensure that the sizes of the two ranges after the split are as equal as possible.

[0055] In the time series scenario, as time series data is continuously written, each write may cause the data volume corresponding to the Range to reach its limit, triggering the corresponding Range split. After the Range Split reaches HashPoint without gaps, it starts to split according to the time interval. For example, when Range1 Split reaches the corresponding range [0, 1), it will continue to split according to the timestamp. For example, Figure 2 The possible Splits are shown in: Range1[0,0 / 1623549277), RangeN"[0 / 1623549277,1). In the above example, the data range included in Range1 is Hash (device unique identifier)%50=0 and the corresponding data reports the corresponding interval of Unix timestamp less than 1623549277.

[0056] Since time series data has the characteristic of reporting data over time, all ranges will be continuously split, and the corresponding range write hotspots are concentrated on the last active shard range of the device. For example, for multiple range shard ranges with HashPoint of 0, the corresponding active range shard is usually RangeN"[0 / 1623549277,1), because this shard represents all intervals with HashPoint of 0 and corresponding data reporting Unix timestamp greater than 1623549277. The timestamps written by time series data are constantly increasing, so the active and inactive range shards can be effectively distinguished according to the sharding method and hot and cold data processing can be performed.

[0057] The Merge rule based on the organization format of the Key is: Merge the inactive Range according to the split time, and reduce the total data volume of the actual Range in the system.

[0058] As time goes by, due to the fact that time series data has the characteristics of massive reporting, but focuses on the more active recent data, this method shows the characteristics of Range Merge based on historical split time. Figure 3 shows the corresponding Merge rule, that is, according to the configured merge time, periodically check whether all Ranges meet the Merger conditions. Condition 1 is that the split range of the Range has been split into the corresponding timestamp range, that is, Figure 3 In the example, Range1[0,0 / 1623549277), RangeN'[0 / 1655085277,0 / 1686621277), RangeN”[0 / 1686621277,0 / 1718243677), and RangeNXX'[1,2) that are not split to specific timestamps are not considered for merging to avoid the impact on active Range Split as much as possible; Condition 2 is that the Range has been split for a certain period of time. For example, if RangeN'[0 / 1655085277,0 / 1686621277) is split on June 3, then RangeN' will be merged into Range1 on June 13, and Range 1 is changed to [0, 0 / 1686621277). In addition, there is an invisible condition in this condition, that is, the last time range will not be merged, because the time range corresponding to this range is the maximum time range, specifically RangeN"'[0 / 1718243677,1) in the example. Once the range is merged, it will no longer be split according to RangeSize.

[0059] Time series data is a collection of data recorded in chronological order. It is data collected, recorded or observed at certain time intervals or timestamps over a period of time. Time series data usually consists of two main parts: timestamp and corresponding observation. Timestamp represents the time point or time period of the data point, which can be in the form of date, time, timestamp, etc.; observation is the value, indicator or event measured or recorded at a given time point or time period. Time series data has a wide range of applications in many fields, such as financial market analysis, weather forecasting, traffic flow monitoring, production process monitoring, etc. The characteristics of time series data include: time-centric, data records always have timestamps. Only append: the data written is almost new data, and INSERT operations are always performed. Data correction operations such as UPDATE and DELETE are abnormal state operations, while append is the norm. Time is adjacent, data is usually written in chronological order, new data is usually related to the most recent time interval, and missing data about old time intervals is rarely updated or backfilled.

[0060] Distributed system: A distributed system is a system that uses multiple independent computers to solve storage and computing problems that cannot be handled by a single node (computer). This is a typical divide-and-conquer idea. Each node is only responsible for a subset of the original problem (i.e., the tasks that the entire system needs to complete). In a distributed storage system, the splitting of tasks is called data sharding.

[0061] Distributed data sharding: (segment, fragment, shard, partition) means dividing a data set into independent and orthogonal data subsets according to certain rules, and then distributing the data subsets to different nodes. Data sharding needs to follow certain rules. Different distributed applications have different rules. Common data sharding methods include hashing, list partitioning, and data range based. The granularity of the shard, that is, the size of the data subset, is also the basic unit of data migration. If the granularity is too coarse, it is not conducive to data balance; if the granularity is too fine, the management and migration costs will be relatively high.

[0062] Selection of sharding feature value: Data sharding is based on key value and feature value. This feature value has different names in different systems, such as sharding key in MongoDB and Partition Key in Oracle. In any case, the selection of this feature value is very important.

[0063] This method makes full use of the characteristics of time series data being written incrementally according to timestamps, constructs reasonable data partitioning rules in distributed scenarios, and ensures that the number of shards in the distributed system is reasonable and controllable:

[0064] 1. The splitting strategy based on data pre-distribution and the solution of hashing the device and taking the remainder within the key avoid the data hotspot problem;

[0065] 2. By supporting the specific Split rules of Range, it ensures that the number of active Range shards is sufficient and avoids the shards of active Range containing too much data;

[0066] 3. Through the merge rule of the historical range, the number of shards of the inactive range is reduced as much as possible.

[0067] The embodiment of the present invention further provides a distributed database system, which groups and arranges time series data according to the characteristics of the time series data, supports Range sharding, and the organization format of Key in the Range sharding scenario is HashPoint / TimeStamp format.

[0068] The specific calculation method of HashPoint is: Hash%N, where Hash represents the unique identification of the device, which is usually the unique number of the timing acquisition device, and each acquisition device is different; N represents the specific remainder number;

[0069] TimeStamp represents the time stamp of time series data collection, specifically the Unix timestamp;

[0070] The hash algorithm in this scheme is not fixed and can be expanded.

[0071] Among them, the initial pre-distribution rule of Range in the Key organization case is: according to the number of nodes, each node is evenly split into N Range shards, and only the HashPoint value is split during the initial split.

[0072] The Split rule based on the organizational format of the Key is: in each Split, the vertical splitting of the HashPoint is prioritized, and then the horizontal splitting of the time dimension is performed.

[0073] The Merge rule based on the organization format of the Key is: Merge the inactive Range according to the split time, and reduce the total data volume of the actual Range in the system.

[0074] The system specifically implements organizing data shards through the method for organizing data shards in a distributed database described in the above embodiment.

[0075] The system supports Range sharding and organizes RangeKey ranges according to Hash (device unique identifier)%N / TimeStamp, and supports Split and Merge of Range under specific rules according to RangeSize, device ID and data timestamp. Using this implementation, hot spot issues in distributed time series scenarios can be effectively solved, whether Range sharding is active or not can be distinguished, and data balance can be ensured.

[0076] An embodiment of the present invention also provides a device for implementing data sharding in a distributed database, comprising: at least one memory and at least one processor;

[0077] The at least one memory is used to store a machine-readable program;

[0078] The at least one processor is used to call the machine-readable program to implement the method for organizing data shards in a distributed database as described in the above embodiment.

[0079] The embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the method for organizing data shards in a distributed database described in the above embodiment is implemented. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0080] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0081] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0082] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0083] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0084] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A method for organizing data shards in a distributed database, characterized in that: According to the characteristics of time series data, time series data is grouped and arranged. Range sharding is supported. The organization methods of Key in the Range sharding scenario include: The Key organization format is HashPoint / TimeStamp format. The specific calculation method of HashPoint is: Hash%N, Hash represents the unique identifier of the device, which is different for each acquisition device; N represents the specific remainder number; TimeStamp represents the time stamp of time series data collection, specifically the Unix timestamp.

2. The method for organizing data fragmentation in a distributed database according to claim 1, characterized in that: The initial pre-distribution rule of Range in Key organization is: according to the number of nodes, each node is evenly split into N Range shards, and only the HashPoint value is split during the initial split.

3. The method for organizing data fragmentation in a distributed database according to claim 1 or 2, characterized in that: The Split rule based on the organizational format of the Key is: in each Split, the vertical splitting of the HashPoint is prioritized, and then the horizontal splitting of the time dimension is performed.

4. The method for organizing data fragmentation in a distributed database according to claim 3, characterized in that: After there is no gap from RangeSplit to HashPoint, start splitting according to time intervals.

5. The method for organizing data fragmentation in a distributed database according to claim 4, characterized in that: Based on the Range sharding method, active Range shards and inactive Range shards are distinguished and cold and hot data processing is performed.

6. The method for organizing data fragmentation in a distributed database according to claim 5, characterized in that: The Merge rule based on the organization format of the Key is: Merge the inactive Range according to the split time, and reduce the total data volume of the actual Range in the system.

7. The method for organizing data fragmentation in a distributed database according to claim 6, characterized in that: According to the configured Merge time, periodic patrol checks whether all ranges meet the Merge conditions. The conditions include: 1) The range of Range has been split into the corresponding timestamp range; 2) Range has been split into specified time; Since the time range corresponding to the Range is the maximum time range, the Range of the last time range will not be merged; Once a Range is merged, it will no longer be split according to the RangeSize.

8. A distributed database system, characterized in that: This distributed database system groups and arranges time series data according to its characteristics and supports Range sharding. The Key organization format in the Range sharding scenario is HashPoint / TimeStamp format. The specific calculation method of HashPoint is: Hash%N, where Hash represents the unique identifier of the device, which is different for each acquisition device; N represents the specific remainder number; TimeStamp represents the time stamp of time series data collection, specifically the Unix timestamp; The system organizes data shards through the method for organizing data shards in a distributed database as described in any one of claims 1 to 7.

9. A device for organizing data fragmentation in a distributed database, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 7.

10. A computer readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, can implement the method according to any one of claims 1 to 7.

Images