Data storage method, device, apparatus, and storage medium

Abstract

Embodiments of the present application disclose a data storage method, device, equipment and storage medium, which can be applied to the field of computer technology. The method is applied to a distributed database and includes the following steps: obtaining at least one data set corresponding to to-be-stored data, each data set being obtained by dividing the to-be-stored data based on data attribute information, and each data set corresponding to a data storage mode; dividing each data set into at least one shard based on the data amount in each data set; and storing each shard in the distributed database based on the data storage mode corresponding to each shard. By using the embodiments of the present application, the flexibility and stability of data storage can be improved, and the applicability is high.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data storage method, apparatus, device, and storage medium. Background Technology

[0002] Existing distributed databases often use specific rules to divide the data to be stored into multiple shards, and store each shard using the same specific storage method. However, as the amount of data to be read and written continues to increase, existing methods often result in some nodes in the distributed database storing a large amount of data, leading to poor data storage flexibility. Furthermore, when some nodes in the distributed database store a large amount of data, they also generate more data processing tasks, resulting in low data storage stability.

[0003] Therefore, how to flexibly and stably store data in a distributed database has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a data storage method, apparatus, device, and storage medium, which can improve the flexibility and stability of data storage and has high applicability.

[0005] On one hand, embodiments of this application provide a data storage method, the method comprising:

[0006] Obtain at least one dataset corresponding to the data to be stored. Each dataset is obtained by dividing the data to be stored based on data attribute information. Each dataset corresponds to a data storage method.

[0007] Based on the amount of data in each of the above datasets, each of the above datasets is divided into at least one partition;

[0008] Based on the data storage method described above for each of the aforementioned shards, each of the aforementioned shards is stored in a distributed database.

[0009] On the other hand, embodiments of this application provide a data storage device, which includes:

[0010] The data acquisition module is used to acquire at least one dataset corresponding to the data to be stored. Each dataset is obtained by dividing the data to be stored based on data attribute information, and each dataset corresponds to a data storage method.

[0011] The data processing module is used to divide each of the above datasets into at least one partition based on the amount of data in each dataset.

[0012] The data storage module is used to store each of the above-mentioned shards in a distributed database based on the data storage method corresponding to each of the above-mentioned shards.

[0013] On the other hand, embodiments of this application provide an electronic device, including a processor and a memory, which are interconnected;

[0014] The aforementioned memory is used to store computer programs;

[0015] The processor described above is configured to execute the data storage method provided in the embodiments of this application when the computer program described above is invoked.

[0016] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the data storage method provided in embodiments of this application.

[0017] In this embodiment, the data to be stored can be pre-divided into different datasets according to data storage methods based on data attribute information. The distributed database can then divide each dataset into at least one shard and store each shard using a corresponding data storage method. Based on this approach, the distributed database can use multiple data storage methods to store different parts of the data to be stored, improving data storage flexibility. Furthermore, storing each shard using different storage methods ensures data independence between shards, improving the storage stability and data management efficiency of the data to be stored in the distributed database. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the data storage method provided in an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of a data storage scenario provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the flow framework of the data storage method provided in the embodiments of this application;

[0022] Figure 4 This is a schematic diagram of the structure of the data storage device provided in the embodiments of this application;

[0023] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] See Figure 1 , Figure 1 This is a flowchart illustrating the data storage method provided in an embodiment of this application. Figure 1 As shown, the data storage method provided in this application embodiment can be applied to distributed databases, and may specifically include the following steps:

[0026] Step S11: Obtain at least one dataset corresponding to the data to be stored.

[0027] In some feasible implementations, the data to be stored is the relevant data that needs to be stored in the distributed database. For example, it can be business data generated by the business system, or relevant data in other independent databases that need to be backed up through the distributed database. The specific data can be determined based on the actual application scenario requirements, and there are no restrictions here.

[0028] The datasets corresponding to the data to be stored can be obtained by pre-dividing the data to be stored based on the data attribute information of each part of the data. That is, at least one dataset corresponding to the data to be stored can be obtained directly before the data to be stored needs to be stored. Alternatively, the data to be stored can be obtained first, and the distributed database can divide the data to be stored based on the data attribute information of each part of the data to obtain at least one dataset. The specific implementation method is not limited here.

[0029] Each dataset corresponds to a data storage method, and the data storage method for each part of the data in each dataset is determined based on the data attribute information of each part of the data in that dataset.

[0030] In other words, for each part of the data to be stored, the data storage method corresponding to that part of the data can be determined based on the data attribute information corresponding to that part of the data. This allows us to obtain the data storage method corresponding to each part of the data to be stored, and thus divide each part of the data to be stored into at least one dataset according to different data storage methods. Each part of the data in each dataset corresponds to the same data storage method.

[0031] Among them, any part of the data to be stored can be at least one piece of data in the data to be stored, such as a business record, or one or more data tables in the data to be stored. The specific determination can be based on the actual application scenario requirements, and there are no restrictions here.

[0032] Optionally, the aforementioned data attribute information includes at least one of data modification information, data access information, data writing information, or data query information.

[0033] Among them, data modification information is used to characterize whether data may be modified after it is stored in a distributed database, and if it may be modified, the number of times or the frequency of modification.

[0034] Data access information is used to characterize data access status, such as the number or frequency of data accesses, or the number or frequency of access anomalies. Furthermore, for any portion of the data to be stored, the data access information for that portion can be determined based on data access information for that portion of data in other storage spaces, or based on access information for the same data in a distributed database. The specific determination can be based on the actual application scenario requirements and is not limited here.

[0035] Among them, the data writing information is used to characterize the data writing status, such as the number or frequency of data writing, the number of successful or failed data writings, etc., and there are no restrictions here.

[0036] The data query information is used to characterize the data query conditions, query scope, query method, etc., and there are no restrictions on it.

[0037] For example, when data attribute information is queried, different query methods can correspond to different data storage methods. Based on this, the data storage method corresponding to each part of the data to be stored can be determined based on the query method corresponding to each part of the data to be stored. Therefore, the parts of the data to be stored that correspond to the same data storage method can be defined as a dataset.

[0038] In some feasible implementations, the above data storage methods include at least one of columnar storage, storage based on multi-way balanced search tree (B-Tree), storage based on log-structured merge tree (LSM Tree), or storage based on hash index. Other data storage and management methods applied to distributed databases may also be included. The specific method can be determined based on the actual application scenario requirements and is not limited here.

[0039] For any part of the data to be stored, the data storage method corresponding to that part of the data can be determined based on one or more of the data attribute information mentioned above, and there are no restrictions here.

[0040] For example, for data that is mainly used for data analysis and is not modified or is modified infrequently, columnar storage can be used as the corresponding data storage method.

[0041] For example, for data with a small volume and a high frequency of access anomalies, the corresponding data storage method can be a B-Tree-based storage method.

[0042] For example, for data with a large volume and high writing frequency, the corresponding data storage method can be an LSM Tree-based storage method.

[0043] For example, for data with high query requirements and no query range, the corresponding data storage method can be a hash index-based storage method.

[0044] It should be noted that the above-mentioned data storage methods for determining the data of each part of the data to be stored are only examples. The specific methods can be determined based on the specific data attribute information and actual application scenario requirements, and no restrictions are imposed here.

[0045] In some feasible implementations, the data in each dataset exists in the form of key-value (kv) pairs. That is, after determining the data to be stored, each part of the data to be stored can be encoded into key-value pairs, and then after dividing the data to be stored into at least one dataset, the data in each dataset still exists in the form of key-value pairs.

[0046] Alternatively, after dividing the data to be stored into at least one dataset based on data attribute information, the data in each dataset can be encoded into key-value pairs.

[0047] Alternatively, after determining the data storage method of each part of the data to be stored based on the data attribute information, each part of the data can be encoded into key-value pairs, so that the data set that actually corresponds to the same data storage method includes the key-value pairs of the corresponding data.

[0048] Furthermore, the key value of each key-value pair in each dataset is associated with the data storage method corresponding to that dataset. That is, for each part of the data to be stored, the data storage method corresponding to that part of the data can be determined by the key value of the key-value pair corresponding to that part of the data.

[0049] For example, after encoding each part of the data into key-value pairs, a corresponding prefix is ​​added before the key to indicate the data storage method corresponding to that key-value pair.

[0050] Step S12: Based on the amount of data in each dataset, divide each dataset into at least one shard.

[0051] In some feasible implementations, for each dataset, if the amount of data in the dataset is less than or equal to a data size threshold, the distributed database does not need to divide the dataset into multiple shards; that is, the distributed database can treat the dataset as a single shard. If the amount of data in the dataset is greater than a certain data size threshold, it indicates that the amount of data in the dataset is large. In this case, the distributed database can divide the dataset into multiple shards, and the amount of data corresponding to each shard is also less than or equal to the data size threshold.

[0052] For example, if a dataset contains 10 TiB of data, a distributed database can divide the dataset into shards, each shard containing less than or equal to 10 GiB of data.

[0053] The specific data volume threshold mentioned above can be determined based on the actual application scenario requirements. For example, it can be 10 GiB or other thresholds, and there are no restrictions here.

[0054] Step S13: Store each fragment based on the data storage method corresponding to each fragment.

[0055] In some feasible implementations, for each shard, the distributed database can store the shard based on the data storage method corresponding to that shard. Each shard corresponds to an independent storage engine. Taking LSM Tree-based storage as an example, the distributed database can use an independent LSM Tree engine to store and manage each shard corresponding to that data storage method.

[0056] Therefore, data in any shard of a distributed database can be physically isolated from data in other shards. This means that data reorganization in one shard will not affect data in other shards. For example, when relocating data from any shard in a distributed database, the data in that shard can be physically copied directly using the storage engine corresponding to its data storage method.

[0057] In this case, the distributed database can store at least one shard corresponding to different data storage methods in a node of the distributed database, or it can store at least one shard corresponding to the same data storage method in a node of the distributed database. The specific method can be determined based on the actual application scenario requirements and is not restricted here.

[0058] like Figure 2 As shown, Figure 2This is a schematic diagram of a data storage scenario provided in an embodiment of this application. Corresponding to Region 1 in the LSM Tree storage method, the distributed database can store Region 1 in the distributed database based on the LSM Tree engine, and manage Region 1 based on the LSM Tree engine. Corresponding to Region 2 in the B-Tree storage method, the distributed database can store Region 2 in the distributed database based on the B-Tree engine, and manage Region 2 based on the B-Tree engine.

[0059] When physical isolation is achieved across shards in a distributed database—meaning each shard corresponds to an independent storage engine—compared to existing technologies where all shards share the same storage engine, this not only effectively reduces the probability of write amplification but also improves data read / write stability and speed, as well as data management efficiency. Furthermore, for each node in the distributed database, storing data using at least one shard corresponding to a different data storage method effectively reduces the storage capacity limitations imposed by the storage engine on each node, thereby increasing the maximum data storage capacity of the distributed database and reducing node consumption.

[0060] Taking the LSM Tree engine as an example, the amount of data that each node in a distributed database can store is no longer limited by the data storage capacity of the LSM Tree engine. Therefore, each node can support a larger amount of data storage, while also avoiding unnecessary data writing and excessive data cleaning tasks when a node stores a lot of data.

[0061] In some feasible implementations, to prevent data loss or storage failures in some shards due to node anomalies in the distributed database, after dividing each dataset into at least one shard, the distributed database can determine at least one backup shard for each shard. The distributed database can further store the shard and its corresponding backup shard based on the data storage method corresponding to that shard.

[0062] In this system, for each shard, the node in the distributed database used to store the backup shard of that shard is a different node from the node used to store the shard itself. Furthermore, each backup shard of that shard is stored on a different node in the distributed database.

[0063] Specifically, for each shard, when storing the shard and its corresponding backup shards, the distributed database can perform consistency verification on the shard and its backup shards based on a distributed consistency protocol to ensure that the data in the shard and its backup shards remains consistent.

[0064] In some feasible implementations, after the distributed database stores each shard, if the data of each shard is accessed a lot, it may consume too much computing resources of the node storing the shard, while consuming less computing resources of other nodes, resulting in an uneven utilization of computing resources among the nodes.

[0065] Based on this, after storing the data to be stored in each shard, the distributed database can determine the access frequency of at least one shard, and then make further adjustments to the shard based on the access frequency and data storage method of the shard.

[0066] Specifically, for at least one data storage method (hereinafter referred to as the first data storage method for ease of description), the distributed database can determine whether there are multiple shards in the distributed database corresponding to each first data storage method that have an access frequency less than a preset threshold (hereinafter referred to as the first shard for ease of description). If so, the first shards corresponding to the first data storage method are merged to obtain at least one merged shard. For example, if there are two first shards in all the shards corresponding to the first data storage method, the distributed database can directly merge these two first shards to obtain a merged shard.

[0067] For example, for each first data storage method, if the distributed database has a data storage capacity limit for each shard, the distributed database can divide the multiple first shards corresponding to that first data storage method into at least one shard set. Each shard set includes multiple first shards, where the data volume of any first shard in each shard set is less than the aforementioned data storage capacity limit, and the sum of the data volumes of all first shards in each shard set does not exceed the aforementioned data storage capacity limit. The distributed database can then merge the first shards in each shard set to obtain a merged shard.

[0068] Among them, the above-mentioned first data storage method can be any of the data storage methods corresponding to the distributed database, or any one or more data storage methods corresponding to the distributed database, without any restrictions.

[0069] Optionally, for at least one data storage method (hereinafter referred to as the second data storage method for ease of description), the distributed database can determine whether there exists a shard with an access frequency greater than or equal to a preset threshold in at least one shard corresponding to each second data storage method (hereinafter referred to as the second shard for ease of description). If such a shard exists, the distributed database will split each second shard into multiple sub-shards. When splitting each second shard, the distributed database can evenly divide the second shard into a preset number of sub-shards, without limitation.

[0070] Among them, the above-mentioned at least one second data storage method can be any of the data storage methods corresponding to the distributed database, or any one or more data storage methods corresponding to the distributed database, without any restrictions.

[0071] The following is combined with Figure 3 The data storage method provided in the embodiments of this application will be further described. Figure 3 This is a schematic diagram of the flow framework of the data storage method provided in the embodiments of this application. For example... Figure 3 As shown, after obtaining at least one dataset corresponding to the data to be stored, the distributed database can divide each dataset into multiple shards based on the amount of data in each dataset, and each shard corresponds to a data storage method. Therefore, for each shard, the distributed database can store the shard according to the data storage method corresponding to that shard.

[0072] Following this, the distributed database can determine the access frequency of each shard in the distributed database, such as determining the access frequency of each shard in the distributed database up to the corresponding time interval at preset time intervals. For each data storage method, if there are multiple first shards in each shard corresponding to that data storage method with access frequencies less than a preset threshold, the distributed database can merge the first shards corresponding to that data storage method to obtain at least one merged shard.

[0073] Similarly, for each data storage method, if there is a second shard in each shard corresponding to the data storage method with an access frequency greater than or equal to a preset threshold, the distributed database can split each second shard into multiple sub-shards.

[0074] After this, the distributed database can redetermine the access frequency of each shard (including merged shards and sub-shards) in the distributed database, and readjust each shard based on the access frequency of each shard, thereby realizing dynamic management of each shard in the distributed database.

[0075] In this embodiment, the data storage method corresponding to each part of the data to be stored can be predetermined through data attribute information. Then, based on the data storage method corresponding to each part, the data to be stored is divided into different datasets, so that each dataset corresponds to a data storage method. The distributed database can then divide each dataset into at least one shard and store each shard based on the corresponding data storage method, thereby achieving physical isolation between shards in the distributed database. Each shard corresponds to an independent data management method, which not only effectively reduces the probability of write amplification but also improves the stability and speed of data read and write, as well as the data management efficiency of the distributed database. Furthermore, for each node in the distributed database, each node can store at least one shard corresponding to different data storage methods, effectively reducing the storage engine's storage capacity limit on each node, thereby increasing the data storage limit of the distributed database and reducing node consumption.

[0076] See Figure 4 , Figure 4 This is a schematic diagram of the structure of the data storage device provided in an embodiment of this application. The data storage device provided in an embodiment of this application includes:

[0077] Data acquisition module 41 is used to acquire at least one dataset corresponding to the data to be stored. Each dataset is obtained by dividing the data to be stored based on data attribute information. Each dataset corresponds to a data storage method.

[0078] Data processing module 42 is used to divide each of the above datasets into at least one fragment based on the amount of data in each dataset;

[0079] Data storage module 43 is used to store each of the above-mentioned fragments in a distributed database based on the above-mentioned data storage method corresponding to each of the above-mentioned fragments.

[0080] In some feasible implementations, the aforementioned data attribute information includes at least one of the following: data volume, data modification information, data access information, data writing information, or data query information.

[0081] In some feasible implementations, the data in each of the aforementioned datasets is stored in key-value pairs, and the key value of each key-value pair in each of the aforementioned datasets is associated with the aforementioned data storage method corresponding to each of the aforementioned datasets.

[0082] In some feasible implementations, the above data storage method includes at least one of columnar storage, storage method based on multi-way balanced search tree (B-Tree), storage method based on log structure merge tree (LSM Tree), or storage method based on hash index.

[0083] In some feasible implementations, after storing each of the above-mentioned shards in a distributed database based on the above-mentioned data storage method corresponding to each of the above-mentioned shards, the data storage module 43 is further used for:

[0084] Determine the access frequency of at least one of the aforementioned shards in the distributed database;

[0085] Based on the access frequency and data storage method of at least one of the above-mentioned fragments, adjustments are made to at least one of the above-mentioned fragments.

[0086] In some feasible implementations, the data storage module 43 described above is used for:

[0087] For at least one of the above data storage methods, in response to the existence of multiple first fragments with access frequencies less than a preset threshold among the multiple fragments corresponding to the above data storage method, the multiple first fragments are merged to obtain at least one merged fragment, and the at least one merged fragment is stored based on the above data storage method.

[0088] In some feasible implementations, the data storage module 43 is further used for:

[0089] For at least one of the above data storage methods, in response to the existence of at least one second fragment with an access frequency greater than or equal to the preset threshold in at least one of the above fragments corresponding to the above data storage method, each of the above second fragments is split into multiple sub-fragments, and each of the above sub-fragments is stored based on the above data storage method.

[0090] In specific implementation, the aforementioned data storage device can perform the above-described functions through its built-in functional modules. Figure 1 The implementation methods provided for each step are detailed in the above-mentioned implementation methods, and will not be repeated here.

[0091] See Figure 5 , Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 5As shown, the electronic device 500 in this embodiment may include: a processor 501, a network interface 504, and a memory 505. Furthermore, the electronic device 500 may also include: a user interface 503, and at least one communication bus 502. The communication bus 502 is used to enable communication between these components. The user interface 503 may include a display screen and a keyboard; optionally, the user interface 503 may also include a standard wired interface or a wireless interface. The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 504 may be a high-speed RAM or non-volatile memory (NVM), such as at least one disk storage device. The memory 505 may optionally be at least one storage device located remotely from the processor 501. Figure 5 As shown, the memory 505, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0092] exist Figure 5 In the illustrated electronic device 500, the network interface 504 provides network communication functionality; the user interface 503 primarily provides an input interface for the user; and the processor 501 can be used to call the device control application stored in the memory 505 to achieve:

[0093] Obtain at least one dataset corresponding to the data to be stored. Each dataset is obtained by dividing the data to be stored based on data attribute information. Each dataset corresponds to a data storage method.

[0094] Based on the amount of data in each of the above datasets, each of the above datasets is divided into at least one partition;

[0095] Based on the data storage method described above for each of the aforementioned shards, each of the aforementioned shards is stored in a distributed database.

[0096] In some feasible implementations, the aforementioned data attribute information includes at least one of the following: data volume, data modification information, data access information, data writing information, or data query information.

[0097] In some feasible implementations, the data in each of the aforementioned datasets is stored in key-value pairs, and the key value of each key-value pair in each of the aforementioned datasets is associated with the aforementioned data storage method corresponding to each of the aforementioned datasets. In some feasible implementations, the aforementioned data storage method includes at least one of columnar storage, storage method based on a multi-way balanced search tree (B-Tree), storage method based on a log structure merge tree (LSM Tree), or storage method based on a hash index.

[0098] In some feasible implementations, after storing each of the above-mentioned data storage methods corresponding to each of the above-mentioned data fragments in a distributed database, the processor 501 is further configured to:

[0099] Determine the access frequency of at least one of the aforementioned shards in the distributed database;

[0100] Based on the access frequency and data storage method of at least one of the above-mentioned fragments, adjustments are made to at least one of the above-mentioned fragments.

[0101] In some feasible implementations, the processor 501 described above is used for:

[0102] For at least one of the above data storage methods, in response to the existence of multiple first fragments with access frequencies less than a preset threshold among the multiple fragments corresponding to the above data storage method, the multiple first fragments are merged to obtain at least one merged fragment, and the at least one merged fragment is stored based on the above data storage method.

[0103] In some feasible implementations, the processor 501 is further configured to:

[0104] For at least one of the above data storage methods, in response to the existence of at least one second fragment with an access frequency greater than or equal to the preset threshold in at least one of the above fragments corresponding to the above data storage method, each of the above second fragments is split into multiple sub-fragments, and each of the above sub-fragments is stored based on the above data storage method.

[0105] It should be understood that in some feasible implementations, the processor 501 described above may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0106] In specific implementation, the aforementioned electronic device 500 can perform the above-described actions through its built-in functional modules. Figure 1 The implementation methods provided for each step are detailed in the above-mentioned implementation methods, and will not be repeated here.

[0107] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement... Figure 1 The methods provided in each step are detailed in the implementation methods provided in the above steps, and will not be repeated here.

[0108] The aforementioned computer-readable storage medium can be an internal storage unit of the data storage device or electronic device provided in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. The aforementioned computer-readable storage medium can also include magnetic disks, optical disks, read-only memory (ROM), or random access memory (RAM), etc. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the electronic device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0109] This application provides a computer program product, which includes a computer program that is executed by a processor. Figure 1 The methods provided for each step in the process.

[0110] The terms "first," "second," etc., used in the claims, description, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or electronic device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or electronic devices. References to "embodiment" herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The presentation of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and / or" as used in this application's description and appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0111] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

[0112] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data storage method, characterized in that, The method includes: At least one dataset corresponding to the data to be stored is obtained. Each dataset is obtained by partitioning the data to be stored based on data attribute information. Each dataset corresponds to a data storage method, and the data storage method corresponding to each dataset is determined based on one or more of the data attribute information. The data attribute information includes at least one of the following: data volume, data modification information, data access information, data write information, or data query information. The data modification information characterizes whether the data may be modified after being stored in the database, and the possible number or frequency of modifications if modification is possible. The data access information characterizes the data access situation. The data write information characterizes the data write situation. The data query information characterizes the data query conditions, query range, and / or query method. The data storage method includes at least one of the following: columnar storage, storage based on a multi-way balanced search tree (B-Tree), storage based on a log structure merge tree (LSM Tree), or storage based on a hash index. Based on the amount of data in each dataset, each dataset is divided into at least one partition; Based on the data storage method corresponding to each shard, each shard is stored in a distributed database; each shard corresponds to an independent storage engine, so that the data of any shard in the distributed database is physically isolated from the data of other shards in the distributed database. Determine the access frequency of at least one of the shards in the distributed database; For at least one of the data storage methods, in response to the existence of multiple first fragments with access frequencies less than a preset threshold among the multiple fragments corresponding to the data storage method, the multiple first fragments are merged to obtain at least one merged fragment, and the at least one merged fragment is stored based on the data storage method; in response to the existence of at least one second fragment with access frequencies greater than or equal to the preset threshold among the at least one fragment corresponding to the data storage method, each second fragment is split to obtain multiple sub-fragments, and each sub-fragment is stored based on the data storage method; wherein, the merging and splitting are performed between fragments corresponding to the same data storage method.

2. The method according to claim 1, characterized in that, The data in each dataset is stored in key-value pairs, and the key value of each key-value pair in each dataset is associated with the data storage method corresponding to each dataset.

3. A data storage device, characterized in that, The device includes: The data acquisition module is used to acquire at least one dataset corresponding to the data to be stored. Each dataset is obtained by dividing the data to be stored based on data attribute information. Each dataset corresponds to a data storage method, and the data storage method corresponding to each dataset is determined based on one or more of the data attribute information. The data attribute information includes at least one of the following: data volume, data modification information, data access information, data write information, or data query information. The data modification information characterizes whether the data may be modified after being stored in the database, and the possible number or frequency of modifications if modification is possible. The data access information characterizes the data access status. The data write information characterizes the data write status. The data query information characterizes the data query conditions, query range, and / or query method. The data storage method includes at least one of the following: columnar storage, storage based on a multi-way balanced search tree (B-Tree), storage based on a log structure merge tree (LSM Tree), or storage based on a hash index. A data processing module is used to divide each dataset into at least one partition based on the amount of data in each dataset; The data storage module is used to store each shard in a distributed database based on the data storage method corresponding to each shard; each shard corresponds to an independent storage engine, so that the data of any shard in the distributed database is physically isolated from the data of other shards in the distributed database; The data storage module is configured to determine the access frequency of at least one of the shards in the distributed database; for at least one of the data storage methods, in response to the existence of multiple first shards with access frequencies less than a preset threshold among the multiple shards corresponding to the data storage method, the multiple first shards are merged to obtain at least one merged shard, and the at least one merged shard is stored based on the data storage method; in response to the existence of at least one second shard with access frequencies greater than or equal to the preset threshold among the at least one shard corresponding to the data storage method, each second shard is split to obtain multiple sub-shards, and each sub-shard is stored based on the data storage method; wherein, the merging and splitting are performed between shards corresponding to the same data storage method.

4. An electronic device, characterized in that, It includes a processor and a memory, which are interconnected; The memory is used to store computer programs; The processor is configured to perform the method as described in claim 1 or 2 when the computer program is invoked.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method of claim 1 or 2.

Images