Distributed storage method, device, equipment and computer readable storage medium

Abstract

The application provides a distributed storage method, device and equipment and a computer readable storage medium. The method comprises the following steps: receiving a first request carrying to-be-stored data sent by a terminal; generating a first storage path of the to-be-stored data according to a preset directory coding rule, wherein the first storage path comprises multiple levels of directories; wherein the name of the deepest level of directory in the first storage path comprises a partition key, a minimum block number, a maximum block number and a merging number, and the partition keys corresponding to different distributed nodes are different; and writing the to-be-stored data into a distributed storage space based on the first storage path. Since the partition keys corresponding to different distributed nodes are different, the storage paths corresponding to the data stored by different distributed nodes are ensured to be different, thus reducing the merging of data, ensuring that the data amounts of different data blocks are relatively small, and further ensuring that the data amounts loaded by different distributed nodes are relatively small when data is queried, thereby improving the query efficiency.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and includes, but is not limited to, a distributed storage method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Currently, many databases with high write throughput requirements rely on LogStructured Merge Tree (LSM-Tree) for read and write operations. LSM-Tree is a hierarchical, ordered, disk-oriented data structure designed to read or write a fixed-size block of data at a time, minimizing random seek operations to optimize write performance. Furthermore, periodic merging effectively removes invalid data, shortens read paths, and improves disk space utilization.

[0003] However, merging multiple data blocks is extremely resource-intensive, consuming significant amounts of CPU and disk I / O (Input / Output) speeds, especially during peak business hours, which can reduce the overall system throughput. Furthermore, during the write process, as the smallest directory unit of data storage (part) is continuously merged, uneven data volume across parts can easily occur. During the query phase, it becomes difficult for each node to guarantee a uniform amount of data loaded, leading to the "weakest link" effect and impacting query efficiency. Summary of the Invention

[0004] In view of the above, embodiments of this application provide a distributed storage method, apparatus, device, and computer-readable storage medium.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a distributed storage method, the method comprising:

[0007] The receiving terminal sends a first request, which carries data to be stored.

[0008] According to the preset directory encoding rules, a first storage path for the data to be stored is generated, and the first storage path includes multiple levels of directories;

[0009] The deepest directory name in the first storage path includes: partition key, minimum block number, maximum block number, and number of merges. The partition key is different for different distributed nodes.

[0010] Based on the first storage path, the data to be stored is written into the distributed storage space.

[0011] In some embodiments, generating a first storage path for the data to be stored according to a preset directory encoding rule includes:

[0012] Obtain the organizational structure and preset directory encoding rules of the distributed storage space;

[0013] Based on the organizational structure and the data to be stored, determine the root directory name, data storage directory name, database directory name, and table directory name of the distributed storage space;

[0014] The deepest level directory name is determined based on the directory encoding rules and the data to be stored;

[0015] The first storage path for the data to be stored is generated based on the root directory name, the data storage directory name, the database directory name, the table directory name, and the deepest level directory name.

[0016] In some embodiments, determining the deepest directory name based on the directory encoding rules and the data to be stored includes:

[0017] Obtain the identifier of the distributed node;

[0018] Determine the partition key based on the identifier and the data to be stored;

[0019] Get the minimum block number, maximum block number, and number of merges assigned by the distributed application coordination service;

[0020] According to the directory encoding rules, the partition key, minimum block number, maximum block number, and number of merges are processed to obtain the deepest level directory name.

[0021] In some embodiments, determining the partition key based on the identifier and the data to be stored includes:

[0022] The data to be stored is parsed to obtain a pre-defined user partition;

[0023] Based on the identifier, at least one preset expansion factor of the distributed node and the logical shard corresponding to each expansion factor are determined, and the logical shards corresponding to different expansion factors are different.

[0024] The partition key is determined based on the user partition, at least one preset expansion factor, and the logical shard corresponding to each expansion factor.

[0025] In some embodiments, writing the data to be stored into the distributed storage space based on the first storage path includes:

[0026] Based on the first storage path, search in the distributed storage space whether there is target data that matches the first storage path;

[0027] When the search result indicates that the target data does not exist, the data to be stored is written into the distributed storage space according to the first storage path.

[0028] In some embodiments, writing the data to be stored into the distributed storage space based on the first storage path further includes:

[0029] When the search result indicates that the target data exists, the target data and the data to be stored are merged to obtain merged data;

[0030] Based on the first storage path, the storage path of the target data is updated to obtain the second storage path;

[0031] The merged data is written to the distributed storage space according to the second storage path.

[0032] In some embodiments, the first request further carries metadata information of the data to be stored; the method further includes:

[0033] The metadata information is sent to the remaining distributed nodes in the distributed system for data synchronization;

[0034] Receive metadata information sent by other distributed nodes in the distributed system for data synchronization;

[0035] Each distributed node in the distributed system has metadata information of the data written to the storage space by all distributed nodes.

[0036] This application provides a distributed storage device, the device comprising:

[0037] The first receiving module is used to receive a first request sent by the terminal, the first request carrying data to be stored;

[0038] The generation module is used to generate a first storage path for the data to be stored according to a preset directory encoding rule. The first storage path includes multiple levels of directories.

[0039] The deepest directory name in the first storage path includes: partition key, minimum block number, maximum block number, and number of merges. The partition key is different for different distributed nodes.

[0040] The write module is used to write the data to be stored into the distributed storage space based on the first storage path.

[0041] This application provides an electronic device, including:

[0042] Processor; and

[0043] Memory for storing computer programs that can run on the processor;

[0044] The computer program, when executed by a processor, implements the steps of the above-described distributed storage method.

[0045] This application provides a computer-readable storage medium storing computer-executable instructions configured to perform the steps of the above-described distributed storage method.

[0046] This application provides a distributed storage method, apparatus, device, and computer-readable storage medium. The method includes: receiving a first request from a terminal carrying data to be stored; generating a first storage path for the data to be stored according to a preset directory encoding rule. The first storage path includes multiple levels of directories, where the deepest level directory name includes a partition key, a minimum block number, a maximum block number, and a merging count. Different distributed nodes have different partition keys, ensuring that the data written by different distributed nodes corresponds to different storage paths, reducing data merging and ensuring that the data volume of different data blocks is relatively similar. Finally, based on the first storage path, the data to be stored is written to the distributed storage space. Because the data volume of different data blocks is relatively similar, when querying data, it can be ensured that the data volume loaded by each distributed node is relatively similar, avoiding a significant bottleneck effect and thus improving query efficiency. Attached Figure Description

[0047] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar parts in different views. The drawings illustrate, by way of example and not limitation, the various embodiments discussed herein.

[0048] Figure 1 This is a schematic diagram illustrating an implementation process of the distributed storage method provided in an embodiment of this application;

[0049] Figure 2 This is a schematic diagram illustrating one implementation process of the step of generating the first storage path for data to be stored in the distributed storage method provided in this application embodiment;

[0050] Figure 3 This is a schematic diagram illustrating one implementation process of writing data to be stored into a distributed storage space in the distributed storage method provided in this application embodiment;

[0051] Figure 4 A schematic diagram of the architecture of a distributed database file system based on a shared storage class LSM engine with storage and computation separation, provided for embodiments of this application;

[0052] Figure 5This is a schematic diagram of a distributed database file directory structure based on a shared storage-class LSM engine, providing a storage-computation separation mechanism, as illustrated in an embodiment of this application.

[0053] Figure 6 This is a schematic diagram of the original distributed write logic in existing technology;

[0054] Figure 7 This is a schematic diagram of the distributed write logic provided in an embodiment of this application;

[0055] Figure 8 This is a schematic diagram of the distributed query logic provided in an embodiment of this application;

[0056] Figure 9 This is a schematic diagram of the composition structure of a distributed storage device provided in an embodiment of this application;

[0057] Figure 10 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0060] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0062] Before providing a more detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application shall be explained as follows.

[0063] Shared memory refers to a parallel architecture in which two or more processors share a single main memory.

[0064] Zookeeper: A distributed service framework, a sub-project of Apache Hadoop. It is mainly used to solve some data management problems often encountered in distributed applications, such as: unified naming service, state synchronization service, cluster management, and management of distributed application configuration items.

[0065] Log Structured Merge Tree (LSM-Tree) is a hierarchical, ordered, disk-oriented data structure. Its core idea is to fully leverage the fact that sequential writes in batches are far more efficient than random writes on disk, sacrificing some read efficiency for maximum write efficiency. The design philosophy primarily involves reading or writing a fixed-size block of data at a time, minimizing random seek operations.

[0066] Online Analytical Processing (OLAP) is a software technology that enables analysts to quickly, consistently, and interactively observe information from various perspectives to achieve a deeper understanding of the data.

[0067] In the distributed query logic of related technologies, the smallest file unit for querying data is a part, and each node is responsible for querying different parts. During the writing process, as parts are continuously merged, uneven distribution of data volume among parts can easily occur. As a result, it is difficult for each node to guarantee a uniform amount of data loaded during queries, which can lead to the "weakest link" effect and compromise query efficiency.

[0068] To address the aforementioned problems, embodiments of this application provide a distributed storage method and apparatus. The method provided in these embodiments will be described below in conjunction with the apparatus used to implement them. Figure 1 This is a schematic diagram illustrating an implementation process of the distributed storage method provided in this application embodiment. The method is applied to distributed nodes, such as... Figure 1 As shown, the method includes the following steps:

[0069] Step S101: Receive the first request sent by the terminal.

[0070] This embodiment of the application can be executed by distributed nodes in a distributed system. When a user needs to store data to the storage space of the distributed system, they execute a user operation on their terminal to trigger the data storage. The terminal responds to this user operation by generating a first request based on the data to be stored. The terminal sends this first request to the distributed system. After receiving the first request from the terminal, the distributed nodes of the distributed system parse it to obtain the data to be stored carried in the first request.

[0071] Step S102: Generate the first storage path for the data to be stored according to the preset directory encoding rules.

[0072] The first storage path here includes multiple levels of directories. The directory name of the deepest level includes: partition key, minimum block number, maximum block number, and merge count. The partition key includes, but is not limited to: user partition, at least one pre-defined expansion factor of the distributed node, and the logical shard corresponding to each expansion factor. Each expansion factor of the distributed node corresponds one-to-one with a logical shard, and the partition key is different for different distributed nodes.

[0073] In some embodiments, step S102 can be achieved through... Figure 2 The following steps are shown to achieve this:

[0074] Step S1021: Obtain the organizational structure of the distributed storage space and the preset directory encoding rules.

[0075] Currently, most LSM-like databases do not employ a shared storage model. To reduce costs and improve write and query performance, this application embodiment uses a shared storage-based LSM engine for data writing and reading. In one implementation, distributed nodes are separated from the shared storage. Distributed nodes can horizontally scale up or down in seconds, and the storage nodes, i.e., the distributed file storage system, can be infinitely expanded. Because shared storage is used, data does not need to be redistributed, and tools that provide distributed coordination services can be used for data synchronization, such as using Zookeeper as the coordination service node.

[0076] The organizational structure of a distributed storage space is the storage structure of data files in a distributed database, consisting of multi-level directories. Directory encoding rules are pre-defined, and directories conforming to these rules can be generated.

[0077] Step S1022: Based on the organizational structure and the data to be stored, determine the root directory name, data storage directory name, database directory name, and table directory name of the distributed storage space.

[0078] Taking a five-level directory structure as an example: The first-level directory is the path to the entire shared storage, the root directory of the database, and all distributed nodes in the distributed system can access and modify all files and directories under the first-level directory. The second-level directory can be used to store metadata (database and table metadata), the top-level directory for storing data (data), configuration files (configs), and permission management, etc. The written data is stored in the data directory, and the method provided in this application embodiment is mainly optimized for this data storage directory. The third-level directory for storing data is the database name, the fourth-level directory is the table name, and the fifth-level directory is the smallest directory unit for data storage, called a part. Each write operation by a distributed node generates at least one part.

[0079] Step S1023: Determine the name of the deepest directory based on the directory encoding rules and the data to be stored.

[0080] In one implementation, determining the deepest level directory name can be achieved by: obtaining the identifier of the distributed node; determining the partition key based on the identifier and the data to be stored; obtaining the minimum block number, maximum block number, and number of merges allocated by the distributed application coordination service; and processing the partition key, minimum block number, maximum block number, and number of merges according to the directory encoding rules to obtain the deepest level directory name.

[0081] For example, the default directory encoding rule can be partition key-minimum block number-maximum block number-merge count. Different distributed nodes have different partition keys, ensuring that data written to different distributed nodes corresponds to different storage paths, reducing data merging and ensuring that the data volume of different data blocks is relatively similar. The minimum and maximum block numbers are unique auto-incrementing sequences generated by ZooKeeper during the initial write, and the minimum and maximum block numbers are equal at this time, with a merge count of 0. As the LSM engine merges data, the minimum and maximum block numbers and the merge count will change accordingly. The smallest directory for data storage contains the compressed data, such as primary key indexes and column information.

[0082] In this embodiment, the partition key can consist of a user partition, a logical shard, and a scaling factor, wherein the user partition is predefined. In one implementation, the partition key can be determined as follows: parsing the data to be stored to obtain a predefined user partition; determining at least one predefined scaling factor and a logical shard corresponding to each scaling factor based on an identifier, with different scaling factors corresponding to different logical shards; and determining the partition key based on the user partition, the predefined at least one scaling factor, and the logical shard corresponding to each scaling factor.

[0083] For example, if a user partition is 202209, the expansion factor is 0, and the corresponding logical shard is 1, then the partition key can be represented as 202209_1_0; the minimum and maximum block numbers generated by Zookeeper are 2, and the number of merges is 0. Thus, based on the directory encoding rules and the data to be stored, the deepest level directory name is determined to be 202209_1_0_2_2_0.

[0084] Step S1024: Generate the first storage path for the data to be stored based on the root directory name, the data storage directory name, the database directory name, the table directory name, and the deepest level directory name.

[0085] Based on the two steps above, the root directory is named root, the data storage directory is named data, the database directory is named test, the table directory is named demo, and the deepest directory is named 202209_1_0_2_2_0. Therefore, the first storage path for the generated data to be stored is root / data / test / demo / 202209_1_0_2_2_0 / .

[0086] Step S103: Based on the first storage path, write the data to be stored into the distributed storage space.

[0087] In this embodiment, a distributed node receives a first request from a terminal carrying data to be stored. Based on a preset directory encoding rule, it generates a first storage path for the data to be stored. This first storage path includes multiple levels of directories. The deepest level directory name in the first storage path includes: partition key, minimum block number, maximum block number, and number of merges. Different distributed nodes have different partition keys, ensuring that the data written by different distributed nodes corresponds to different storage paths, reducing data merging and ensuring that the data volume of different data blocks is relatively similar. Finally, based on the first storage path, the data to be stored is written to the distributed storage space. Because the data volume of different data blocks is relatively similar, when querying data, it ensures that the data volume loaded by each distributed node is relatively similar, avoiding a significant bottleneck effect and thus improving query efficiency.

[0088] In some embodiments, the first request carries not only the data to be stored, but also metadata information of the data to be stored, and different distributed nodes can synchronize data through a coordination service node (such as Zookeeper).

[0089] One specific implementation involves: sending metadata information to the remaining distributed nodes in the distributed system for data synchronization; and receiving metadata information sent by the remaining distributed nodes in the distributed system for data synchronization. Through data synchronization, each distributed node in the distributed system possesses the metadata information of all data written to the storage space by all distributed nodes. By distributing large amounts of data across different distributed nodes for writing, and synchronizing metadata information among different distributed nodes, when some distributed nodes fail, the data can be recovered by other distributed nodes without affecting the overall normal operation. Furthermore, even if the data stored on the failed computer is corrupted, the corrupted data can be recovered by other distributed nodes.

[0090] In some embodiments, computers outside the distributed system can be added to the distributed system for file storage and retrieval through simple configuration, or some distributed nodes can be deleted from the distributed system to achieve flexible expansion.

[0091] In some embodiments, the above Figure 1 Step S103 in the illustrated embodiment, "writing the data to be stored into the distributed storage space based on the first storage path," can be achieved through... Figure 3 The following steps are shown to achieve this:

[0092] Step S1031: Based on the first storage path, search in the distributed storage space for whether there is target data that matches the first storage path.

[0093] When the search result indicates that the target data does not exist, the data to be stored can be directly written to the distributed storage space, and then proceed to step S1032; when the search result indicates that the target data exists, proceed to step S1033.

[0094] Here, whether there is target data that matches the first storage path can be determined by the storage path. That is, check in the storage space whether the data has been stored under the first storage path. If it has been stored, then the data is the target data and the storage path of the target data matches the first storage path.

[0095] Step S1032: Write the data to be stored into the distributed storage space according to the first storage path.

[0096] For example, if the first storage path is root / data / test / demo / 202209_1_0_2_2_0 / , and the target data's storage path is root / data / test / demo / 202207_1_0_1_1_0 / , these two paths do not match because the user partitions in the partition keys are different. The data to be stored does not need to be merged and is directly written to the storage space. If the target data's storage path is root / data / test / demo / 202209_2_0_1_1_0 / , these two paths do not match because the logical partitions in the partition keys are different. The data to be stored does not need to be merged and is directly written to the storage space. If the target data's storage path is root / data / test / demo / 202209_1_1_1_1_0 / , these two paths do not match because the extension factors in the partition keys are different. The data to be stored does not need to be merged and is directly written to the storage space.

[0097] Step S1033: Merge the target data and the data to be stored to obtain merged data.

[0098] Step S1034: Update the storage path of the target data according to the first storage path to obtain the second storage path.

[0099] For example, if the first storage path is root / data / test / demo / 202209_1_0_2_2_0 / , and the target data's storage path is root / data / test / demo / 202209_1_0_1_1_0 / , with the same partition key, the target data and the data to be stored are merged. Based on the first storage path, the target data's storage path is changed to root / data / test / demo / 202209_1_0_1_2_1 / . The partition key in this second storage path remains unchanged. The minimum block number is the smaller of the minimum block number in the target data's storage path and the minimum block number in the first storage path. The maximum block number is the larger of the maximum block number in the target data's storage path and the maximum block number in the first storage path. The merge count is incremented by 1.

[0100] Step S1035: Write the merged data into the distributed storage space according to the second storage path.

[0101] In this embodiment of the application, when there is target data with the same path as the data to be stored in the distributed storage space, small files can be merged. The design and optimization fully utilize the principle that the performance of sequential writes in batches on the disk is much higher than that of random writes, so as to optimize the write performance of distributed storage. Furthermore, by periodically merging and slimming down, invalid data is effectively removed, which not only shortens the read path and improves read efficiency, but also improves disk space utilization.

[0102] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0103] In shared distributed file storage technology, large amounts of data are distributed across different nodes, significantly reducing the risk of data loss. Distributed file storage offers redundancy; the failure of some nodes does not affect the overall normal operation, and even if the data stored on a failed computer is corrupted, it can be recovered using other nodes. Furthermore, computers outside the cluster can be easily added to the distributed file storage to share data, demonstrating strong scalability.

[0104] ZooKeeper is a distributed, open-source coordination system for distributed applications. It's an open-source implementation of Google's Chubby, acting as the cluster manager, monitoring the status of each node and taking appropriate actions based on their feedback. Ultimately, it provides users with a simple, easy-to-use interface and a high-performance, stable system. The ZooKeeper distributed service framework primarily addresses common data management problems encountered in distributed applications, providing distributed, highly available coordination services.

[0105] In distributed file storage, many databases with high write throughput requirements revolve around Log Structured Merge Tree (LSM-Tree). LSM-Tree is a hierarchical, ordered, disk-oriented data structure. Its core idea is to fully leverage the principle that "sequential batch writes to disk are far more efficient than random writes" in its design and optimization, thereby maximizing write performance. Furthermore, through periodic merging and slimming, it effectively removes invalid data, shortens read paths, and increases available disk space. However, merging operations are very resource-intensive, consuming significant amounts of CPU and disk I / O, especially during peak business periods. This can reduce the overall system throughput, a challenging problem faced by most databases.

[0106] Currently, most LSM-like databases do not employ a shared storage model. This application provides a method and apparatus for optimizing distributed database file organization based on a shared storage-based LSM engine with storage and computation separation. This reduces storage costs and improves disk write and query performance. This application also provides a distributed architecture as shared storage, ensuring more even distribution of data across each node and mitigating the bottleneck effect.

[0107] Figure 4 This is a schematic diagram of the architecture of a distributed database file system based on a shared storage-class LSM engine with storage and computation separation, provided in an embodiment of this application. Figure 4 As shown, in this system, the compute nodes and shared storage (distributed file system) are separated, and the compute nodes can be horizontally scaled up or down in seconds. Because shared storage is used, data does not need to be redistributed, and the storage nodes (i.e., the distributed file system) can be expanded indefinitely. ZooKeeper is used as a coordination service node, providing distributed coordination task services, including distributed file merging and data synchronization.

[0108] Figure 5 This is a schematic diagram of a distributed database file directory structure based on a shared storage-class LSM engine, illustrating the separation of storage and computation, as provided in this application embodiment. Figure 5 As shown, the root directory is the path to the entire shared storage, which is the first-level directory of the database. All files and directories under the first-level directory can be accessed and modified by the compute nodes. The second-level directory mainly stores metadata (database and table metadata), data (the top-level directory for storing data), configs (configuration files), and permission management, etc.

[0109] In this embodiment, the optimization mainly focuses on the data storage directory. The third-level directory for storing data is the database name, the fourth-level directory is the table name, and the fifth-level directory is the smallest directory unit for data storage, referred to as a part. Each write operation generates at least one part. The directory encoding rule is: user-defined partition key - logical shard - pre-expansion factor - minimum block number - maximum block number - number of merges.

[0110] User-defined partition keys and data partitioning are of great significance for Online Analytical Processing (OLAP) databases. With the help of data partitioning, unnecessary data catalogs can be skipped in subsequent queries, thereby improving query performance. For example, users can partition by month or day, and when querying September data, they can skip the data catalogs of other months.

[0111] Logical sharding is designed primarily to distribute queries more evenly, thereby improving query efficiency and supporting local memory tables. Each compute node corresponds to 1 * pre-expansion factor logical shard.

[0112] The minimum and maximum block numbers are unique auto-incrementing sequences generated by ZooKeeper during the initial write, and the minimum and maximum block numbers are equal. The initial merge count is 0. As the LSM engine merges, the minimum block number, maximum block number, and merge count are generated accordingly.

[0113] The smallest directory for data storage contains the compressed data, primary key indexes, and column information.

[0114] The following is combined Figure 4 The system architecture diagram shown is as follows: Figure 5 The diagram showing the data file directory structure illustrates the distributed database file organization optimization method based on the storage-computation separation of a shared storage-type LSM engine provided in this application embodiment. This method mainly achieves the following three aspects:

[0115] Firstly, implement distributed write logic.

[0116] Figure 6 This is a schematic diagram of the original distributed write logic in the existing technology. See [link / reference] Figure 6 :

[0117] 1) The three computing nodes each write three batches of data. The data is partitioned by month and formed into three part files 202209_1_1_0, 202209_2_2_0 and 202209_3_3_0 under the same path of shared storage. The file name is user-defined partition key_minimum block number_maximum block number_merge count. The block number is uniformly assigned by Zookeeper and is auto-incremented and unique.

[0118] 2) Each computing node synchronizes data with each other, and each computing node has full metadata information.

[0119] 3) Data generated by different storage nodes can be merged. In existing technologies, when data generated by multiple storage nodes is merged concurrently, a large number of merge plan conflicts may occur.

[0120] 4) The three generated part files can be merged if they are from the same partition (i.e., the user-defined partition key value is the same), such as... Figure 6 The merged partition is 202206_1_2_1, where 202209 is a user-defined partition key, the first 1 is the smallest block number, the 2 is the largest block number, and the second 1 is the number of merges.

[0121] Because existing technologies merge files by partition, the part files in different partitions differ significantly, which can easily lead to uneven part sizes.

[0122] Figure 7 See the schematic diagram of the distributed write logic provided in the embodiments of this application. Figure 7 :

[0123] 1) Three batches of data are written to each of the three compute nodes. The data is partitioned by month and generated into three part files (202209_1_0_1_1_0, 202209_2_0_2_2_0, and 202209_3_0_3_3_0) on the same path in shared storage. The directory encoding rule is: user-defined partition key (corresponding to the user partition mentioned above) _ logical shard _ pre-expansion factor _ minimum block number _ maximum block number _ merge count. The block number is uniformly assigned by ZooKeeper and is auto-incrementing and unique.

[0124] 2) Each computing node synchronizes data with each other, and each computing node has full metadata information.

[0125] 3) Data generated by different storage nodes can be merged. Parts can only be merged with the same partition key. The partition key consists of user-defined partitions, logical shards, and pre-expansion factors.

[0126] 4) The three generated part files with the same partition key can be merged. Figure 7 The three partition keys shown are different and cannot be merged.

[0127] Secondly, implement distributed query logic.

[0128] In existing distributed query logic, the smallest file unit for querying is a part. Each computing node is responsible for querying different parts. As parts are continuously merged, it is difficult to ensure that the amount of data loaded by each computing node is equal. The reading efficiency of computing node A, which reads data from a part with a large amount of data, will be lower than that of computing node B, which reads data from a part with a small amount of data. The overall query efficiency will be affected by the weakest link effect because the reading efficiency of computing node A is lower than that of other computing nodes.

[0129] Figure 8 This is a schematic diagram of the distributed query logic provided in the embodiments of this application, such as... Figure 8 The distributed query logic provided in this application embodiment adds logical sharding and a pre-expansion factor. Data between different user-defined partitions, different logical shards, and different pre-expansion factors cannot be merged. See [link to relevant documentation]. Figure 8 The four batches of data inserted cannot be merged. They are queried separately. Each computing node queries different files according to the remainder rule. The amount of data read is relatively equal, and there will be no obvious bottleneck effect, thus improving the overall query efficiency.

[0130] Based on the foregoing embodiments, this application provides a distributed storage device. The modules and units included in the device can be implemented by a processor in a computer device; of course, they can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPG A), etc.

[0131] This application embodiment further provides a distributed storage device. Figure 9 This is a schematic diagram of the composition structure of a distributed storage device provided in an embodiment of this application, such as... Figure 9 As shown, the distributed storage device 900 includes:

[0132] The first receiving module 901 is used to receive a first request sent by the terminal, wherein the first request carries data to be stored.

[0133] The generation module 902 is used to generate a first storage path for the data to be stored according to a preset directory encoding rule. The first storage path includes multiple levels of directories.

[0134] The deepest directory name in the first storage path includes: partition key, minimum block number, maximum block number, and number of merges. The partition key is different for different distributed nodes.

[0135] The writing module 903 is used to write the data to be stored into the distributed storage space based on the first storage path.

[0136] In some embodiments, the generation module 902 is further configured to:

[0137] Obtain the organizational structure and preset directory encoding rules of the distributed storage space;

[0138] Based on the organizational structure and the data to be stored, determine the root directory name, data storage directory name, database directory name, and table directory name of the distributed storage space;

[0139] The deepest level directory name is determined based on the directory encoding rules and the data to be stored;

[0140] The first storage path for the data to be stored is generated based on the root directory name, the data storage directory name, the database directory name, the table directory name, and the deepest level directory name.

[0141] In some embodiments, the generation module 902 is further configured to:

[0142] Obtain the identifier of the distributed node;

[0143] Determine the partition key based on the identifier and the data to be stored;

[0144] Get the minimum block number, maximum block number, and number of merges assigned by the distributed application coordination service;

[0145] According to the directory encoding rules, the partition key, minimum block number, maximum block number, and number of merges are processed to obtain the deepest level directory name.

[0146] In some embodiments, the generation module 902 is further configured to:

[0147] The data to be stored is parsed to obtain a pre-defined user partition;

[0148] Based on the identifier, at least one preset expansion factor of the distributed node and the logical shard corresponding to each expansion factor are determined, and the logical shards corresponding to different expansion factors are different.

[0149] The partition key is determined based on the user partition, at least one preset expansion factor, and the logical shard corresponding to each expansion factor.

[0150] In some embodiments, the writing module 903 is further configured to:

[0151] Based on the first storage path, search in the distributed storage space whether there is target data that matches the first storage path;

[0152] When the search result indicates that the target data does not exist, the data to be stored is written into the distributed storage space according to the first storage path.

[0153] In some embodiments, the writing module 903 is further configured to:

[0154] When the search result indicates that the target data exists, the target data and the data to be stored are merged to obtain merged data;

[0155] Based on the first storage path, the storage path of the target data is updated to obtain the second storage path;

[0156] The merged data is written to the distributed storage space according to the second storage path.

[0157] In some embodiments, the first request further carries metadata information of the data to be stored; the distributed storage device 900 further includes:

[0158] The sending module is used to send the metadata information to the other distributed nodes in the distributed system for data synchronization;

[0159] The second receiving module is used to receive metadata information sent by other distributed nodes in the distributed system for data synchronization.

[0160] Each distributed node in the distributed system has metadata information of the data written to the storage space by all distributed nodes.

[0161] It should be noted that the descriptions of the above distributed storage device embodiments are similar to the method descriptions above, and have the same beneficial effects as the method embodiments. For technical details not disclosed in the distributed storage device embodiments of this application, those skilled in the art should refer to the descriptions of the method embodiments of this application for understanding.

[0162] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0163] Accordingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in the distributed storage method provided in the above embodiments.

[0164] This application provides an electronic device. Figure 10 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Figure 10 The exemplary structure of the electronic device 700 shown can be used to foresee other exemplary structures of the electronic device 1000. Therefore, the structure described herein should not be regarded as a limitation. For example, some components described below may be omitted, or components not described below may be added to suit the specific needs of certain applications.

[0165] Figure 10The illustrated electronic device 1000 includes: a processor 1001, at least one communication bus 1002, a user interface 1003, at least one external communication interface 1004, and a memory 1005. The communication bus 1002 is configured to enable communication between these components. The user interface 1003 may include a display screen, and the external communication interface 1004 may include standard wired and wireless interfaces. The processor 1001 is configured to execute a program of a distributed storage method stored in the memory to implement the steps of the distributed storage method provided in the above embodiments.

[0166] The descriptions of the above embodiments of the electronic devices and storage media are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the embodiments of the electronic devices and storage media of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0167] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0168] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0170] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0172] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0173] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0174] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A distributed storage method applied to a distributed node, characterized in that, The method comprises: receiving a first request sent by a terminal, the first request carrying to-be-stored data; generating a first storage path of the to-be-stored data according to a preset directory encoding rule, the first storage path comprising multiple levels of directories; wherein the name of the deepest level of directory in the first storage path comprises a partition key, a minimum block number, a maximum block number and a merging number, and the partition keys corresponding to different distributed nodes are different; writing the to-be-stored data into a distributed storage space based on the first storage path.

2. The method of claim 1, wherein, The generating of the first storage path of the to-be-stored data according to the preset directory encoding rule comprises: obtaining an organization structure of the distributed storage space and a preset directory encoding rule; determining a root directory name, a storage data directory name, a database directory name and a table directory name of the distributed storage space according to the organization structure and the to-be-stored data; determining a deepest level of directory name according to the directory encoding rule and the to-be-stored data; generating the first storage path of the to-be-stored data according to the root directory name, the storage data directory name, the database directory name, the table directory name and the deepest level of directory name.

3. The method of claim 2, wherein, The determining of the deepest level of directory name according to the directory encoding rule and the to-be-stored data comprises: obtaining an identity of the distributed node; determining a partition key according to the identity and the to-be-stored data; obtaining a minimum block number, a maximum block number and a merging number distributed by a distributed application program coordination service; processing the partition key, the minimum block number, the maximum block number and the merging number according to the directory encoding rule to obtain the deepest level of directory name.

4. The method of claim 3, wherein, The determining of the partition key according to the identity and the to-be-stored data comprises: analyzing the to-be-stored data to obtain a pre-defined user partition; determining at least one preset expansion factor of the distributed node and a logical shard corresponding to each expansion factor according to the identity, and the logical shards corresponding to different expansion factors are different; determining the partition key according to the user partition, the at least one preset expansion factor and the logical shard corresponding to each expansion factor.

5. The method of claim 1, wherein, The writing of the to-be-stored data into the distributed storage space based on the first storage path comprises: searching for whether there is target data matching the first storage path in the distributed storage space according to the first storage path; when the searching result represents that there is no target data, writing the to-be-stored data into the distributed storage space according to the first storage path.

6. The method of claim 5, wherein, The writing of the to-be-stored data into the distributed storage space based on the first storage path further comprises: when the searching result represents that there is target data, performing merging processing on the target data and the to-be-stored data to obtain merged data; updating the storage path of the target data according to the first storage path to obtain a second storage path; writing the merged data into the distributed storage space according to the second storage path.

7. The method of claim 1, wherein, The first request further carries metadata information of the to-be-stored data; the method further comprises: sending the metadata information to the remaining distributed nodes in the distributed system for data synchronization; Receiving metadata information sent by the rest of the distributed nodes in the distributed system for data synchronization; Each distributed node in the distributed system has metadata information of data written to the storage space by all distributed nodes.

8. A distributed storage apparatus, characterized by comprising: The device comprises: A first receiving module configured to receive a first request sent by a terminal, the first request carrying to-be-stored data; A generating module configured to generate a first storage path of the to-be-stored data according to a preset directory coding rule, the first storage path comprising multiple levels of directories; The name of the deepest level of directory in the first storage path comprises a partition key, a minimum block number, a maximum block number, and a merging number, and the partition keys corresponding to different distributed nodes are different; A writing module configured to write the to-be-stored data into a distributed storage space based on the first storage path.

9. An electronic device, comprising: Comprise: A processor; And A memory configured to store a computer program executable on the processor; When the computer program is executed by the processor, the steps of the distributed storage method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer executable instructions are configured to execute the steps of the distributed storage method according to any one of claims 1 to 7. The computer executable instructions are configured to execute the steps of the distributed storage method according to any one of claims 1 to 7.

Images