Heterogeneous computing-based distributed storage method, system, device and medium

Abstract

The application discloses a distributed storage method, system, device and medium based on heterogeneous computing, belongs to the technical field of distributed database storage, and aims to solve the technical problem of how to avoid the unbalanced condition in a heterogeneous environment and realize effective differentiated data distribution. The technical scheme is as follows: according to the configuration of the Memory and CPU of each node in the cluster, the upper limit of the file size and the weight of the processing capacity are calculated, the data is divided into partitions by using a hash key, and the specific position of each partition is located by the node weight; meanwhile, when the data exceeds the maximum threshold of the current file during the storage of the data, a new branch is created by using a binary tree fractal growth method, and effective differentiated data distribution in the heterogeneous computing environment is realized. The system comprises a threshold acquisition module, a weight acquisition module, a data partition module, a weight partition module and a storage module.

Description

Technical Field

[0001] This invention relates to the field of distributed data storage technology, specifically to a distributed storage method, system, device, and medium based on heterogeneous computing. Background Technology

[0002] A distributed database refers to a database that uses high-speed computer networks to connect multiple physically dispersed data storage units to form a logically unified database. The basic idea of ​​a distributed database is to distribute the data from a centralized database across multiple data storage nodes connected by a network, in order to obtain greater storage capacity and higher concurrent access capacity.

[0003] In recent years, with the rapid growth of data volume, distributed database technology has also developed rapidly. Traditional relational databases have begun to evolve from a centralized model to a distributed architecture. Relational-based distributed databases, while retaining the data model and basic characteristics of traditional databases, have moved from centralized storage to distributed storage, and from centralized computing to distributed computing. Their characteristics are as follows:

[0004] ① High scalability: Distributed databases must be highly scalable, capable of dynamically adding storage nodes to achieve linear expansion of storage capacity.

[0005] ② High concurrency: Distributed databases must respond promptly to read / write requests from a large number of users and be able to perform random read / write operations on massive amounts of data.

[0006] ③ High availability: Distributed databases must provide fault tolerance mechanisms to enable redundant backups of data and ensure high reliability of data and services.

[0007] MMAP maps a file or other object into memory. The file is mapped across multiple pages. If the file size is not the sum of the page sizes, the unused space on the last page will be zeroed out. MMAP plays a significant role in user-space mapping call systems.

[0008] With the widespread use of distributed data storage, there is a lack of effective differentiated data allocation in heterogeneous computing environments. Furthermore, in MMAP scenarios, there is no guarantee that files will be compatible with MMAP.

[0009] Therefore, how to avoid imbalance in heterogeneous environments and achieve effective differentiated data allocation is a technical problem that urgently needs to be solved. Summary of the Invention

[0010] The technical objective of this invention is to provide a distributed storage method, system, device, and medium based on heterogeneous computing to address the problem of how to avoid imbalance in heterogeneous environments and achieve effective differentiated data allocation.

[0011] The technical task of this invention is achieved as follows: a distributed storage method based on heterogeneous computing. This method calculates the upper limit of file size and the weight of processing capacity according to the memory and CPU configuration of each node in the cluster, divides the data into partitions using a hash key, and then locates the specific position of the partition by the node weight. At the same time, when storing data, if the data exceeds the maximum threshold of the current file, a new branch is created using the fractal growth method of a binary tree, thereby realizing effective differentiated data allocation in a heterogeneous computing environment.

[0012] As a preferred method, the specific details are as follows:

[0013] The formula for calculating the file size threshold for the MMAP driver is as follows:

[0014] MAX_FILE_SIZE = Memory size * (1 + ln(number of CPU cores)) * X;

[0015] Where MAX_FILE_SIZE represents the maximum file size; X is a constant value defined by the percentage of resources that can be used, with a default value of 50%;

[0016] The formula for calculating node weights is as follows:

[0017] NODE_WEIGHT = Memory size * (1 / (1 + 1 / Number of CPU cores));

[0018] Where NODE_WEIGHT represents the node weight;

[0019] Data partitioning: When creating large data tables, a Partitioned Secondary Index (PSI) is used to store data in the corresponding machine nodes and file directories;

[0020] Partitioning is defined by node weight: Each machine node in the cluster has a weight calculated, and the number of partitions used by each machine is determined based on the weight.

[0021] A binary tree structure is used for file storage.

[0022] More preferably, the second partition index PSI is as follows:

[0023] Specify the column or column list of the partition, perform a hash calculation on the column or column list of the partition, obtain a 32-bit and / or 64-bit hash integer, and store the data in the corresponding machine node and file directory according to the hash integer.

[0024] Ideally, the memory and CPU configuration of each node in the cluster determines the upper limit of file size, while the memory and CPU configuration of each node in the cluster determines the cluster data allocation weight.

[0025] More specifically, the method for fractal growth of binary trees is as follows:

[0026] One bucket corresponds to one physical file. Check if the size of the physical file exceeds the maximum file size threshold:

[0027] If the size of the physical file exceeds the threshold maximum file size (MAX_FILE_SIZE), a subdirectory is created. The subdirectory has a binary structure defined by NEXT_MASK_BIT, and new data is added to the subdirectory. Here, NEXT_MASK_BIT refers to the previous bit of the current mask.

[0028] A distributed storage system based on heterogeneous computing, the system comprising:

[0029] The threshold acquisition module is used to calculate the file size threshold for the MMAP driver, using the following formula:

[0030] MAX_FILE_SIZE = Memory size * (1 + ln(number of CPU cores)) * X;

[0031] Where MAX_FILE_SIZE represents the maximum file size; X is a constant value defined by the percentage of resources that can be used, with a default value of 50%;

[0032] The weight acquisition module is used to calculate node weights, using the following formula:

[0033] NODE_WEIGHT = Memory size * (1 / (1 + 1 / Number of CPU cores));

[0034] Where NODE_WEIGHT represents the node weight;

[0035] The data partitioning module is used to store data to the corresponding machine nodes and file directories using a Partitioned Secondary Index (PSI) when creating large data tables.

[0036] The weighted partitioning module is used to calculate the weight of each machine node in the cluster and determine the number of partitions used for each machine based on the weight.

[0037] The storage module is used for file storage using a binary tree structure.

[0038] As a preferred option, the second partition index PSI is as follows:

[0039] Specify the column or column list of the partition, perform a hash calculation on the column or column list of the partition, obtain a 32-bit and / or 64-bit hash integer, and store the data in the corresponding machine node and file directory according to the hash integer.

[0040] As a preferred method, the fractal growth of the binary tree structure is as follows:

[0041] One bucket corresponds to one physical file. Check if the size of the physical file exceeds the maximum file size threshold:

[0042] If the size of the physical file exceeds the threshold maximum file size (MAX_FILE_SIZE), a subdirectory is created. The subdirectory has a binary structure defined by NEXT_MASK_BIT, and new data is added to the subdirectory. Here, NEXT_MASK_BIT refers to the previous bit of the current mask.

[0043] An electronic device includes: a memory and at least one processor;

[0044] The memory contains computer programs;

[0045] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the distributed storage method based on heterogeneous computing as described above.

[0046] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the distributed storage method based on heterogeneous computing as described above.

[0047] The distributed storage method, system, device, and medium based on heterogeneous computing of the present invention have the following advantages:

[0048] (i) This invention takes into account the available resources of the machine and solves the imbalance problem in heterogeneous environments; at the same time, it ensures that the performance of MMAP can be fully utilized.

[0049] (ii) In a heterogeneous computing environment, the present invention can effectively differentiate data allocation while ensuring file adaptation to MMAP;

[0050] (III) This invention fully utilizes computing resources in a heterogeneous environment to achieve high-performance ultra-large data processing;

[0051] (iv) This invention supports ultra-large data tables, ensuring balanced response time in heterogeneous environments, and guaranteeing memory map performance for each file;

[0052] (v) This invention enables the mixed use of new and old machine nodes, maximizing the computational potential of resource-determined weight allocation;

[0053] (vi) The performance of the super-large table of the present invention is linearly related to the machine performance and the number of nodes. Attached Figure Description

[0054] The invention will be further described below with reference to the accompanying drawings.

[0055] Appendix Figure 1 This is a flowchart of a distributed storage method based on heterogeneous computing.

[0056] Appendix Figure 2 This is a diagram illustrating the definition of partitions based on node weights. Detailed Implementation

[0057] The following detailed description of the distributed storage method, system, device, and medium based on heterogeneous computing of the present invention is provided with reference to the accompanying drawings and specific embodiments.

[0058] Example 1:

[0059] This embodiment provides a distributed storage method based on heterogeneous computing. This method calculates the upper limit of file size and the weight of processing capacity based on the memory and CPU configuration of each node in the cluster. Data is then divided into partitions using a hash key, and the specific location of each partition is determined by the node weight. Simultaneously, when storing data, if the data exceeds the maximum threshold of the current file, a new branch is created using a binary tree fractal growth method, achieving effective differentiated data allocation in a heterogeneous computing environment. (See attached...) Figure 1 As shown, the details are as follows:

[0060] S1. Calculate the file size threshold for the MMAP driver using the following formula:

[0061] MAX_FILE_SIZE = Memory size * (1 + ln(number of CPU cores)) * X;

[0062] Where MAX_FILE_SIZE represents the maximum file size; X is a constant value defined by the percentage of resources that can be used, with a default value of 50%;

[0063] S2. Calculate the node weights using the following formula:

[0064] NODE_WEIGHT = Memory size * (1 / (1 + 1 / Number of CPU cores));

[0065] Where NODE_WEIGHT represents the node weight;

[0066] S3. Data Partitioning: When creating large data tables, the Partitioned Secondary Index (PSI) is used to store data in the corresponding machine nodes and file directories.

[0067] S4. Define partitions by node weight: Each machine node in the cluster has a calculated weight, and the number of partitions used by each machine is determined based on the weight; see attached. Figure 2 As shown, the cluster has 3 machines with weights W0, W1, and W2. The ratio of the number of partitions on the three machines is W0:W1:W2.

[0068] S5. Use a binary tree structure for file storage.

[0069] The second partition index PSI in step S3 of this embodiment is as follows:

[0070] Specify the column or column list of the partition, perform a hash calculation on the column or column list of the partition, obtain a 32-bit and / or 64-bit hash integer, and store the data in the corresponding machine node and file directory according to the hash integer.

[0071] The specific implementation can be described as follows.

[0072] CREATE TABLE table_name(…,secondary_index_column,…);

[0073] PARTITION BY secondary_index_column;

[0074] (HASHBUCKET(secondary_index_column)MASK N).

[0075] HASHBUCKET is a hash function that returns a 32-bit / 64-bit integer. MASK defines the number of BUCKETs as power(2, x), where x is the mask bit number.

[0076] For example, MASK 8 means there are 256 buckets.

[0077] It is recommended that number_of_buckets / number_of_nodes > 100 to ensure that data is evenly distributed across machine nodes.

[0078] In this embodiment, the memory and CPU configuration of each node in the cluster determines the upper limit of file size, while the memory and CPU configuration of each node in the cluster determines the cluster data allocation weight.

[0079] In this embodiment, the method for fractal growth of binary trees is as follows:

[0080] One bucket corresponds to one physical file. Check if the size of the physical file exceeds the maximum file size threshold:

[0081] If the physical file size exceeds the maximum file size threshold (MAX_FILE_SIZE), a subdirectory is created. This subdirectory has a binary structure, defined by NEXT_MASK_BIT, and new data is added to it. NEXT_MASK_BIT refers to the bit preceding the current mask. For example, if the current mask is 8 bits, it means the last 8 bits have already been masked, and NEXT_MASK_BIT would be the 9th bit from the end. Key code is as follows:

[0082] Data Storage

[0083] ---#hk

[0084] ---#h k+1

[0085] |---0

[0086] |---1

[0087] ---#h k+2

[0088] ---#h K+3

[0089] N0de.

[0090] example

[0091] ·HASH_BUCKET#=? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 1XXXXXXX

[0092] ·XXXXXXXX->k+1

[0093] ·New row is inserted into h#k+1if file size<threshold,otherwise

[0094] ·create 2new directory,0and 1

[0095] ·New row is inserted into file under#h(k+1) / 1.

[0096] Example 2:

[0097] This embodiment provides a distributed storage system based on heterogeneous computing, the system comprising:

[0098] The threshold acquisition module is used to calculate the file size threshold for the MMAP driver, using the following formula:

[0099] MAX_FILE_SIZE = Memory size * (1 + ln(number of CPU cores)) * X;

[0100] Where MAX_FILE_SIZE represents the maximum file size; X is a constant value defined by the percentage of resources that can be used, with a default value of 50%;

[0101] The weight acquisition module is used to calculate node weights, using the following formula:

[0102] NODE_WEIGHT = Memory size * (1 / (1 + 1 / Number of CPU cores));

[0103] Where NODE_WEIGHT represents the node weight;

[0104] The data partitioning module is used to store data to the corresponding machine nodes and file directories using a Partitioned Secondary Index (PSI) when creating large data tables.

[0105] The weighted partitioning module is used to calculate the weight of each machine node in the cluster and determine the number of partitions used for each machine based on the weight.

[0106] The storage module is used for file storage using a binary tree structure.

[0107] In this embodiment, the second partition index PSI is as follows:

[0108] Specify the column or column list of the partition, perform a hash calculation on the column or column list of the partition, obtain a 32-bit and / or 64-bit hash integer, and store the data in the corresponding machine node and file directory according to the hash integer.

[0109] In this embodiment, the fractal growth of the binary tree structure is specifically as follows:

[0110] One bucket corresponds to one physical file. Check if the size of the physical file exceeds the maximum file size threshold:

[0111] If the size of the physical file exceeds the threshold maximum file size (MAX_FILE_SIZE), a subdirectory is created. The subdirectory has a binary structure defined by NEXT_MASK_BIT, and new data is added to the subdirectory. Here, NEXT_MASK_BIT refers to the previous bit of the current mask.

[0112] Example 3:

[0113] This invention also provides an electronic device, including: a memory and a processor;

[0114] The memory stores the instructions executed by the computer.

[0115] The processor executes computer execution instructions stored in the memory, causing the processor to execute the distributed storage method based on heterogeneous computing in any embodiment of the present invention.

[0116] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0117] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0118] Example 4:

[0119] This invention also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the distributed storage method based on heterogeneous computing in any embodiment of this invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0120] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0121] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0122] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0123] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed storage method based on heterogeneous computing, characterized in that, This method calculates the upper limit of file size and the weight of processing capacity based on the memory and CPU configuration of each node in the cluster. Data is then divided into partitions using a hash key, and the specific location of each partition is determined by the node weight. Simultaneously, when storing data, if the data exceeds the maximum threshold of the current file, a new branch is created using a binary tree fractal growth method, achieving effective differentiated data allocation in a heterogeneous computing environment. Specifically: The formula for calculating the file size threshold for the MMAP driver is as follows: MAX_FILE_SIZE = Memory size (1 + ln(number of CPU cores)) X; Where MAX_FILE_SIZE represents the maximum file size; X is a constant value defined by the percentage of available resources, with a default value of 50%; The formula for calculating node weights is as follows: NODE_WEIGHT = Memory size (1 / (1 + 1 / number of CPU cores)); Where NODE_WEIGHT represents the node weight; Data partitioning: When creating large data tables, a second partition index (PSI) is used to store data to the corresponding machine nodes and file directories; Partitioning is defined by node weight: Each machine node in the cluster has a weight calculated, and the number of partitions used by each machine is determined based on the weight. A binary tree structure is used for file storage.

2. The distributed storage method based on heterogeneous computing according to claim 1, characterized in that, The specific details of the second partition index PSI are as follows: Specify the column or column list of the partition, perform a hash calculation on the column or column list of the partition, obtain a 32-bit and / or 64-bit hash integer, and store the data in the corresponding machine node and file directory according to the hash integer.

3. The distributed storage method based on heterogeneous computing according to claim 1 or 2, characterized in that, The memory and CPU configuration of each node in the cluster determines the upper limit of file size, while the memory and CPU configuration of each node in the cluster determines the cluster data allocation weight.

4. The distributed storage method based on heterogeneous computing according to claim 3, characterized in that, The specific methods for fractal growth of binary trees are as follows: One bucket corresponds to one physical file. Check if the size of the physical file exceeds the maximum file size threshold: If the size of the physical file exceeds the maximum file size threshold, a subdirectory is created. The subdirectory has a binary structure defined by NEXT_MASK_BIT, and new data is added to the subdirectory. Here, NEXT_MASK_BIT refers to the previous bit of the current mask.

5. A distributed storage system based on heterogeneous computing, characterized in that, The system includes: The threshold acquisition module is used to calculate the file size threshold driven by MMAP. When storing data, if the data exceeds the maximum threshold of the current file, a new branch is created using the binary tree fractal growth method; the formula is as follows: MAX_FILE_SIZE = Memory size (1 + ln(number of CPU cores)) X; Where MAX_FILE_SIZE represents the maximum file size; X is a constant value defined by the percentage of available resources, with a default value of 50%; The weight acquisition module is used to calculate node weights, using the following formula: NODE_WEIGHT = Memory size (1 / (1 + 1 / number of CPU cores)); NODE_WEIGHT represents the node weight, which divides the data into partitions using the hash key and then locates the specific position of the partition using the node weight. The data partitioning module is used to store data to the corresponding machine nodes and file directories using the second partition index PSI when creating large data tables; The weighted partitioning module is used to calculate the weight of each machine node in the cluster and determine the number of partitions used for each machine based on the weight. The storage module is used for file storage using a binary tree structure.

6. The distributed storage system based on heterogeneous computing according to claim 5, characterized in that, The specific details of the second partition index PSI are as follows: Specify the column or column list of the partition, perform a hash calculation on the column or column list of the partition, obtain a 32-bit and / or 64-bit hash integer, and store the data in the corresponding machine node and file directory according to the hash integer.

7. The distributed storage system based on heterogeneous computing according to claim 5, characterized in that, The fractal growth of the binary tree structure is as follows: One bucket corresponds to one physical file. Check if the size of the physical file exceeds the maximum file size threshold: If the size of the physical file exceeds the maximum file size threshold, a subdirectory is created. The subdirectory has a binary structure defined by NEXT_MASK_BIT, and new data is added to the subdirectory. Here, NEXT_MASK_BIT refers to the previous bit of the current mask.

8. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the distributed storage method based on heterogeneous computing as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the distributed storage method based on heterogeneous computing as described in any one of claims 1 to 4.

Images