Data distribution and load balancing method and equipment for distributed storage system
By replacing the storage nodes of Redis Cluster with performance-layer threads and mapping read and write objects within the target hash ring, the slot allocation problem of Redis Cluster is solved, achieving thread-level load balancing and fast failover, making it suitable for high-concurrency, low-latency distributed storage systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JINAN INSPUR DATA TECH CO LTD
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-15
AI Technical Summary
Redis Cluster uses static slot allocation, which requires manual reallocation when scaling up or experiencing a failure. This makes it difficult to achieve fine-grained automatic load balancing and cannot meet the needs of high-concurrency, low-latency scenarios.
Replace storage nodes with performance layer threads, map performance layer threads through the target hash ring, pre-map read and write objects to each performance layer thread within the target hash ring, achieve fine-grained load balancing at the thread level, and automatically remap slots in case of failure.
It achieves highly automated, fine-grained load balancing, supports rapid failover and data distribution, and meets the needs of high-concurrency, low-latency scenarios.
Smart Images
Figure CN122044879A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of distributed storage technology, and in particular to a method and device for data distribution and load balancing in a distributed storage system. Background Technology
[0002] Traditional data distribution methods often trigger large-scale data migrations when nodes change, affecting system stability. Consistent hashing, through a hash ring structure, maps data and nodes onto the ring and determines ownership by clockwise search. This allows for the redistribution of adjacent data only when nodes are added or removed, significantly reducing migration overhead and thus being widely used in distributed system design.
[0003] As a typical application of consistent hashing, Redis Cluster uses fixed slots for data sharding, but its slot allocation is statically and explicitly managed. When a node fails or scales up or down, slots must be reassigned manually or through complex protocols, resulting in low automation and limited recovery efficiency. Furthermore, its management granularity remains at the physical node level, failing to achieve finer-grained thread-level load balancing and making it difficult to meet the demands for precise data distribution and rapid failover in high-concurrency, low-latency scenarios.
[0004] Given the above, how to solve the problem that Redis Cluster currently uses static slot allocation, relies on manual reallocation during expansion and failure, and is difficult to achieve fine-grained automatic load balancing is an urgent problem for technical personnel in this field. Summary of the Invention
[0005] This invention provides a method and device for data distribution and load balancing in a distributed storage system, which at least solves the problem that the current Redis Cluster uses static slot allocation, relies on manual reallocation during expansion and failure, and is difficult to achieve fine-grained automatic load balancing.
[0006] This invention provides a method for data distribution and load balancing in a distributed storage system, comprising:
[0007] When a service instance fails, the corresponding performance layer thread is identified; the service instance includes at least a storage node and the performance layer process under the storage node.
[0008] Within the target hash ring, the slot numbers corresponding to the faulty performance layer threads are mapped to the next normal performance layer threads;
[0009] The target hash ring is pre-mapped with multiple performance layer threads through a hash function, and each read / write object in the distributed storage system is pre-mapped to each performance layer thread within the target hash ring.
[0010] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described distributed storage system data distribution and load balancing methods.
[0011] The beneficial effects of this invention are as follows: Compared with the common practice of consistent hashing, this solution replaces storage nodes with performance layer threads, that is, maps performance layer threads to the target hash ring; at the same time, it pre-maps each read and write object in the system to each performance layer thread in the target hash ring, which greatly reduces the granularity of data object management and realizes fine-grained load balancing at the thread level; on this basis, when a service instance fails in the system, it is only necessary to determine the faulty performance layer thread corresponding to the service instance and map the slot number corresponding to the faulty performance layer thread to the next normal performance layer thread in the target hash ring to complete the slot remapping. There is no need to rely on manual or complex protocols to reallocate slots, which has a high degree of automation and meets the needs of distributed storage systems for fine data distribution and fast failover in high-concurrency and low-latency scenarios.
[0012] In addition, the present invention also provides an electronic device with the same effect. Attached Figure Description
[0013] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0014] Figure 1 A flowchart illustrating a data distribution and load balancing method for a distributed storage system provided in an embodiment of the present invention;
[0015] Figure 2 A schematic diagram of the target hash ring provided in an embodiment of the present invention;
[0016] Figure 3 This is a schematic diagram of a distributed storage system data distribution and load balancing device provided in an embodiment of the present invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.
[0018] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0019] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0020] Currently, Redis Cluster uses a fixed number of slots for data sharding. However, its slot allocation is static and explicitly managed. When a node fails, it needs to be manually or through complex protocols to reallocate slots, resulting in low automation. Furthermore, the management granularity remains at the physical node level, failing to achieve thread-level fine-grained management. Therefore, to address these issues, this invention provides a method for data distribution and load balancing in a distributed storage system.
[0021] It should be noted that this solution is applied to a distributed storage system, which consists of the following core components: Performance Layer (PL) threads, which act as front-end processing units responsible for receiving client requests and executing data distribution logic; each PL thread is bound to a Central Processing Unit (CPU) core; Storage nodes (Stores) that host Performance Layer Group (PLG) instances (i.e., slots), with each PLG corresponding to a local key-value (KV) storage instance; Central Control Service (CCS) that maintains the global resource view, hash ring state, and PLG mapping information; and a Performance Layer Group Monitor (PLG Monitor) responsible for monitoring PLG status, handling failover, and dynamic scaling. PLGs are the units in the storage pool that handle Business Operations (OPs), serving as an intermediary layer between the client and the database (DB). Read and write operations of business objects are assigned to different PLGs for execution. PLG instances are hosted on storage nodes, and PLGs are distributed across different storage nodes using a consistent hashing algorithm. The following is a detailed description of the data distribution and load balancing method for the distributed storage system provided by this invention:
[0022] Figure 1This is a flowchart illustrating a data distribution and load balancing method for a distributed storage system provided in an embodiment of the present invention. Figure 1 As shown, the method includes:
[0023] S10: When a service instance fails, determine the corresponding fault performance layer thread for the service instance.
[0024] First, when a service instance fails in the system, the faulty PL thread corresponding to the service instance is identified. It should be noted that since PL processes run on storage nodes, and each PL process has multiple PL threads, a service instance includes at least the storage node and the PL processes under the storage node. That is, the failure may be in the storage node or the PL processes under the storage node.
[0025] It is understandable that whether it is a storage node failure or a PL process failure under the storage node, the corresponding PL thread will be affected, so it is necessary to identify the corresponding failed PL thread.
[0026] S11: Within the target hash ring, map the slot numbers corresponding to the faulty performance layer threads to the next normal performance layer threads.
[0027] Compared to the common practice of consistent hashing, this invention replaces the storage nodes with PL threads to obtain the target hash ring. Figure 2 This is a schematic diagram of the target hash ring provided in an embodiment of the present invention. Figure 2 As shown, the target hash ring is pre-mapped with multiple PL threads through a hash function, and each read / write object in the distributed storage system is pre-mapped to each PL thread within the target hash ring. This greatly reduces the granularity of data object management and achieves thread-level fine-grained load balancing.
[0028] Based on this, when a system fault occurs and the faulty PL thread is identified, the read / write objects corresponding to the faulty PL thread need to be migrated in order to achieve fault recovery. Specifically, within the target hash ring, the slot numbers (i.e., PLG numbers) corresponding to the faulty PL thread are mapped to the next normal PL thread, thereby achieving lossless fault switching.
[0029] It should be noted that this embodiment does not restrict the specific process of constructing the target hash ring and remapping the slot numbers; it depends on the specific implementation.
[0030] In this embodiment, compared to the common practice of consistent hashing, this solution replaces storage nodes with performance layer threads, that is, maps performance layer threads to the target hash ring. At the same time, each read and write object in the system is pre-mapped to each performance layer thread in the target hash ring, which greatly reduces the granularity of data object management and achieves fine-grained load balancing at the thread level. On this basis, when a service instance fails in the system, it is only necessary to determine the faulty performance layer thread corresponding to the service instance and map the slot number corresponding to the faulty performance layer thread to the next normal performance layer thread in the target hash ring to complete the slot remapping. There is no need to rely on manual or complex protocols to reallocate slots. The degree of automation is high, which meets the needs of distributed storage systems for fine data distribution and fast failover in high-concurrency and low-latency scenarios.
[0031] Based on the above embodiments, in some embodiments, the process of constructing the target hash ring includes:
[0032] S101: When the performance layer process starts, the key-value interface is called to start the physical threads of each performance layer.
[0033] S102: Determine the listening address of each performance layer physical thread as the unique identifier of each performance layer physical thread.
[0034] S103: Add multiple virtual threads for each physical thread of the performance layer, and determine the hash value of each virtual thread of the performance layer according to each unique identifier and hash algorithm.
[0035] S104: Record each hash value in the hash ring and establish a mapping relationship between each hash value and the corresponding physical thread of the performance layer.
[0036] S105: Establish the reverse mapping relationship between the physical threads of each performance layer and the corresponding virtual threads of each performance layer.
[0037] To construct the target hash ring, when the PL process (icfs-pl) starts, it first calls the key-value interface to start each PL physical thread. Since each PL physical thread is bound to a CPU core and a fixed port, the listening address (IP+port) of each PL physical thread can be determined as the unique identifier of each PL physical thread.
[0038] Subsequently, multiple PL virtual threads are added to each physical PL thread, and the hash value of each PL virtual thread is determined based on its unique identifier and hash algorithm. For example, assuming each storage node is allocated M CPU cores and port numbers, and the number of storage nodes is N, then the system starts a total of M×N physical PL threads. To increase the scale of PL threads in the hash ring, the physical PL threads are increased by a factor of K, resulting in K×M×N virtual PL threads. The hash value of each virtual PL thread calculated by the hash algorithm is as follows: Hash(IP+port+1), Hash(IP+port+2), ..., Hash(IP+port+K).
[0039] Furthermore, each hash value is recorded in a hash ring, and a mapping relationship between each hash value and the corresponding PL physical thread is established.
[0040] Table 1 Hash Ring Data Structure Table
[0041]
[0042] As shown in Table 1, each hash value is specifically recorded in the hash ring's storage structure std::set.<uint64_t> In, and based on std::unordered_map<uint64_t,PL> Store the mapping relationship between each hash value (i.e., PL virtual thread) and the corresponding PL physical thread. Finally, based on std::unordered_map <std::string,std::unordered_set<uint64_t> The system manages the inverse mapping relationship between each PL physical thread and its corresponding PL virtual thread. By establishing a target hash ring through a virtual node mechanism, the uniformity of data distribution is significantly improved, avoiding hotspot issues.
[0043] Furthermore, after the target hash ring is established, the amount of data and request traffic carried by each storage node and PL virtual thread can be periodically checked. By analyzing indicators, it can be identified whether there are load skew or hot data issues. Further, it is necessary to sample and verify whether the routing results after hash calculation of data keys are correct, ensuring that the mapping relationship is not disordered. Finally, based on the imbalance obtained from monitoring, data rebalancing can be triggered by dynamically adjusting the number of PL virtual threads or manually migrating slots. Through the above process, it is possible to ensure that data is always evenly distributed on the ring, which not only allows for the rapid detection and isolation of faulty nodes and minimizes service interruptions, but also avoids single-point overload through load balancing, thereby improving overall throughput and reducing access latency, laying the foundation for stable expansion of the data layer.
[0044] Based on the above embodiments, in some embodiments, the read / write object is mapped to a performance layer thread within the target hash ring, including:
[0045] S111: When there is a read / write operation on the distributed storage system, determine the read / write object corresponding to the read / write operation and determine the target slot number corresponding to the read / write object.
[0046] S112: In the target hash ring, map the target slot number to the corresponding target performance layer virtual thread, and determine the corresponding target performance layer physical thread and its listening address.
[0047] To map read / write objects to PL threads within the target hash ring, this embodiment performs two mappings: mapping the read / write object to the slot, and mapping the slot to the target hash ring via the PL thread. Specifically, when a read / write operation occurs on the distributed storage system, the read / write object corresponding to the operation is determined, and the target slot number corresponding to the read / write object is also determined. Subsequently, within the target hash ring, the target slot number is mapped to the corresponding target PL virtual thread, and the corresponding target PL physical thread and its listening address (i.e., IP + port) are determined, thus completing the addressing. The two mappings are explained in detail below:
[0048] (1) Mapping of read / write objects to slots;
[0049] In practical implementation, after determining the read / write objects, the key-value pairs of these objects are first determined, followed by the total number of slots in the distributed storage system. It can be understood that a slot (i.e., a PLG) is used to carry the data distribution of the storage pool; one PLG corresponds to one embedded key-value store database (RocksDB) instance at the KV layer. When creating a PL pool, the number of PLGs can be set to 2. L When L=12, 4096 PLGs will be created.
[0050] Further, a hash function is performed on the key to obtain the hash value corresponding to the key; finally, the target slot number is obtained by taking the modulo of the total number of slots based on the hash value corresponding to the key.
[0051] (2) The slot is mapped to the target hash ring via the PL thread;
[0052] In practical implementation, based on an initial hash ring, the total number of slots is uniformly selected starting from 0 (2). L If there are 1, 2 slots, then the hash value corresponding to slot P on the hash ring is P×2. (64-L)Based on this, to determine the specific mapping position of the target slot number on the hash ring, after determining the target slot number, it is necessary to determine the hash value corresponding to the target slot number. Then, based on the hash value corresponding to the target slot number, the next active PL virtual thread is searched clockwise in the target hash ring. Finally, the next active PL virtual thread is taken as the target PL virtual thread, and the mapping from the target slot number to the target PL virtual thread is executed, thereby completing the addressing.
[0053] In summary, this solution provides a fast mapping mechanism based on slot hashing, supporting large-scale PLG management.
[0054] Based on the above embodiments, in some embodiments, determining the fault performance layer thread corresponding to the service instance includes:
[0055] S121: Determine the physical thread of the fault performance layer corresponding to the service instance.
[0056] S122: Based on the reverse mapping relationship, determine each fault performance layer virtual thread under the physical thread of the fault performance layer, and mark each fault performance layer virtual thread as unavailable.
[0057] To determine the faulty PL thread corresponding to a service instance, this embodiment specifically identifies the corresponding faulty PL physical thread based on the actual faulty storage node or faulty PL process. Subsequently, according to the pre-established inverse mapping relationship between PL physical threads and their corresponding PL virtual threads, each faulty PL virtual thread under the faulty PL physical thread is determined, and each faulty PL virtual thread is marked as unavailable (DOWN). This achieves precise locking of the faulty PL thread, facilitating re-addressing of each PLG under the faulty PL virtual thread.
[0058] Based on the above embodiments, in some embodiments, within the target hash ring, the slot numbers corresponding to the faulty performance layer threads are mapped to the next normal performance layer threads, including:
[0059] S131: Based on the hash value of the faulty performance layer virtual thread, search clockwise in the target hash ring for the next available performance layer virtual thread.
[0060] S132: Select the next available performance layer virtual thread as the next normal performance layer thread, and perform the mapping of slot number to the next normal performance layer thread.
[0061] After identifying the faulty PL virtual thread, the next PL virtual thread in an UP state is searched clockwise within the target hash ring based on its hash value. PL virtual threads in a DOWN state are skipped during this process. Finally, the next UP PL virtual thread is designated as the next normal PL thread, and the mapping of that slot number to the next normal PL thread is executed. It's important to note that if a PLG's thread mapped before the fault did not fail, it will remain mapped to the original thread and will not be migrated.
[0062] In this embodiment, the PLG is remapped based on the hash value of the faulty PL virtual thread, thus achieving lossless fault switching.
[0063] To achieve online capacity expansion, based on the above embodiments, in some embodiments, the method further includes:
[0064] S141: Perform a resizing operation on the target hash ring.
[0065] S142: Based on the expanded target hash ring, re-establish the mapping relationship between each slot number and each performance layer virtual thread.
[0066] Specifically, after the target hash ring is constructed, a further expansion operation can be performed on the target hash ring. Based on the expanded target hash ring, the mapping relationship between each slot number and each PL virtual thread is re-established. After the expansion is completed, the PLG will be re-addressed, and some PLGs will be assigned to new PL virtual threads.
[0067] It's important to note that there are two different methods for expanding the target hash ring: one is to add a new physical PL thread based on the PL process, and then add a corresponding new virtual PL thread to the new physical PL thread; the new virtual PL thread is then added to the target hash ring and marked as UP. The other method is to add a new virtual PL thread based on an existing physical PL thread, add the new virtual PL thread to the target hash ring, and mark the new virtual PL thread as UP. In practice, either method can be chosen, or both can be used simultaneously.
[0068] In this way, storage expansion was achieved through PL virtual nodes, enabling highly flexible and fine-grained load balancing.
[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0070] Figure 3This is a schematic diagram of a data distribution and load balancing device for a distributed storage system provided in an embodiment of the present invention. Figure 3 As shown, the device includes:
[0071] The determination module 11 is used to determine the faulty performance layer thread corresponding to the service instance when a service instance failure occurs; wherein, the service instance includes at least a storage node and the performance layer process under the storage node.
[0072] Mapping module 12 is used to map the slot numbers corresponding to the faulty performance layer threads to the next normal performance layer threads within the target hash ring.
[0073] The target hash ring is pre-mapped with multiple performance layer threads through a hash function, and each read / write object in the distributed storage system is pre-mapped to each performance layer thread within the target hash ring.
[0074] In some embodiments, the process of constructing the target hash ring includes: when the performance layer process starts, calling the key-value interface to start each performance layer physical thread; determining the listening address of each performance layer physical thread as the unique identifier of each performance layer physical thread; adding multiple performance layer virtual threads for each performance layer physical thread, and determining the hash value of each corresponding performance layer virtual thread according to each unique identifier and hash algorithm; recording each hash value in the hash ring, and establishing a mapping relationship between each hash value and the corresponding performance layer physical thread; and establishing a reverse mapping relationship between each performance layer physical thread and the corresponding performance layer virtual thread.
[0075] In some embodiments, mapping read / write objects to performance layer threads within the target hash ring includes: when there is a read / write operation on the distributed storage system, determining the read / write object corresponding to the read / write operation and determining the target slot number corresponding to the read / write object; in the target hash ring, mapping the target slot number to the corresponding target performance layer virtual thread and determining the corresponding target performance layer physical thread and its listening address.
[0076] In some embodiments, determining the target slot number corresponding to the read / write object includes: determining the key value of the read / write object and determining the total number of slots in the distributed storage system; performing a hash function calculation on the key value to obtain the hash value corresponding to the key value; and taking the modulo of the total number of slots based on the hash value corresponding to the key value to obtain the target slot number.
[0077] In some embodiments, mapping a target slot number to a corresponding target performance layer virtual thread in the target hash ring includes: determining the hash value corresponding to the target slot number; based on the hash value corresponding to the target slot number, searching clockwise for the next active performance layer virtual thread in the target hash ring; and using the next active performance layer virtual thread as the target performance layer virtual thread, and performing the mapping of the target slot number to the target performance layer virtual thread.
[0078] In some embodiments, the determining module 11 includes:
[0079] The first determination submodule is used to determine the physical thread of the fault performance layer corresponding to the service instance;
[0080] The second determination submodule is used to determine each fault performance layer virtual thread under the fault performance layer physical thread according to the reverse mapping relationship, and mark each fault performance layer virtual thread as unavailable.
[0081] In some embodiments, the mapping module 12 includes:
[0082] The lookup module is used to search clockwise in the target hash ring for the next available performance layer virtual thread based on the hash value of the failed performance layer virtual thread.
[0083] The execution module is used to select the next available performance layer virtual thread as the next normal performance layer thread and to map the slot number to the next normal performance layer thread.
[0084] In some embodiments, it also includes:
[0085] The expansion module is used to perform expansion operations on the target hash ring;
[0086] The remapping module is used to re-establish the mapping relationship between each slot number and each performance layer virtual thread based on the expanded target hash ring.
[0087] In some embodiments, the expansion module specifically adds new performance layer physical threads based on the performance layer process and adds corresponding new performance layer virtual threads for the new performance layer physical threads; adds the new performance layer virtual threads to the target hash ring and marks the new performance layer virtual threads as available; or, adds new performance layer virtual threads based on existing performance layer physical threads, adds the new performance layer virtual threads to the target hash ring and marks the new performance layer virtual threads as available.
[0088] For a description of the features of the distributed storage system data distribution and load balancing device in the corresponding embodiment, please refer to the relevant description of the distributed storage system data distribution and load balancing method in the corresponding embodiment, which will not be repeated here.
[0089] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the distributed storage system data distribution and load balancing method.
[0090] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the distributed storage system data distribution and load balancing method when running.
[0091] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0092] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the distributed storage system data distribution and load balancing method.
[0093] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the distributed storage system data distribution and load balancing method.
[0094] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0095] The above provides a detailed description of a data distribution and load balancing method and device for a distributed storage system provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of the embodiments above are only intended to help understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of the present invention.
Claims
1. A method for data distribution and load balancing in a distributed storage system, characterized in that, include: When a service instance fails, the corresponding performance layer thread for the service instance is determined; wherein, the service instance includes at least a storage node and a performance layer process under the storage node; Within the target hash ring, the slot numbers corresponding to the faulty performance layer threads are mapped to the next normal performance layer threads; The target hash ring is pre-mapped with multiple performance layer threads via hash functions, and each read / write object in the distributed storage system is pre-mapped to each of the performance layer threads within the target hash ring.
2. The data distribution and load balancing method for a distributed storage system according to claim 1, characterized in that, The process of constructing the target hash ring includes: When the performance layer process starts, the key-value interface is called to start the physical threads of each performance layer; The listening address of each physical thread in the performance layer is determined as the unique identifier of each physical thread in the performance layer; Multiple performance layer virtual threads are added to each of the performance layer physical threads, and the hash value of each performance layer virtual thread is determined according to the unique identifier and hash algorithm. Each hash value is recorded in a hash ring, and a mapping relationship is established between each hash value and the corresponding physical thread of the performance layer. Establish reverse mapping relationships between each physical thread of the performance layer and the corresponding virtual thread of the performance layer.
3. The data distribution and load balancing method for a distributed storage system according to claim 2, characterized in that, Mapping the read / write object to the performance layer thread within the target hash ring includes: When there is a read / write operation on the distributed storage system, determine the read / write object corresponding to the read / write operation, and determine the target slot number corresponding to the read / write object; In the target hash ring, the target slot number is mapped to the corresponding target performance layer virtual thread, and the corresponding target performance layer physical thread and its listening address are determined.
4. The data distribution and load balancing method for a distributed storage system according to claim 3, characterized in that, Determining the target slot number corresponding to the read / write object includes: Determine the key value of the read / write object and the total number of slots in the distributed storage system; Perform a hash function calculation on the key value to obtain the hash value corresponding to the key value; The target slot number is obtained by taking the modulo of the total number of slots based on the hash value corresponding to the key value.
5. The data distribution and load balancing method for a distributed storage system according to claim 3, characterized in that, In the target hash ring, mapping the target slot number to the corresponding target performance layer virtual thread includes: Determine the hash value corresponding to the target slot number; Based on the hash value corresponding to the target slot number, the next active performance layer virtual thread is searched clockwise in the target hash ring. The next active performance layer virtual thread is used as the target performance layer virtual thread, and the mapping of the target slot number to the target performance layer virtual thread is executed.
6. The data distribution and load balancing method for a distributed storage system according to claim 2, characterized in that, Determining the fault performance layer thread corresponding to the service instance includes: Determine the physical thread of the fault performance layer corresponding to the service instance; Based on the reverse mapping relationship, each fault performance layer virtual thread under the physical thread of the fault performance layer is determined, and each fault performance layer virtual thread is marked as unavailable.
7. The data distribution and load balancing method for a distributed storage system according to claim 6, characterized in that, Within the target hash ring, the slot numbers corresponding to the faulty performance layer threads are mapped to the next normal performance layer threads, including: Based on the hash value of the faulty performance layer virtual thread, the next available performance layer virtual thread is searched clockwise in the target hash ring. The next available performance layer virtual thread is used as the next normal performance layer thread, and the mapping of the slot number to the next normal performance layer thread is executed.
8. The data distribution and load balancing method for a distributed storage system according to any one of claims 1 to 7, characterized in that, Also includes: Perform a resizing operation on the target hash ring; Based on the expanded target hash ring, the mapping relationship between each slot number and each performance layer virtual thread is re-established.
9. The data distribution and load balancing method for a distributed storage system according to claim 8, characterized in that, Performing a resizing operation on the target hash ring includes: A new performance layer physical thread is added based on the performance layer process, and a corresponding new performance layer virtual thread is added for the new performance layer physical thread; the new performance layer virtual thread is added to the target hash ring, and the new performance layer virtual thread is marked as available. Alternatively, a new performance layer virtual thread can be added based on the existing performance layer physical thread, the new performance layer virtual thread can be added to the target hash ring, and the new performance layer virtual thread can be marked as available.
10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the distributed storage system data distribution and load balancing method as described in any one of claims 1 to 9.