Data storage method, electronic device, storage medium, and program product

Abstract

The present invention provides a data storage method, an electronic device, a storage medium, and a program product, which can be applied to the field of storage technology. The method includes: determining distances between the data to be stored and a plurality of storage nodes according to the data coordinates of the data to be stored in a preset space and the node centroid coordinates of the plurality of storage nodes, to obtain a plurality of distances; wherein each of the plurality of storage nodes has a node area in the preset space, and the node centroid coordinates represent the coordinates of the node area centroid of the node area in the preset space; and determining a target node for storing the data to be stored from the plurality of storage nodes according to the plurality of distances.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and more specifically to a data storage method, electronic equipment, storage medium and program product. Background Art

[0002] In a distributed storage system, data is usually scattered across multiple physical nodes, but due to differences in hardware resources, data distribution is often uneven. This imbalance can cause excessive storage pressure on some nodes, while other nodes have idle resources, which in turn affects overall system performance and restricts real-time response capabilities in large-scale data scenarios. Therefore, how to optimize data distribution, design efficient routing mechanisms, and improve processing efficiency have become key challenges that need to be addressed in current distributed storage technology. Summary of the invention

[0003] In view of the above problems, the present invention provides a data storage method, an electronic device, a storage medium and a program product.

[0004] According to one aspect of the present invention, there is provided a data storage method, comprising: determining the distance between the data to be stored and a plurality of storage nodes based on the data coordinates of the data to be stored in a preset space and the node centroid coordinates of the plurality of storage nodes, to obtain a plurality of distances; wherein the plurality of storage nodes each have a node area in the preset space, and the node centroid coordinates represent the coordinates of the centroid of the node area of the node area in the preset space; and determining a target node for storing the data to be stored from the plurality of storage nodes based on the plurality of distances.

[0005] Another aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the data storage method of the present invention.

[0006] Another aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the data storage method of the present invention are implemented.

[0007] Another aspect of the present invention further provides a computer program product, including a computer program or instructions, which implement the steps of the data storage method of the present invention when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0009] Figure 1Shows an application scenario diagram of the data storage method according to an embodiment of the present invention;

[0010] Figure 2 Shows a flowchart of the data storage method according to an embodiment of the present invention;

[0011] Figure 3A Shows an operation diagram of the data distribution method according to an embodiment of the present invention;

[0012] Figure 3B Shows an operation diagram of the hyperbolic coordinate generation method according to an embodiment of the present invention;

[0013] Figure 3C Shows an operation diagram of the elastic space partitioning method according to an embodiment of the present invention;

[0014] Figure 3D Shows an operation diagram of the new node process according to an embodiment of the present invention;

[0015] Figure 4 Shows a schematic diagram of the cross - level replica strategy according to an embodiment of the present invention;

[0016] Figure 5 Shows a schematic diagram of the system architecture of the data storage system and its core components according to an embodiment of the present invention;

[0017] Figure 6 Shows a structural block diagram of the data storage device according to an embodiment of the present invention;

[0018] Figure 7 Shows a block diagram of an electronic device suitable for implementing the data storage method according to an embodiment of the present invention. Detailed implementation manners

[0019] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a thorough understanding of the embodiments of the present invention. However, obviously, one or more embodiments can be implemented without these specific details. In addition, in the following description, descriptions of well - known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present invention.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0021] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0022] In cases where expressions similar to "at least one of A, B, and C, etc." are used, generally, it should be interpreted according to the meaning that those of ordinary skill in the art usually understand such expressions (for example, "a system having at least one of A, B, and C" should include, but not be limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0023] Distributed storage systems such as HDFS (Hadoop Distributed File System), Ceph (an open-source distributed storage system), GlusterFS (an open-source distributed file system), etc. have significant deficiencies in the following aspects:

[0024] Uneven data distribution: Static hashing algorithms such as consistent hashing are prone to data skew when nodes are dynamically expanded, resulting in overloaded load on some nodes while other nodes' resources are idle. For example, the fixed block allocation strategy of HDFS cannot adapt to the capacity differences of heterogeneous storage nodes, and the data skew rate is as high as 15% - 30%.

[0025] Low load balancing efficiency: Traditional load balancing algorithms such as round-robin and random allocation are difficult to dynamically balance multi-dimensional resource metrics such as storage capacity, network bandwidth, and Central Processing Unit (CPU) utilization in high-concurrency scenarios. The Controlled Replication Under Scalable Hashing (CRUSH) algorithm in Ceph allocates data through pseudo-random mapping. Although it supports weight adjustment, it lacks optimization for locality awareness, and the cross-region access latency increases by 30% - 50%.

[0026] High fault tolerance recovery latency: The replica distribution of the multi-replica strategy lacks spatial optimization. When a fault occurs, data needs to be retrieved across regions, resulting in an extended recovery time. For example, the replica strategy of GlusterFS stores data in a fixed replica group, relies on full-data replication, depends on asynchronous repair, has low recovery efficiency, and consumes a huge amount of bandwidth resources.

[0027] Embodiments of the present invention provide a data storage method, an electronic device, a storage medium, and a program product, including: determining distances between the data to be stored and multiple storage nodes according to the data coordinates of the data to be stored in a preset space and the node centroid coordinates of the multiple storage nodes, to obtain multiple distances; wherein each of the multiple storage nodes has a node area in the preset space, and the node centroid coordinates represent the coordinates of the node area centroid of the node area in the preset space; and determining a target node for storing the data to be stored from the multiple storage nodes according to the multiple distances.

[0028] Figure 1 FIG. shows an application scenario diagram of the data storage method according to an embodiment of the present invention.

[0029] As Figure 1 shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0030] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0031] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0032] The server 105 may be a server providing various services, such as a background management server (only as an example) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process data such as received user requests, and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0033] It should be noted that the data storage method provided by the embodiments of the present invention can generally be executed by the server 105. Correspondingly, the data storage device provided by the embodiments of the present invention can generally be set in the server 105. The data storage method provided by the embodiments of the present invention can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the data storage device provided by the embodiments of the present invention can also be set in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0034] It should be understood that Figure 1 the numbers of the terminal devices, networks, and servers in

[0035] are merely illustrative. According to the implementation requirements, there can be any number of terminal devices, networks, and servers. Figure 1 The following will be based on Figures 2 to 5 the scenario described below, and will describe in detail the data storage method of the disclosed embodiments through

[0036] Figure 2 FIG. shows a flowchart of the data storage method according to an embodiment of the present invention.

[0037] As Figure 2 shown, the method includes operations S210 to S220.

[0038] In operation S210, according to the data coordinates of the data to be stored in the preset space and the node centroid coordinates of multiple storage nodes, determine the distances between the data to be stored and the multiple storage nodes, and obtain a plurality of distances; wherein, each of the multiple storage nodes has a node area in the preset space, and the node centroid coordinates represent the coordinates of the node area centroid of the node area in the preset space.

[0039] According to an embodiment of the present invention, the data to be stored can refer to new data, or can refer to the data in the existing data that needs to be migrated due to problems such as load balancing, etc., which is not limited herein. The preset space can represent a data space. In this data space, since all information is presented in the form of data, it can be expanded and arranged freely without being restricted by the physical space. The preset space can be a space constructed using a hyperbolic geometry model such as the Poincaré disk model, the Klein model, the upper half-plane model, etc. The hyperbolic geometry model is a mathematical framework for describing non-Euclidean geometry (negating Euclid's fifth postulate), and its core feature is that the space curvature is negative, allowing there to be an infinite number of parallel lines passing through a point outside a straight line.

[0040] For each data to be stored and each storage node, they can be mapped to a preset space and represented in the form of data coordinates. For example, the data to be stored can be represented by data coordinates in the preset space. The storage node can be represented by the node centroid coordinates in the preset space. The node area can characterize the area occupied by the data coordinates of the data managed by the storage nodes within the node area in the preset space.

[0041] After determining the data coordinates of the data to be stored and the node centroid coordinates of each storage node, the distances between the data to be stored and each storage node in the preset space can be determined according to the data coordinates and the node centroid coordinates of each node.

[0042] In operation S220, according to multiple distances, a target node for storing the data to be stored is determined from multiple storage nodes.

[0043] For this operation, after obtaining multiple distances characterizing the distances between the data to be stored and multiple storage nodes, storage nodes satisfying a preset distance range can be determined from the multiple storage nodes as the target nodes for storing the data to be stored. The preset distance range can be custom-set according to service requirements and is not limited herein.

[0044] Through the above embodiments of the present invention, by mapping both data and nodes to a preset space in the form of coordinates and combining distances to determine a data storage scheme, the physical space limitation can be broken, the geometric characteristics of the preset space can be efficiently utilized, the data distribution can be optimized, and relatively high routing and processing efficiency can be exhibited.

[0045] The inventor found in the process of implementing the inventive concept that the data distribution mechanism of traditional distributed storage systems relies on static hashing or range sharding strategies, resulting in uneven data distribution and insufficient scalability. For example, when the consistent hashing algorithm expands nodes, full-scale data migration is required, leading to a data skew rate as high as 15% - 30%.

[0046] According to an embodiment of the present invention, before performing the above operation S210, the data distribution of data and storage nodes in the preset space can be constructed first for calculating distances.

[0047] For example, the preset space can adopt a hyperbolic geometry model.

[0048] Figure 3A An operation diagram of a data distribution method according to an embodiment of the present invention is shown.

[0049] As Figure 3A shown, the data distribution method 300 includes operations S310 - S320.

[0050] In operation S310, hyperbolic coordinate generation.

[0051] This operation includes generating data coordinates for data and node coordinates for storage nodes.

[0052] In operation S320, elastic space partitioning.

[0053] This operation mainly divides the storage nodes into node regions in a preset space.

[0054] For the above operation S310, in combination with the hyperbolic coordinate generation algorithm, the following method can be used to calculate data coordinates: perform a hash calculation on the data identifier of the data to be stored to obtain a coordinate factor representing the position information of the data to be stored in the preset space and an angle factor representing the distribution direction of the data to be stored in the preset space. Determine the data coordinates based on the coordinate factor and the angle factor.

[0055] According to an embodiment of the present invention, the data identifier can represent identification information for uniquely determining data. For example, in a block storage scenario, the data identifier can be used as a block identifier (Identifier, ID), which is generated from the unique identifier after file fragmentation. In an object storage scenario, the data identifier can be used as an object key, which is the unique key value specified by the user when uploading an object and is mapped and generated through a hash algorithm. In a file storage scenario, the data identifier can use the combined hash value of the file path + fragmentation offset to ensure the unique coordinates of the file fragments in the preset space.

[0056] Figure 3B An operation diagram of the hyperbolic coordinate generation method according to an embodiment of the present invention is shown.

[0057] As Figure 3B shown, the hyperbolic coordinate generation S310 includes operations S311 to S312.

[0058] In operation S311, hash calculation.

[0059] This operation can convert the data identifier k into a coordinate factor and an angle factor through an improved hash function.

[0060] In operation S312, coordinate transformation.

[0061] This operation can perform a coordinate transformation on the coordinate factor and the angle factor to generate hyperbolic coordinates.

[0062] According to an embodiment of the present invention, the Poincaré disk model is an important representation form of the hyperbolic geometry model and has the following characteristics:

[0063] Exponential scalability: The distance in the hyperbolic space grows exponentially, which is naturally suitable for hierarchical data distribution, such as a tree structure or a hierarchical network topology.

[0064] Locality Preservation: In the Poincaré disk, data of adjacent nodes can be mapped to nearby regions, reducing the cross-node access overhead.

[0065] Dynamic Partitioning Ability: By adjusting the disk radius or partitioning strategy, it can flexibly adapt to the expansion requirements of the storage cluster.

[0066] In the case where the preset space includes the Poincaré disk, for the above operation S311, the disk radius of the Poincaré disk can be modulo by the first hash value of the data identifier to obtain a coordinate factor. The second hash value of the data identifier is modulo by 2π to obtain an angle factor.

[0067] For example, the hyperbolic hash algorithm can be constructed by combining the following formula (1) and formula (2) to implement hash calculation.

[0068] d = ln (1 + SHA-256 (k) mod R) (1)

[0069] θ = SHA-1 (k) mod 2π(2)

[0070] For formula (1), first, the Secure Hash Algorithm 256-bit (SHA-256) can be used to perform a hash operation on the data identifier k to generate a hash value of a fixed length. Then, the hash value is modulo by R to obtain an integer between 0 and R - 1. Here, R is the dynamically adjusted disk radius, and the initial value is, for example, 10. Finally, the natural logarithm can be taken after adding 1 to this integer, and the result is mapped to the parameter d. Through this process of hashing and mathematical transformation, the data identifier k is converted into the coordinate factor required for subsequent coordinate calculations.

[0071] For formula (2), first, the Secure Hash Algorithm 1 (SHA-1) with a fixed 160 bits can be used to perform a hash operation on the data identifier k. Secondly, the hash result is modulo by 2π to obtain an angle factor θ between 0 and 2π. This angle factor is used for subsequent coordinate conversion and determines the distribution direction of the data in the Poincaré disk space.

[0072] The core logic of formula (1) and formula (2) is that regardless of the storage type, the data identifier k can be converted into the parameters (d, θ) required for hyperbolic coordinates through SHA-256 / SHA-1, realizing a unified spatial mapping.

[0073] It should be noted that SHA-256 / SHA-1 is only an example. In actual applications, other hash algorithms can also be selected for hash calculation, which is not limited here.

[0074] In the case of obtaining the angular factor and the coordinate factor based on the above method, for the above operation S312, the exponential calculation with the natural constant as the base can be performed on the coordinate factor to obtain the exponential transformation result of the coordinate factor. According to the distribution direction of the coordinate points represented by the angular factor in the Poincaré disk, and the distance from the coordinate points determined based on the exponential transformation result to the center of the Poincaré disk, the data coordinates are determined.

[0075] For example, the following formula (3) can be constructed to realize the mapping from the hyperbolic space to the Poincaré disk.

[0076] (3)

[0077] Formula (3) performs a transformation on the coordinate factor d through exponential operation to map the "distance" in the hyperbolic space into the Poincaré disk. Control the distance from the coordinate point to the center of the disk. When d = 0, this value is 0, corresponding to the center of the Poincaré disk; as d increases, the value approaches 1, and the coordinate point approaches the edge of the Poincaré disk, reflecting the boundary characteristics of the hyperbolic space. cos θ and sin θ: Determine the direction of the coordinate point in the disk through the angular factor θ, ensuring a uniform distribution of the angles of the points on the plane of the Poincaré disk.

[0078] By combining the coordinate factor and the angular factor generated by hashing, the data identifier is mapped to the coordinates (x, y) in the Poincaré disk. The combination of exponential operation and trigonometric function makes the coordinate points evenly distributed in the disk space, reduces the probability of hash collision, and at the same time satisfies the spatial characteristics of hyperbolic geometry, providing a geometric basis for subsequent data distribution and processing.

[0079] For the above operation S320, the node regions can be divided by combining the elastic space partitioning strategy.

[0080] Figure 3C The operation diagram of the elastic space partitioning method according to an embodiment of the present invention is shown.

[0081] As Figure 3C shown, the elastic space partitioning S320 includes operations S321~S323.

[0082] In operation S321, node weight calculation.

[0083] This operation is used to calculate the node weight factors of each storage node.

[0084] In operation S322, Voronoi diagram generation.

[0085] This operation can divide the Poincaré disk space using a weight-aware Voronoi diagram (Voronoi Diagram).

[0086] In operation S323, a dynamic adjustment mechanism.

[0087] This operation is used to optimize the Voronoi diagram to obtain node regions.

[0088] For the above operation S321, the node capacity factor of each storage node can be determined based on the storage capacity of each storage node and the storage capacities of multiple storage nodes. The node bandwidth factor of each storage node can be determined based on the network bandwidth of each storage node and the network bandwidths of multiple storage nodes. The node weight factor is obtained by performing a weighted calculation on the node storage capacity factor and the node network bandwidth factor.

[0089] For example, based on formula (4) below, the node weight factor can be dynamically calculated from its storage capacity and network bandwidth dynamically.

[0090] (4)

[0091] In formula (4), represents the node weight factor of the th storage node. The higher the value, the stronger the resource-bearing capacity of the storage node in the system. is the storage capacity of the th storage node, reflecting the storage ability of the node. is the maximum value of the storage capacities of all nodes in the cluster to which the storage node belongs. By normalizing the storage capacity, can be mapped to the interval [0, 1] to eliminate the influence of the storage capacity magnitude differences among different nodes. is the network bandwidth of the th storage node, reflecting the data transmission ability of the storage node. is the maximum value of the network bandwidths of all nodes in the cluster to which the storage node belongs. Similarly, normalizes the network bandwidth.

[0092] Weight coefficient design logic 0.6 and 0.4: These two coefficients are the weight ratios of storage capacity and network bandwidth. Among them, 0.6 indicates that the storage capacity dominates in the calculation of the node weight factor, and 0.4 reflects the secondary influence of the network bandwidth. This allocation can be adjusted according to system requirements. For example, in a storage-intensive scenario, the weight ratio of the storage capacity can be further increased to balance the importance of different resources.

[0093] Through the above embodiments of the present invention, by dynamically integrating the storage capacity and network resources of storage nodes, the comprehensive capabilities of storage nodes can be quantified. Nodes with high storage capacity or high network bandwidth have larger node weight factors. In subsequent Voronoi diagram partitioning, they can obtain a broader spatial partition based on the weights, implementing the resource allocation logic of "the stronger the ability, the greater the responsibility", and ultimately optimizing the load balancing and resource utilization of the cluster.

[0094] For the above operation S322, a weighted Voronoi diagram can be generated with the node coordinates as the seed points to ensure that high-weight nodes cover a wider area. The difference between the weighted Voronoi diagram and the ordinary Voronoi diagram is that: the ordinary Voronoi diagram divides the space according to the Euclidean distance, and the points within the region are the closest to the corresponding seed points. The weighted Voronoi diagram introduces the node weight factor , redefines the distance metric, so that the regions of high-weight nodes cover a wider area, matching their resource capabilities.

[0095] To generate a weighted Voronoi diagram, first, based on the hash calculation and coordinate transformation methods described in the foregoing formulas (1) to (3), the node coordinates of the storage nodes in the preset space can be determined according to the node identifiers of the storage nodes. Then, according to the distance between the node coordinates and the data coordinates of the existing data in the preset space, as well as the node weight factor, the weighted distance between each storage node and the existing data can be determined. And according to the weighted distance, the node regions divided for the storage nodes can be determined, so that each node region meets the first preset condition.

[0096] According to the embodiments of the present invention, the first preset condition can represent that the region partitioning approaches the optimal state. Specifically, the optimal state can refer to that the region partitioning achieves energy minimization in the hyperbolic geometric space, that is, the total hyperbolic geometric distance between the data coordinates and the node coordinates in each node region is the smallest, while satisfying the dynamic balance of the node weights and the coverage range.

[0097] For example, first, the node coordinates of each storage node in the cluster can be used as the seed points of the Voronoi diagram. These node coordinates can be distributed in a space such as the Poincaré disk, constituting the basic base points for region partitioning. Then, in combination with formula (5), based on the node weight factor dominating the influence of the node region, the spatial points representing the existing data in the preset space based on the data coordinates to the storage nodes represented by the node coordinates weighted distance .

[0098] (5)

[0099] In formula (5), for high-weight nodes Larger. At the same Euclidean distance, its weighted distance is smaller. This makes it easier for spatial points to belong to the area of high-weight nodes.

[0100] By traversing the spatial points within the disk, calculating their weighted distances to all storage nodes, and dividing the spatial points into the node areas with the minimum weighted distance, a weighted Voronoi diagram can be generated. Thus, based on the weighted Voronoi diagram, the node areas divided for each storage node can be determined. In this process, due to the weighted distance advantage, high-weight nodes can attract more spatial points, ultimately covering a wider area and achieving "nodes with stronger resources manage a larger range".

[0101] Through the above embodiments of the present invention, storage nodes with large storage capacity and high bandwidth can manage larger partitions, enabling data tasks to be reasonably allocated to "more capable" nodes, and optimizing cluster load balancing and resource utilization.

[0102] According to an embodiment of the present invention, for the above operation S323, the following optimization method can be combined for optimization: Determine the initial area divided for the storage nodes according to the weighted distance. Move the storage nodes to the positions of the centroids of the initial areas in the preset space, and re-determine the candidate areas divided for the storage nodes. When it is determined that the candidate areas meet the first preset condition, the candidate areas are determined as the node areas. Implement the optimization process of determining the node areas divided for the storage nodes according to the weighted distance.

[0103] For example, iterative optimization can be combined with Lloyd. The Lloyd algorithm is an iterative optimization algorithm, and its core logic is as follows:

[0104] a) Calculate the centroid: For each Voronoi region, calculate the "centroid" of all data points within the Voronoi region. In the Poincaré disk space, it can be represented as calculating the geometric center based on the hyperbolic geometric distance.

[0105] b) Update the seed points: Move the storage nodes (Voronoi seed points) to the centroid positions of the corresponding Voronoi regions and re-divide the Voronoi regions.

[0106] c) Through multiple rounds of iteration, make the division of the Voronoi regions gradually approach the optimal state.

[0107] According to an embodiment of the present invention, the storage nodes can include existing nodes and newly added nodes. The above method of determining the node areas by dividing the weighted Voronoi diagram can be applied to all existing nodes and newly added nodes simultaneously.

[0108] In addition, for both existing nodes and newly added nodes, their node identifiers can include the cluster identifier of the cluster to which the storage node belongs. The cluster can represent a cluster divided based on physical location and a cluster divided based on logical region. For storage nodes belonging to the same cluster, since their node identifiers contain the same cluster identifier, for each storage node in the same cluster, when generating node coordinates based on the node identifiers containing the same cluster identifier, it can be ensured that the coordinates of storage nodes with the same node identifier are adjacent in the disk, forming a local cluster. For example, the storage nodes in the same data center are concentrated in a certain sector of the disk. In the elastic space partitioning phase, the cluster identifier serves as an implicit parameter for node weight calculation. For example, the network bandwidth weight of nodes across data centers may be lower, affecting the region partitioning of the weighted Voronoi diagram, and it can enable high-resource regions (such as high-configuration racks) to obtain a larger coverage area.

[0109] During the optimization process of the Lloyd algorithm, when adding a new node, the cluster identifier can help quickly locate the boundary of the local area and only iteratively adjust the weighted Voronoi regions within the same area, such as adjusting the centroid of the nodes within the rack, reducing the overhead caused by global re-partitioning.

[0110] Specifically, in response to detecting a newly added node, the newly added area divided for the newly added node can be determined according to the node coordinates of the newly added node. Based on the node areas related to the cluster identifier and the newly added area among multiple node areas, a local area cluster is obtained; each area in the local area cluster is iteratively optimized so that each area in the local area cluster meets the first preset condition.

[0111] Figure 3D An operation diagram of the process for adding a new node according to an embodiment of the present invention is shown.

[0112] As Figure 3D shown, the process 330 for adding a new node includes operations S331 to S333.

[0113] In operation S331, a new seed point is initialized.

[0114] After the newly added node joins the system, its coordinates become a new Voronoi seed point, initially partitioning the area. At this time, the original area boundary may become unbalanced due to the intervention of the newly added node.

[0115] In operation S332, the boundary is iteratively optimized.

[0116] The first round of iteration: Calculate the centroid of all node areas (including the newly added area), move the storage nodes in the corresponding node areas to the corresponding centroid positions, and regenerate the Voronoi diagram.

[0117] Multi-round adjustment: Repeat the process of "calculating the centroid - updating the node positions". In each iteration, the regional boundaries are gradually adjusted, and the storage nodes to which the spatial points belong gradually stabilize.

[0118] In operation S333, minimize data migration.

[0119] Through iteration, only the boundaries of the local region clusters affected by the newly added nodes are finely adjusted. Compared with directly re-partitioning, this progressive adjustment avoids large-scale global data reallocation and minimizes the amount of data migration.

[0120] Through the above embodiments of the present invention, since data migration consumes network and computing resources, when adding new nodes, local adjustment can be performed through iterative optimization of the Lloyd algorithm to reduce the amount of migrated data, reduce system overhead and load. And after the newly added nodes break the original partition balance, the regional division can be rebalanced through mathematical iteration to ensure that the load of each node matches the weight, maintain the overall performance of the cluster, and quickly restore balance.

[0121] The above data distribution method is based on the hyperbolic hashing algorithm of the Poincaré disk, which can evenly map data to the hyperbolic space, and the data skew rate is reduced to less than 5%. This algorithm can break through the linear partitioning limit of traditional hashing and utilize the exponential scalability of the hyperbolic space to achieve seamless expansion of the cluster scale from hundreds of nodes to tens of thousands of nodes.

[0122] In addition, in a storage system based on the Poincaré disk model, the dynamic radius adjustment of the hyperbolic coordinate generation algorithm and the improvement of the hash function are the keys to ensuring uniform data distribution.

[0123] According to an embodiment of the present invention, in response to detecting a change in the number of nodes of a plurality of storage nodes for storing existing data, the disk radius of the Poincaré disk can be adjusted.

[0124] For example, in the scenario of a node joining or leaving the cluster, when a new storage node joins the cluster, or an existing storage node leaves the cluster, the overall structure and load situation of the cluster will change. At this time, the original hyperbolic coordinate distribution may no longer be able to ensure data uniformity.

[0125] New node joining: The addition of new nodes will increase storage resources and data processing capabilities. In order to enable the newly added nodes to fully play their roles and make the data reach uniform distribution again in the new cluster environment, the dynamic radius needs to be adjusted. For example, after a new node joins, the system may appropriately increase the disk radius according to the new total number of nodes and resource distribution situation, so that the newly added nodes can cover a larger hyperbolic space area, thereby attracting more data to be stored on this node.

[0126] Existing node exits: When a node exits the cluster due to reasons such as failure or maintenance, the data originally stored on that node needs to be redistributed to other nodes. At this time, adjusting the radius of the disk can enable the remaining nodes to re-divide the hyperbolic space area, avoiding over-concentration of data on certain nodes.

[0127] For example, in the scenario of large-scale data migration in the system, when the system needs to perform large-scale data migration, such as due to replacement of storage media, relocation of data centers, etc., the original data distribution will be disrupted. In order to quickly restore the uniform distribution of data after the data migration is completed, it is necessary to adjust the radius of the disk.

[0128] Migration process: During the data migration process, the system will recalculate the radius of the disk according to the new storage environment and node configuration. Then, according to the new radius and the hyperbolic coordinate generation algorithm, the data will be remapped to each node to ensure uniform distribution of the data in the new cluster environment.

[0129] According to an embodiment of the present invention, in response to detecting that a storage node meets the load imbalance condition, the radius of the Poincaré disk can be adjusted.

[0130] For example, if it is monitored that there is an obvious imbalance in the data load of each storage node in the cluster, that is, the storage utilization rate of some nodes is too high while the storage utilization rate of other nodes is low, it is necessary to adjust the radius of the disk.

[0131] Specific manifestation: By real-time monitoring of multi-dimensional resource metrics such as the storage capacity of the storage node, the number of input / output operations per second (Input / Output Operations Per Second, abbreviated as IOPS), and network bandwidth, when it is found that the storage utilization rate of a certain storage node exceeds a certain threshold (such as 80%), while the storage utilization rate of other nodes is low (such as 20%), it indicates that the cluster meets the load imbalance condition.

[0132] Adjustment purpose: By adjusting the radius of the disk, the distribution of data in the hyperbolic space is changed, enabling the data to migrate from high-load nodes to low-load nodes, thereby achieving balanced distribution of the data.

[0133] For example, to ensure the long-term stability of the system and the uniform distribution of data, the system will perform regular maintenance. During the regular maintenance process, the parameters of the hyperbolic coordinate generation algorithm can be checked and adjusted, including the adjustment of the radius of the disk.

[0134] Regular inspection: The system sets a fixed time interval, such as weekly or monthly, and evaluates the overall status of the cluster at the corresponding time points. If a certain degree of deviation in data distribution is found, the dynamic radius will be fine-tuned to ensure that data can always be evenly distributed across all nodes.

[0135] Based on the data distribution completed by the above method, the above operation S210 can be started to calculate the distance between the data to be stored and the storage nodes. Then, in combination with the load balancing algorithm, operation S220 can be executed to achieve the storage of the data to be stored and other data processing.

[0136] According to an embodiment of the present invention, based on the above data distribution, the intelligentization of the load balancing strategy can be realized.

[0137] In the process of implementing the inventive concept, the inventors found that traditional load balancing methods such as polling and the CRUSH algorithm are only scheduled based on a single metric (such as storage capacity) and it is difficult to dynamically coordinate multi-dimensional resources. For example, the CRUSH algorithm of Ceph has an average request latency of 120ms in a thousand-node cluster.

[0138] According to an embodiment of the present invention, for the load balancing process, the node load factor of each storage node can be calculated by first combining a multi-dimensional load evaluation model. This method may include: obtaining the node load factor according to the storage utilization rate of the storage node, the number of read and write operations per unit time, and the network latency. Among them, the number of read and write operations per unit time can be statistically analyzed using IOPS. The network latency can be statistically analyzed using the average round-trip latency (Round-Trip Time, abbreviated as RTT) from the node to the client.

[0139] For example, the storage node can periodically report the metrics in Table 1 below.

[0140] Table 1:

[0141]

[0142] In Table 1, Used represents the used capacity, and Total represents the total capacity. represents the storage utilization rate, represents IOPS, represents the network latency.

[0143] Node load factor can be calculated using formula (6).

[0144] (6)

[0145] In formula (6), the node load factor By weighted fusion of metrics such as storage utilization, IOPS, and network latency, comprehensively evaluate the load of storage nodes. The weights (such as 0.4, 0.3, 0.3) can be dynamically adjusted according to the actual business scenario. For example, during peak business hours, increase the weight of IOPS, and pay more attention to the impact of read / write performance on the load.

[0146] Through normalizing and weighted calculation of multi-dimensional metrics, this model can achieve refined evaluation of the load of storage nodes and adapt to the requirements of different business scenarios.

[0147] Based on the above node load factor, for the above operation S220, the initial node corresponding to the minimum distance among multiple distances can be determined from multiple storage nodes according to the minimum distance. When the node load factor of the determined initial node meets the second preset condition, the initial node is determined as the target node. When the node load factor of the determined initial node does not meet the second preset condition, the target node is determined according to the distance and node load factor between the data to be stored and each storage node.

[0148] According to an embodiment of the present invention, the second preset condition may include that the node load factor meets a preset range, etc., and is not limited thereto.

[0149] For example, a hyperbolic locality scheduling algorithm as shown in formula (7) can be constructed to determine the target node.

[0150] (7)

[0151] In formula (7), represents the spatial point corresponding to the requested data and the centroid of the node region of each storage node the distance between them. is the centroid of the weighted Voronoi region responsible for by node and can be obtained by calculating the hyperbolic centroid of all data points in the node region through the Lloyd algorithm. Initially can be the node coordinates of storage node After iteration it is dynamically adjusted to the centroid position of the node region to ensure the optimization of the node region division, that is, the total hyperbolic distance is the smallest.

[0152] Based on formula (7), a basic routing policy can be set: data requests are preferentially routed to the storage node with the closest distance. An overload redirection mechanism can also be set: if the node load factor exceeds a preset threshold (such as > 0.8), then the target node can be reselected through formula (7).

[0153] Through the above embodiments of the present invention, by fusing distance and node load factor to determine the target node, it is possible to suppress the selection priority of nodes with too high load while retaining the advantages of data locality, and achieve a more balanced node scheduling.

[0154] According to an embodiment of the present invention, during the optimization process of the aforementioned Lloyd algorithm, in case of load imbalance or node failure, it is also possible to quickly locate the boundary of the local area in combination with the cluster identifier, so as to implement the load balancing strategy within the cluster.

[0155] For example, in response to detecting that the current node load factor of the first node exceeds the load threshold, according to the first cluster identifier of the first cluster to which the first node belongs, determine the first node area cluster related to the first cluster identifier from multiple node areas. Iteratively optimize each first node area in the first node area cluster so that each first node area in the first node area cluster meets the above first preset condition.

[0156] For example, in response to detecting that the second node fails, according to the second cluster identifier of the second cluster to which the second node belongs, determine the second node area cluster related to the second cluster identifier from multiple node areas, and remove the second node coordinates corresponding to the second node from the second node area cluster. Iteratively optimize each second node area in the second node area cluster after removing the second node coordinates so that each second node area in the second node area cluster meets the above first preset condition.

[0157] Through the above embodiments of the present invention, it is possible to only iteratively adjust the weighted Voronoi regions within the same area, reducing the overhead caused by global re-partitioning.

[0158] The above load balancing strategy optimizes the node area division by introducing a prediction model and a dynamic weight formula, in combination with the Voronoi diagram and the Lloyd algorithm. Through locality awareness, the standard deviation of node utilization can be significantly reduced, and the cross-node access latency can be reduced.

[0159] According to an embodiment of the present invention, based on the above data distribution, it is also possible to implement the hierarchicalization of the fault tolerance and recovery mechanism.

[0160] The inventor found during the process of implementing the concept of the present invention that traditional distributed storage systems rely on full-data replication. For example, it takes 30 minutes to recover a single node in GlusterFS.

[0161] According to an embodiment of the present invention, a hierarchical fault tolerance architecture with separation of hot and cold data can be constructed, including constructing a cross-level replica strategy and a hierarchical recovery mechanism to implement the fault tolerance and recovery mechanism.

[0162] The cross - level replica strategy may include: storing a local replica of the data to be stored on other nodes in the cluster to which the target node belongs; and storing a cross - region replica of the data to be stored on cluster nodes of other clusters outside the cluster to which the target node belongs, where the distance between the node coordinates of the cluster nodes of the other clusters in the preset space and the node coordinates of the target node in the preset space is greater than a preset threshold.

[0163] In some embodiments, Figure 4 FIG. shows a schematic diagram of a cross - level replica strategy according to an embodiment of the present invention.

[0164] As Figure 4 shown, the cross - level replica strategy 400 includes a local replica 410 and a cross - region replica 420. For the local replica 410, 3 replicas in the same region can be set. For the cross - region replica 420, 2 erasure - code replicas can be set.

[0165] For the 3 replicas in the same region, the local replica 410 requires storing 3 copies within the node area with the same cluster identifier, and realizes fast data interaction by means of memory synchronization technology. Due to the short data transmission distance and the fast memory processing speed, low - latency synchronization can be achieved using a high - speed network, so that the read latency ≤ 1 millisecond (ms), meeting the low - latency service requirements, such as scenarios like real - time computing and high - frequency trading.

[0166] For the 2 erasure - code replicas, the cross - region replica 420 stores 2 copies in the node areas with different cluster identifiers according to the principle of the farthest hyperbolic distance, ensuring that the replicas are distributed in physically isolated areas and enhancing the disaster tolerance ability. Then, an erasure code can be adopted, such as 10 + 4 encoding, that is, 10 data blocks+4 parity blocks. By using algorithmic redundancy to replace traditional multi - replica redundancy, the storage overhead is reduced to 1.4 times. This method can not only utilize the characteristics of hyperbolic space to disperse the replica positions to enhance the disaster tolerance ability, but also optimize the storage cost through erasure codes, and is applicable to scenarios with requirements for cross - region disaster tolerance but need to control storage resources, such as off - site backup, distributed data center and other scenarios.

[0167] The hierarchical recovery mechanism may include a hot - data recovery mechanism and a cold - data recovery mechanism.

[0168] For the hot data recovery mechanism, it can be directly reconstructed from the local copy 410. By using the memory snapshot and log replay technology, the recovery time is ≤1 second. Parallel recovery is supported. Among them, the core principle of the memory snapshot is copy-on-write and incremental storage. Copy-on-write means that only the modified data blocks are copied when the snapshot is generated, and the unmodified parts share the original data, reducing I / O and memory occupancy. Incremental storage only records the differences between two snapshots, and the storage volume is only 1.2% of the original data. The log replay technology can classify the logs into different queues according to operation types such as write, read, and delete, and support parallel processing. Write operations are processed first to ensure data consistency. Dynamically adjust the batch size and automatically optimize according to the system load. Use a lock-free queue to improve the parallel processing efficiency.

[0169] The cold data recovery mechanism can be asynchronously repaired based on erasure coding (EC), and the resource occupancy rate is limited to less than 5%. The priority of the repair task is adjustable, and the background silent mode is supported.

[0170] In addition, based on the above cluster identifier, when a certain cluster (such as a rack, data center) fails, the cluster identifier can help quickly identify the set of affected storage nodes and trigger targeted recovery strategies.

[0171] For example, when the local copy 410 fails, it can be preferentially recovered from other copies in the local area cluster corresponding to the same cluster identifier to achieve low latency.

[0172] For example, when the cross-region copy 420 fails, erasure coding repair can be started from the local area cluster corresponding to the cluster identifier with the farthest hyperbolic distance.

[0173] Through the above embodiments of the present invention, hot data adopts the memory snapshot + log replay synchronization mechanism, and the recovery time is expected to reach the second level; cold data is stored by cross-region erasure coding (10 + 4 coding), greatly improving the full recovery efficiency. This design breaks through the space limitation of the traditional copy strategy, optimizes the copy distribution according to the principle of the farthest hyperbolic coordinates, and reduces the network transmission overhead.

[0174] Based on the above data storage method, Figure 5 The schematic diagram of the system architecture of the data storage system and its core components according to the embodiments of the present invention is shown.

[0175] As Figure 5 shown, the data storage system 500 includes a metadata coordination device 510, a storage node device 520, and a client access device 530.

[0176] The metadata coordination device 510 can maintain hyperbolic space mapping relationships, node load status, and replica distribution information based on a distributed key-value store, supporting high availability and strong consistency. The metadata coordination device 510 includes: a hyperbolic mapping engine 511 for calculating data coordinates through an improved hash function; a load status database 512 for storing node metrics such as CPU, memory, storage, and network in real time; and a fault detection module 513 for judging node status based on a heartbeat mechanism and a timeout threshold.

[0177] The storage node device 520 supports heterogeneous storage media such as solid-state drives (SSDs), hard disk drives (HDDs), and non-volatile memory express (NVMe), and forms a network with dynamic partitioning in the Poincaré disk space to implement data sharding storage, multi-replica management, and fault isolation. The storage node device 520 includes: a data storage engine 521 that supports block storage, object storage, and file system interfaces; a replica synchronization agent 522 that implements in-group data consistency based on the Replicated And Fault Tolerant (RAFT) protocol; and a resource monitoring agent 523 that periodically reports load metrics to the metadata coordination device 510.

[0178] The client access device 530 can integrate multi-protocol adapters such as Simple Storage Service (S3), Network File System (NFS), and Internet Small Computer System Interface (iSCSI), supporting local caching and prefetching mechanisms. It also supports providing a user interface, protocol adaptation, and intelligent routing 531. The client access device 530 includes an intelligent routing 531 for selecting the optimal storage node according to the hyperbolic locality scheduling algorithm.

[0179] For the data storage system 500, during the device deployment phase, for the metadata coordination device 510, a 3-node etcd (a distributed key-value storage database) cluster is deployed, and the hyperbolic space parameters are configured. For example, the initial radius R = 10 and the number of sectors N = 64 can be configured. The FPGA acceleration module is enabled to improve the hash calculation speed and reduce the metadata query latency. The heartbeat detection interval is set to 5 seconds, and node failure is determined after 3 consecutive timeouts. For the storage node device 520, the storage nodes can be grouped by rack or data center. Inside the group, the Remote Direct Memory Access (RDMA) network (with latency < 1 μs) is adopted, and Internet Protocol Security (IPSec) tunnel encryption can be enabled for cross-group communication. And a unique cluster identifier can be assigned to each storage node, and the hyperbolic coordinate calculation library is loaded. For the client access device 530, multi-protocol adapters such as S3, NFS, and iSCSI can be integrated to support data compression and local caching.

[0180] During the data storage phase, it can include processes such as client preprocessing, metadata query and routing, and dynamic replica allocation.

[0181] For client preprocessing, the client access device 530 can generate data identifiers for relevant data through data sharding. And the hyperbolic hash algorithm can be called to calculate the data coordinates, and a candidate node list is requested from the metadata coordination device 510.

[0182] For example, after generating the data coordinates in the preprocessing phase, the client access device 530 can actively send a request to the metadata coordination device 510, and the request content is "Query the candidate node list suitable for storing this data according to the current data coordinates".

[0183] For metadata query and routing, the metadata coordination device 510 can return the optimal node address according to the load status database 512 and the hyperbolic distance. Thus, the client access device 530 can establish a connection with the target node represented by the optimal node address through the intelligent routing 531 module and transmit the data shards.

[0184] For example, after receiving the above request, the metadata coordination device 510 can calculate the candidate nodes for determining the target node based on the following information:

[0185] Hyperbolic distance: The hyperbolic distance between the data coordinates and the centroid of all node regions, and the storage node with the closest distance is preferentially selected.

[0186] Node load: The real-time node load factor of the storage node is obtained through the load status database 512, and the overloaded nodes are filtered out.

[0187] Node weight: Combine the storage capacity and network bandwidth weights of nodes to ensure that high-resource nodes are given priority.

[0188] The candidate node list returned by the metadata coordination device 510 can contain several optimal node addresses (such as Internet Protocol (IP) + port) for the client to select.

[0189] For dynamic replica allocation, the primary replica can be stored on the target node, the local replica can be stored on the storage nodes of the cluster to which the target node belongs, and the cross-region replica can be allocated to the storage nodes of other clusters according to the principle of the farthest hyperbolic distance. The metadata coordination device 510 can update the replica distribution information to the key-value storage.

[0190] For example, after the client access device 530 completes the data shard transmission, the metadata coordination device 510 can update the replica distribution information to the distributed key-value storage. The content of the key-value pair is as follows:

[0191] Key: Usually the data identifier generated by the client access device 530. This key uniquely identifies a data shard to ensure the uniqueness of subsequent query, update, and recovery operations.

[0192] Value: It is a structured data containing replica distribution information. For example, it can include: the node address where the primary replica is located, the node address where the local replica of the statistical group is located, the cluster and node addresses where the cross-region replica is located, the replica version number, the last update time, etc., and is not limited to this.

[0193] The key-value pair maintains the mapping relationship between the data shard and the storage node, ensuring that the metadata coordination device 510 can quickly locate the data location and verify the replica consistency during subsequent reading, fault recovery, and load balancing.

[0194] In addition, performance optimization can also be performed on the data storage system, including adaptive parameter adjustment and hardware acceleration.

[0195] For adaptive parameter adjustment, the metadata coordination device 510 can predict the load trend of nodes based on the Long Short-Term Memory (LSTM) model and dynamically adjust the weight coefficients of the metrics used to determine the node weight factor.

[0196] For example: Peak hours (9:00 - 18:00): Increase the IOPS weight to 0.5 and decrease the storage utilization weight to 0.3. Off-peak hours: Restore the default weights (0.4 / 0.3 / 0.3).

[0197] This mechanism captures the non - linear law of load changes through an LSTM model, dynamically optimizes the weight coefficients, and can achieve the intelligent evolution of resource allocation strategies. Compared with the static weight method, it reduces the prediction error, improves the system throughput, and provides core guarantee for the stability and performance of distributed systems in complex scenarios.

[0198] For hardware acceleration, an FPGA parallel acceleration module can be adopted to achieve parallel processing of coordinate calculations, realize the real - time nature of data mapping and storage, improve the throughput, and meet the high - concurrency requirements.

[0199] Through the above - mentioned embodiments of the present invention, a method for implementing a data storage system based on a preset space is provided. Through the preset space, especially the space mapping characteristics of the hyperbolic geometric space, data uniform distribution, dynamic load balancing, and hierarchical fault - tolerance mechanisms are realized. It is applicable not only to fixed - network environments such as traditional data centers, but also can be applied to mobile networks and emerging technology fields through adaptive adjustment. Specifically, data uniform distribution is achieved through the hyperbolic hashing algorithm, reducing the data skew rate. Combining multi - dimensional resource metrics with hyperbolic locality awareness reduces the standard deviation of node utilization. The hot - data recovery time ≤ 1 second, improving the full - volume data recovery efficiency. In addition, seamless expansion from hundreds of nodes to tens of thousands of nodes is supported, and the service interruption time approaches zero.

[0200] Figure 6 The structural block diagram of a data storage device according to an embodiment of the present invention is shown.

[0201] As Figure 6 shown, the data storage device 600 includes a distance determination module 610 and a target node determination module 620.

[0202] The distance determination module 610 is configured to determine the distances between the data to be stored and multiple storage nodes according to the data coordinates of the data to be stored in the preset space and the node centroid coordinates of the multiple storage nodes, obtaining multiple distances. Among them, each of the multiple storage nodes has a node area in the preset space, and the node centroid coordinates represent the coordinates of the node area centroid of the node area in the preset space.

[0203] The target node determination module 620 is configured to determine a target node for storing the data to be stored from the multiple storage nodes according to the multiple distances.

[0204] According to an embodiment of the present invention, the data storage device further includes a hash calculation module and a data coordinate determination module.

[0205] The hash calculation module is configured to perform hash calculation on the data identifier of the data to be stored, obtaining a coordinate factor for representing the position information of the data to be stored in the preset space and an angle factor for representing the distribution direction of the data to be stored in the preset space.

[0206] A data coordinate determination module, configured to determine data coordinates according to a coordinate factor and an angle factor.

[0207] According to an embodiment of the present invention, the preset space includes a Poincaré disk. The hash calculation module includes a coordinate factor obtaining unit and an angle factor obtaining unit.

[0208] The coordinate factor obtaining unit is configured to perform a modulo operation on the disk radius of the Poincaré disk based on the first hash value of the data identifier to obtain a coordinate factor.

[0209] The angle factor obtaining unit is configured to perform a modulo operation on 2π based on the second hash value of the data identifier to obtain an angle factor.

[0210] According to an embodiment of the present invention, the data coordinate determination module includes an exponential transformation unit and a data coordinate determination unit.

[0211] The exponential transformation unit is configured to perform an exponential calculation with the natural constant as the base on the coordinate factor to obtain an exponential transformation result of the coordinate factor.

[0212] The data coordinate determination unit is configured to determine data coordinates according to the distribution direction of the coordinate points represented by the angle factor within the Poincaré disk and the distance from the coordinate points determined based on the exponential transformation result to the center of the Poincaré disk.

[0213] According to an embodiment of the present invention, each of the multiple storage nodes has a node weight factor. The data storage device further includes a node coordinate determination module, a weighted distance determination module, and a node area division module.

[0214] The node coordinate determination module is configured to determine the node coordinates of the storage node in the preset space according to the node identifier of the storage node.

[0215] The weighted distance determination module is configured to determine the weighted distance between each storage node and the existing data according to the distance between the node coordinates and the data coordinates of the existing data in the preset space, and the node weight factor.

[0216] The node area division module is configured to determine the node areas divided for the storage nodes according to the weighted distance, such that each node area satisfies a first preset condition.

[0217] According to an embodiment of the present invention, the node area division module includes an initial area division unit, a candidate area division unit, and a node area determination unit.

[0218] The initial area division unit is configured to determine the initial areas divided for the storage nodes according to the weighted distance.

[0219] A candidate region division unit, configured to move a storage node to a position of the centroid of an initial region in a preset space, and re-determine a candidate region divided for the storage node.

[0220] A node region determination unit, configured to determine a candidate region as a node region when it is determined that the candidate region meets a first preset condition.

[0221] According to an embodiment of the present invention, the node identifier includes a cluster identifier of a cluster to which the storage node belongs. The data storage device further includes a new region determination module, a local region cluster acquisition module, and an iterative optimization module.

[0222] The new region determination module is configured to, in response to detecting a new node, determine a new region divided for the new node according to the node coordinates of the new node.

[0223] The local region cluster acquisition module is configured to obtain a local region cluster according to node regions related to the cluster identifier and new regions in a plurality of node regions.

[0224] The iterative optimization module is configured to iteratively optimize each region in the local region cluster, so that each region in the local region cluster meets the first preset condition.

[0225] According to an embodiment of the present invention, the data storage device further includes a node capacity factor determination module, a node bandwidth factor determination module, and a node weight factor determination module.

[0226] The node capacity factor determination module is configured to determine a node capacity factor of each storage node according to the storage capacity of each storage node and the storage capacities of a plurality of storage nodes.

[0227] The node bandwidth factor determination module is configured to determine a node bandwidth factor of each storage node according to the network bandwidth of each storage node and the network bandwidths of a plurality of storage nodes.

[0228] The node weight factor determination module is configured to perform weighted calculation on the node storage capacity factor and the node network bandwidth factor to obtain a node weight factor.

[0229] According to an embodiment of the present invention, the target node determination module includes an initial node determination unit and a target node determination unit.

[0230] The initial node determination unit is configured to determine an initial node corresponding to the minimum distance from a plurality of storage nodes according to the minimum distance with the smallest value among a plurality of distances.

[0231] The first target node determination unit is configured to determine the initial node as a target node when it is determined that the node load factor of the initial node meets a second preset condition.

[0232] According to an embodiment of the present invention, each of the multiple storage nodes has a node load factor. The target node determination module further includes a second target node determination unit.

[0233] The second target node determination unit is configured to determine a target node according to the distance between the data to be stored and each storage node and the node load factor when it is determined that the node load factor of the initial node does not meet the second preset condition.

[0234] According to an embodiment of the present invention, the data storage device further includes a node load factor acquisition module.

[0235] The node load factor acquisition module is configured to obtain the node load factor according to the storage utilization rate of the storage node, the number of read and write operations per unit time, and the network latency.

[0236] According to an embodiment of the present invention, the data storage device further includes a local copy storage module and a cross-region copy storage module.

[0237] The local copy storage module is configured to store the local copy of the data to be stored to other nodes in the cluster to which the target node belongs.

[0238] The cross-region copy storage module is configured to store the cross-region copy of the data to be stored to the cluster nodes of other clusters outside the cluster to which the target node belongs, and the distance between the node coordinates of other clusters in the preset space and the node coordinates of the target node in the preset space is greater than a preset threshold.

[0239] According to an embodiment of the present invention, any multiple of the distance determination module 610 and the target node determination module 620 may be combined and implemented in one module, or any one of them may be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules may be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the distance determination module 610 and the target node determination module 620 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging circuits, etc., implemented by hardware or firmware, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in any appropriate combination of several of them. Alternatively, at least one of the distance determination module 610 and the target node determination module 620 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions may be executed.

[0240] Figure 7A block diagram of an electronic device suitable for implementing a data storage method according to an embodiment of the present invention is shown.

[0241] As Figure 7 shown, the electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage section 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include on-board memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0242] In the RAM 703, various programs and data required for the operation of the electronic device 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. The processor 701 performs various operations of the method flow according to an embodiment of the present invention by executing the programs in the ROM 702 and / or the RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 may also perform various operations of the method flow according to an embodiment of the present invention by executing the programs stored in the one or more memories.

[0243] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, and the input / output (I / O) interface 705 is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read from it can be installed into the storage section 708 as needed.

[0244] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist alone without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the data storage method according to the embodiments of the present invention is implemented.

[0245] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the above-described ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703.

[0246] An embodiment of the present invention further includes a computer program product, which includes a computer program, and the computer program includes program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to cause the computer system to implement the data storage method provided by the embodiments of the present invention.

[0247] When the computer program is executed by the processor 701, the above functions defined in the system / apparatus of the embodiments of the present invention are executed. According to an embodiment of the present invention, the above-described systems, apparatuses, modules, units, etc. may be implemented by computer program modules.

[0248] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and is downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code included in the computer program may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0249] In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-described functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the above-described system, device, apparatus, module, unit, etc. can be implemented by computer program modules.

[0250] The embodiments of the present invention have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present invention, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should fall within the scope of the present invention.

Claims

1. A data storage method, characterized in that, The method includes: Determining distances between the data to be stored and multiple storage nodes according to the data coordinates of the data to be stored in a preset space and the node centroid coordinates of the multiple storage nodes, to obtain multiple distances; wherein, each of the multiple storage nodes has a node area in the preset space, the node centroid coordinates represent the coordinates of the node area centroid of the node area in the preset space, the preset space includes a Poincaré disk, and the data coordinates include hyperbolic coordinates; Determining a target node for storing the data to be stored from the multiple storage nodes according to the multiple distances; Adjusting the disk radius of the Poincaré disk in response to detecting a change in the number of nodes of the multiple storage nodes for storing existing data; Adjusting the disk radius of the Poincaré disk in response to detecting that the storage nodes meet the load imbalance condition; Remapping the data to each storage node according to the new disk radius and the hyperbolic coordinate generation algorithm, where the hyperbolic coordinate generation algorithm is used to calculate the hyperbolic coordinates of the data.

2. The method according to claim 1, wherein The method further includes: Performing a hash calculation on the data identifier of the data to be stored to obtain a coordinate factor representing the position information of the data to be stored in the preset space and an angle factor representing the distribution direction of the data to be stored in the preset space; Determining the data coordinates according to the coordinate factor and the angle factor.

3. The method according to claim 2, characterized in that, The performing a hash calculation on the data identifier of the data to be stored to obtain a coordinate factor representing the position information of the data to be stored in the preset space and an angle factor representing the distribution direction of the data to be stored in the preset space includes: Taking the modulus of the disk radius of the Poincaré disk based on the first hash value of the data identifier to obtain the coordinate factor; Taking the modulus of 2π based on the second hash value of the data identifier to obtain the angle factor.

4. The method according to claim 3, characterized in that, The determining the data coordinates according to the coordinate factor and the angle factor includes: Performing an exponential calculation with the natural constant as the base on the coordinate factor to obtain an exponential transformation result of the coordinate factor; Determining the data coordinates according to the distribution direction of the coordinate point represented by the angle factor within the Poincaré disk and the distance from the coordinate point determined based on the exponential transformation result to the center of the Poincaré disk.

5. The method according to claim 1, wherein Each of the multiple storage nodes has a node weight factor; The method further includes: Determining the node coordinates of the storage node in the preset space according to the node identifier of the storage node; Determining the weighted distances between each storage node and the existing data according to the distance between the node coordinates and the data coordinates of the existing data in the preset space, and the node weight factor; Determining the node areas divided for the storage nodes according to the weighted distances, such that each of the node areas meets a first preset condition.

6. The method according to claim 5, wherein The determining the node areas divided for the storage nodes according to the weighted distances includes: Determining an initial area divided for the storage nodes according to the weighted distances; Move the storage node to the position of the centroid of the initial region in the preset space, and re-determine the candidate region divided for the storage node; When it is determined that the candidate region meets the first preset condition, determine the candidate region as the node region.

7. The method according to claim 5 or 6, characterized in that, The node identifier includes the cluster identifier of the cluster to which the storage node belongs; The method further includes: In response to detecting a newly added node, determine a newly added region divided for the newly added node according to the node coordinates of the newly added node; Obtain a local region cluster according to the node regions related to the cluster identifier in the multiple node regions and the newly added region; Iteratively optimize each region in the local region cluster so that each region in the local region cluster meets the first preset condition.

8. The method according to claim 5, wherein The method further includes: Determine the node capacity factor of each storage node according to the storage capacity of each storage node and the storage capacities of the multiple storage nodes; Determine the node bandwidth factor of each storage node according to the network bandwidth of each storage node and the network bandwidths of the multiple storage nodes; Perform a weighted calculation on the node capacity factor and the node bandwidth factor to obtain the node weight factor.

9. The method according to claim 1, characterized in that The determining a target node for storing the data to be stored from the multiple storage nodes according to the multiple distances includes: Determine an initial node corresponding to the minimum distance from the multiple storage nodes according to the minimum distance among the multiple distances; When it is determined that the node load factor of the initial node meets the second preset condition, determine the initial node as the target node.

10. The method according to claim 9, wherein Each of the multiple storage nodes has a node load factor; The determining a target node for storing the data to be stored from the multiple storage nodes according to the multiple distances further includes: When it is determined that the node load factor of the initial node does not meet the second preset condition, determine the target node according to the distance between the data to be stored and each storage node and the node load factor.

11. The method according to claim 9 or 10, characterized in that, The method further includes: Obtain the node load factor according to the storage utilization rate, the number of read and write operations per unit time, and the network latency of the storage node.

12. The method according to claim 1, characterized in that, The method further includes: Store a local copy of the data to be stored in other nodes of the cluster to which the target node belongs; Store a cross-region copy of the data to be stored in cluster nodes of other clusters outside the cluster to which the target node belongs, and the distance between the node coordinates of the other cluster nodes in the preset space and the node coordinates of the target node in the preset space is greater than a preset threshold.

13. An electronic device, comprising: One or more processors; A memory for storing one or more computer programs, Characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing a computer program or instructions thereon, characterized in that, The computer program or instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

Images