Data index construction method for distributed data storage

Abstract

The invention relates to the technical field of distributed data storage, and discloses a data index construction method for distributed data storage, comprising the following steps: constructing a hypergraph model of a distributed storage system through a high-order graph theory, the hypergraph model comprising a plurality of storage nodes, a plurality of data blocks and a plurality of query requests, the method comprises the following steps: performing load balancing optimization on storage nodes and a query request on the basis of a minimum cut theorem in a high-order graph model, mapping data blocks and the storage nodes to a consistent hash ring according to a consistent hash algorithm, reducing data migration through a virtual node mechanism, storing the data blocks on a plurality of storage nodes in a redundant manner by adopting a redundant coding method, and storing the data blocks on the storage nodes in a redundant manner. And a load distribution strategy of the storage nodes is adjusted through a game theory model, and load adjustment is optimized by adopting an optimal control method. The method can ensure high availability, expansibility and reliability of the system while improving the efficiency of the data storage system, and is suitable for a large-scale distributed data storage system.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed data storage, and in particular to a method for constructing a data index for distributed data storage. Background Art

[0002] As the amount of data and system scale continue to grow, efficient storage and fast access to data have become key issues. Traditional distributed storage methods often rely on simple hash algorithms and static data allocation mechanisms. Although they can ensure the uniformity of data distribution and high availability, they gradually expose system performance and scalability issues when faced with large-scale data storage and frequent node expansion.

[0003] Traditional load balancing methods usually use a static approach to configure storage nodes, lacking the ability to adapt to dynamic load changes in real time. When some nodes become bottlenecks due to excessive traffic, the system often cannot effectively adjust the load, causing some nodes to be overloaded while others are idle, which not only reduces resource utilization efficiency, but also affects the speed of data query and the overall performance of the system.

[0004] Current distributed data storage systems have large data migration overhead when expanding. Although traditional consistent hashing algorithms can solve some data migration problems, they still require a lot of data redistribution when the number of nodes changes dramatically. Especially in scenarios of large-scale expansion and dynamic node changes, the cost of data migration will be very high, resulting in slower system response speed and even affecting data consistency and stability. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a data index construction method for distributed data storage, which solves the problems of low data index construction efficiency, uneven load distribution, high data migration cost and insufficient storage resource utilization in existing distributed data storage systems.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for constructing a data index for distributed data storage, comprising the following steps: A hypergraph model of a distributed storage system is constructed by high-order graph theory, wherein the hypergraph model includes multiple storage nodes, multiple data blocks, and multiple query requests; Load balancing optimization of storage nodes and query requests based on the minimum cut theorem in the high-order graph model; Map data blocks and storage nodes to the consistent hash ring according to the consistent hash algorithm, and reduce data migration through the virtual node mechanism; A redundant encoding method is used to redundantly store data blocks on multiple storage nodes; The load distribution strategy of storage nodes is adjusted through the game theory model, and the optimal control method is used to optimize the load adjustment.

[0007] Preferably, each hyperedge in the high-order graph theory model represents the dependency relationship between multiple data blocks and multiple storage nodes or query requests, and the cutting cost of the hyperedge is optimized by the minimum cut theorem to ensure the shortest query path and achieve load balancing.

[0008] Preferably, in the consistent hashing algorithm, storage nodes and data blocks are mapped to a consistent hashing ring through hash values, each storage node is mapped to the hashing ring through multiple virtual nodes, and the node load is optimized through the virtual node mechanism.

[0009] Preferably, the redundant coding adopts Reed-Solomon coding, stores each data block in multiple storage nodes, and further reduces data storage overhead through data compression method.

[0010] Preferably, the game theory model adjusts the load distribution strategy of the storage nodes through the Nash equilibrium principle, so that each storage node selects the optimal load distribution solution while considering the loads of other nodes.

[0011] Preferably, define the system load state variable , represents the load status of each storage node at time t. Assuming that the system contains N storage nodes, the load status variable It represents the load status of node i at time t. The update of load status depends on the number of requests processed by the node, the amount of data stored, and the query load factor. Specifically: ; in, is the computing power of the storage node, is the query request volume of node i, The amount of data stored for node i; Set the system's load optimization objective function J, which describes the load distribution of all nodes and minimizes the load imbalance of the entire system by adjusting the node load. The load optimization objective function J represents the cumulative load cost within the time window T, specifically: ; Among them, the integral symbol is the accumulation of the entire system load cost in the time interval [0,T], is the load cost function of the node, reflecting the increase in cost as the node load increases, and T is the optimization time window; Solve the optimal control strategy for load distribution through optimal control theory , Where is the load adjustment amount of storage node i at time t, and the control goal is to dynamically adjust , so that the system load is minimized within a given time window and the load balancing constraints are met: ; The constraints include: ; in, is the total system load, which represents the sum of the loads of all storage nodes. is the load limit of a single node, and the constraint The load of each storage node i must be less than or equal to the maximum load limit ， Indicates that the constraint condition applies to every storage node i; Based on the Lagrange multiplier method or variational method, by solving the optimal control problem in the above formula, the optimal load adjustment strategy for each storage node i is obtained , so that the system can achieve load balancing on a global scale and minimize the overall load cost; According to the optimal control strategy solved , dynamically adjust the load distribution of each storage node, so that each storage node performs corresponding load adjustment under the constraints of load balancing and performance optimization, thereby achieving the global optimal load distribution of the system.

[0012] Preferably, when the system is expanded, the method of minimizing data migration by combining a consistent hashing algorithm and a high-order graph model includes the following steps: The data mapping on the hash ring is adjusted through the consistent hashing algorithm, combined with the load balancing strategy of the high-order graph model to minimize the amount of data migration between nodes during the expansion process.

[0013] Preferably, the load balancing optimization adopts a graph cutting and reorganization strategy based on high-order graph theory, and optimizes load distribution by adjusting the dependency between data blocks and storage nodes when the node loads are uneven.

[0014] The present invention provides a method for constructing a data index for distributed data storage, which has the following beneficial effects: 1. The present invention optimizes the load distribution between storage nodes and query requests through high-order graph models and the minimum cut theorem, significantly reduces the query path length, and improves data retrieval efficiency. At the same time, combined with game theory models and optimal control theory, the load of storage nodes is dynamically adjusted to ensure system load balancing, avoid overload or idle phenomena, and improve the overall performance of the system.

[0015] 2. The present invention uses a consistent hashing algorithm and a virtual node mechanism to minimize the amount of data migration when the system is expanded, and only migrates necessary data blocks, avoiding the high overhead of full data migration. The system can be expanded smoothly, reducing performance losses caused by expansion, and improving the scalability and sustainability of the system.

[0016] 3. The present invention adopts redundant coding technology to store data blocks redundantly on multiple nodes, ensuring that data can be quickly restored when a single node fails, thereby enhancing the fault tolerance and reliability of the system. In addition, through the combination of high-order graph models and consistent hashing algorithms, the system can continuously optimize resource allocation during dynamically changing loads and expansion processes to ensure stable system operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Please see attached Figure 1 The embodiment of the present invention provides a method for constructing a data index for distributed data storage, comprising the following steps: S1. Constructing a hypergraph model of a distributed storage system through high-order graph theory, where the hypergraph model includes multiple storage nodes, multiple data blocks, and multiple query requests; S2, load balancing optimization of storage nodes and query requests based on the minimum cut theorem in the high-order graph model; S3, map data blocks and storage nodes to the consistent hash ring according to the consistent hash algorithm, and reduce data migration through the virtual node mechanism; S4, redundantly storing the data blocks on multiple storage nodes using a redundant encoding method; S5. Use the game theory model to adjust the load distribution strategy of the storage node and use the optimal control method to optimize the load adjustment; S6. When the system is expanded, combine the consistent hashing algorithm and the high-order graph model to minimize data migration.

[0020] Regarding S1, in this embodiment, a hypergraph model of a distributed storage system is constructed by high-order graph theory, and the hypergraph model includes multiple storage nodes, multiple data blocks, and multiple query requests. First, a hypergraph model is constructed by high-order graph theory, and the hypergraph model includes multiple storage nodes, multiple data blocks, and multiple query requests. The construction of the hypergraph model aims to provide a theoretical basis for subsequent load balancing optimization, data mapping, and query path optimization. Specifically, the model abstracts and models the relationship between the elements in the system, so that the optimization of data storage and query can be carried out more accurately and efficiently.

[0021] In one possible implementation, the nodes in the hypergraph model are divided into three categories: storage nodes, data blocks, and query requests. Each type of node plays a different role in the graph and is connected to other nodes through hyperedges to form a complex relationship network.

[0022] Storage nodes: Storage nodes are the core components of a distributed system, usually referring to hardware devices or virtualized storage instances that carry data storage functions. In some embodiments, storage nodes can be independent servers, storage arrays, or cloud storage instances.

[0023] Data block: A data block is a basic data unit stored in the system. Each data block can be power monitoring data, voltage and current sensor acquisition data, etc. In the embodiment of the present invention, the granularity of the data block is flexible and adjustable, and can be a single sensor data point or a multi-dimensional data set.

[0024] Query request: A query request indicates the need to access data in the system. It is usually initiated by an end user and requires obtaining certain data blocks or related information from a storage node. There are various types of query requests, including time range query, conditional filtering query, etc.

[0025] In this embodiment, the hyperedge in the hypergraph model represents the dependency relationship between storage nodes, data blocks, and query requests. Specifically, the setting of the hyperedge can be defined in the following two ways: The hyperedge between a storage node and a data block indicates that the data block is stored on one or more storage nodes.

[0026] The hyperedge between a query request and a data block indicates that a query request involves access to one or more data blocks.

[0027] The hypergraph model can clearly express the storage distribution of data blocks, the access path of query requests, and the dependencies between nodes, thus laying the foundation for subsequent optimization steps.

[0028] To better understand the hypergraph model, we can use a specific example to illustrate: suppose in a power monitoring system, the storage node includes multiple power module monitoring devices, the data block includes the real-time voltage and current data of each power module, and the query request is the query operation of the end user requesting power data. The hypergraph model abstracts the storage node, data block, and query request into three different types of nodes in the graph, and connects them through hyperedges to form a complete dependency network.

[0029] In a possible implementation, the construction of the hypergraph model can also be weighted based on the mapping relationship between storage nodes and data blocks, as well as the priority and access mode of the query request. Specifically, for each query request , you can set an access weight , indicating the processing priority of the query request; and for each storage node , you can set a load weight , indicating the load condition of the node.

[0030] Based on the above model, the hypergraph model can be formally described using the following mathematical representation: ; in, is the node set of the hypergraph, including storage nodes , data block and query request , E is a set of hyperedges, representing the relationship between nodes: Each hyperedge Can be defined by the following subsets: ; Among them, e is a relationship that connects storage nodes, data blocks, and query requests.

[0031] In order to more accurately express the load situation between nodes, the strength of the relationship between each storage node and the data block can be defined by weight The weight can be adjusted dynamically based on the storage capacity, processing power, and access frequency of the data block of the storage node. and data blocks The strength of the relationship between It can be calculated according to the following formula: ; in, Represents a data block The frequency of visits, Is a storage node capacity.

[0032] In this way, the hypergraph model not only provides the topological relationship between storage nodes and data blocks, query requests, but also takes into account factors such as load and capacity, thereby providing more accurate information for load balancing optimization.

[0033] The hypergraph model of this embodiment is not limited to describing a single type of data block and storage node. In some embodiments, the hypergraph model can also be extended to support multi-level, multi-dimensional complex data storage structures. For example, storage nodes may be classified according to different data types and access frequencies, and query requests can be further subdivided into multiple sub-requests according to their query conditions. This multi-level, multi-dimensional hypergraph model can more comprehensively express the complex relationships of the storage system.

[0034] In addition, the hypergraph model can also be used in combination with other data structures, such as hash tables, B-trees, etc., to achieve fast query and efficient access to data. With this multi-level data structure combination, the hypergraph model can better adapt to dynamic changes in large-scale distributed storage systems.

[0035] The hypergraph model constructed by this embodiment through high-order graph theory provides a basic framework for subsequent storage optimization, query path optimization and load balancing. Through this method, a more efficient distributed storage data index construction can be achieved, and reliable guarantees can be provided for the scalability, fault tolerance and query performance of the system.

[0036] For S2, in this embodiment, the minimum cut theorem in the high-order graph model is used to optimize the load balance between storage nodes and query requests. Specifically, the hyperedges in the hypergraph are optimized by the minimum cut theorem to achieve the purpose of load balancing, ensuring that the query requests are distributed as evenly as possible among the storage nodes, and avoiding performance bottlenecks caused by overloading of some storage nodes.

[0037] The minimum cut theorem is widely used in graph theory, especially in scenarios such as network traffic control and load balancing, which can help achieve reasonable resource allocation. The core idea of ​​the theorem is to minimize the cost involved by cutting certain edges or hyperedges in the graph, thereby balancing the resource consumption between nodes while ensuring network connectivity.

[0038] In this embodiment, the minimum cut theorem is applied to the hyperedges in the high-order graph model to optimize the load distribution between storage nodes and query requests. Specifically, by optimizing the cut cost of the hyperedge, the load of the storage node can be evenly distributed when responding to the query request, avoiding query delays or system bottlenecks caused by load imbalance.

[0039] Specifically, for each query request and storage nodes , you can define the cutting cost according to its access frequency and load conditions. Set the cutting cost Represents a storage node and query request The load relationship between them, the cutting cost can be calculated by the following formula: ; in, is the load weight, which represents the processing capacity or request frequency of the storage node or query request. The goal of this cutting cost is to minimize it to achieve load balancing.

[0040] Specifically, for each storage node , its load weight It can be defined by the following formula: ; Among them, Q is the query request set involved, For query request For storage nodes The request load is is the capacity of the storage node.

[0041] By optimizing the cutting cost, we can ensure that query requests are evenly distributed on multiple storage nodes, avoiding some storage nodes from being overloaded due to frequent queries. The specific optimization goal can be expressed by the following formula: ; Among them, E is a set of hyperedges, which represents the dependency relationship between nodes. The realization of this optimization goal will improve the query efficiency of the system and reduce the response time.

[0042] In one possible implementation, load balancing optimization is performed in steps. First, the strength of the relationship between each storage node and the query request is calculated, and the cutting cost of the hyperedge is determined. Then, based on the cutting cost, the minimization optimization is performed, and the flow algorithm in graph theory (such as the maximum flow-minimum cut algorithm) is used to perform cutting, thereby achieving balanced load distribution.

[0043] In graph theory, the maximum flow-minimum cut theorem is a classic theorem, which shows that in a flow network, the maximum flow is equal to the capacity of the minimum cut. This theorem is applied to the present invention, that is, to minimize the response delay of query requests by cutting hyperedges, thereby achieving load balancing. Specifically, for each query request , the minimum cut theorem optimization can be achieved through the following process: Compute traffic network: Each storage node and query request Viewed as nodes in a flow network, the connection strength (i.e., flow) between nodes is defined by the cut cost.

[0044] Maximum flow calculation: The maximum flow from the source node (representing the data generation source) to the sink node (representing the final target of the data query) is calculated through the maximum flow algorithm.

[0045] Minimum cut calculation: Use the minimum cut algorithm to determine the cut edges that divide the traffic network into sources and sinks, and achieve load balancing by minimizing the cost of the cut edges.

[0046] This method can effectively distribute query requests evenly to multiple storage nodes and avoid overloading some nodes due to too many query requests.

[0047] Through load balancing optimization based on the minimum cut theorem, the query efficiency and response speed of the storage system can be significantly improved. The optimized system can evenly distribute query requests among storage nodes, thereby reducing the load difference between nodes and avoiding performance bottlenecks caused by high load on some nodes.

[0048] Specifically, the system performance improvement can be reflected in the following aspects: Reduced query response time: Since query requests can be evenly distributed to multiple storage nodes, query response time is effectively controlled, avoiding delays caused by query requests being too concentrated on certain storage nodes.

[0049] Improved system throughput: Through load balancing optimization, the overall system throughput has been improved and can handle more query requests simultaneously.

[0050] In the case of load balancing, if a storage node fails, the system can reduce the pressure on other nodes by redistributing query requests, thereby enhancing the system's fault tolerance.

[0051] In some embodiments, the application of the minimum cut theorem is not limited to load balancing between storage nodes and query requests, but can also be extended to optimizing resource allocation between storage nodes. By optimizing resource usage between storage nodes, the resource utilization of the system can be further improved, energy consumption can be reduced, and storage efficiency can be improved.

[0052] In addition, for different types of query requests and data blocks, more constraints can be introduced into the hypergraph model, such as query priority, access frequency of data blocks, etc., to achieve more refined load balancing. In this case, the optimization goal of the minimum cut theorem is not only to minimize query latency, but also to comprehensively consider the priority of query requests and the access mode of data blocks, further improving the overall performance of the system.

[0053] In summary, the present invention optimizes the load distribution between storage nodes and query requests through the minimum cut theorem in high-order graph theory, which can effectively achieve system load balancing, improve query efficiency and system throughput, and provide strong theoretical support and technical means for the design and application of distributed storage systems.

[0054] For S3, in this embodiment, a consistent hashing algorithm is used to map data blocks and storage nodes to a consistent hashing ring to improve the scalability and fault tolerance of the system. Specifically, each data block and storage node is mapped to a position on the ring through a consistent hashing algorithm, and a virtual node mechanism is introduced to further reduce the need for data migration, ensuring data stability and query efficiency when the system expands or nodes change.

[0055] The consistent hashing algorithm is a distributed hashing method that aims to efficiently map storage nodes and data blocks into a ring structure, and use this structure to solve the data migration problem when nodes are added or reduced. In traditional hashing algorithms, when storage nodes change, a large number of data blocks need to be recalculated and mapped, causing drastic fluctuations in system performance. The consistent hashing algorithm ensures a more balanced distribution of data blocks by mapping storage nodes and data blocks to the same hash ring, and can also effectively reduce data migration caused by the increase or decrease of nodes.

[0056] Specifically, suppose there are N storage nodes and M data blocks All will be mapped to a position on the consistent hashing ring through the hash function. Set the hash function to ,in For storage nodes or data blocks, hash functions Will Mapped to a point on the ring. The mapping result of each node and data block forms a ring, and the mapping relationship between the storage node and the data block is determined according to its position on the hash ring.

[0057] In this mapping mode, the storage node will be responsible for storing all the data blocks that come after it. For example, if data block Mapped to a position on the hash ring, and The location is located in the storage node Afterwards, the data block Will be stored in the node In this way, the consistent hashing algorithm can ensure that data blocks are evenly distributed across multiple storage nodes.

[0058] On the basis of consistent hashing, this embodiment further introduces a virtual node mechanism to further reduce the impact of node increase or decrease on data distribution. Specifically, in traditional consistent hashing, when the number of nodes is small, some nodes may face overload problems. In order to alleviate this problem, virtual nodes can be introduced, that is, each physical storage node is mapped to multiple virtual nodes, so that the position of each physical node on the hash ring will be expanded, thereby increasing the flexibility of the system.

[0059] The introduction of virtual nodes is as follows: Assume that each physical storage node Mapped to v virtual nodes These virtual nodes are evenly distributed on the hash ring, which enhances the load balancing of the system. When the number of physical nodes increases or decreases, only the position of the virtual node needs to be adjusted, thus reducing the large-scale migration of data blocks.

[0060] Through consistent hashing and virtual node mechanisms, this embodiment can effectively reduce the overhead of data migration, especially when the system is expanded. Specifically, assume that at a certain moment, the number of storage nodes in the system increases from N to N+1. Traditional hash algorithms usually require recalculation and mapping of all data blocks, resulting in a large amount of data migration. After adopting the consistent hashing algorithm, only a few data blocks will be migrated to the new node, and these data blocks are all data blocks located before the position of the newly added node. Through the further introduction of virtual nodes, the scale of data migration has been further optimized.

[0061] Specifically, assuming there are N storage nodes, through the introduction of virtual nodes, the added nodes will correspond to multiple virtual nodes. When a new node is added, the new virtual node will take over some of the data blocks of the original virtual node, while most of the data blocks will remain unchanged, thereby greatly reducing the amount of data migration.

[0062] For example, suppose the storage node and Mapped to their respective virtual nodes , and , When the new node When joining, its virtual node will take over some data blocks, while the distribution of other data blocks will not be affected. Therefore, only some data blocks will be migrated to the new node, thus avoiding large-scale data migration in the system.

[0063] The combination of consistent hashing algorithm and virtual node mechanism ensures that the storage system can smoothly migrate data during expansion, avoiding sharp fluctuations in system performance during expansion. Specifically, by properly designing the number and distribution of virtual nodes, it can ensure that the load of data blocks is evenly distributed, and when the system is expanded, the migration amount of data blocks is kept to a minimum.

[0064] In some embodiments, the virtual node mechanism is not limited to the addition and subtraction of storage nodes, but can also be used to improve the fault tolerance of the system. For example, when some storage nodes fail, the mapping of virtual nodes can be adjusted to quickly restore data availability and reduce system downtime. In addition, the number of virtual nodes can be dynamically adjusted according to the load situation to adapt to storage requirements of different scales and loads.

[0065] Through the consistent hashing algorithm and virtual node mechanism, this embodiment can achieve efficient storage node mapping and data block distribution, optimize data access efficiency, and improve the fault tolerance and scalability of the system. This technical solution provides reliable theoretical support for the construction of distributed storage systems and provides an effective solution for large-scale, highly available data storage systems.

[0066] In S4, in this embodiment, a redundant coding method is used to redundantly store data blocks on multiple storage nodes to improve the fault tolerance of the system and the availability of data. Redundant coding can ensure that data will not be lost when some storage nodes fail, thereby improving the reliability and stability of the system.

[0067] Redundant coding technology generates multiple copies by encoding data, ensuring that even if some copies are lost or damaged, the system can still recover the data through other copies. In distributed storage systems, redundant coding is widely used to improve the fault tolerance of the system, especially when a node or data block fails.

[0068] Common redundant coding methods include replication coding, RS coding (Reed-Solomon coding), etc. In this embodiment, Reed-Solomon coding is used, which has high fault tolerance and is widely used in distributed storage environments. By splitting and encoding a data block into multiple redundant blocks, the risk of data loss can be effectively reduced.

[0069] Reed-Solomon coding is a coding method based on finite fields that can effectively correct errors or losses in data. Specifically, Reed-Solomon coding divides the original data and generates several redundant data blocks. When a data block is lost or damaged, the system can use the remaining redundant data blocks to repair it.

[0070] Assume that the data block is divided into k data blocks , and generate m redundant data blocks , through Reed-Solomon coding, a coding group containing a total of k+m data blocks is formed. These data blocks include original data blocks and redundant data blocks. In order to be able to restore the original data, when a system failure occurs, only any number of data blocks are needed to restore all the original data.

[0071] Specifically, the mathematical formula of Reed-Solomon coding is: ; Where C represents the encoded data set, which contains k data blocks and m redundant blocks. The generation process of redundant blocks can be realized through finite field operations, and the specific method is as follows: ; in, is the value of the original data block, is the pre-computed encoding matrix, The details depend on the encoding method and the chosen finite field.

[0072] According to the principle of Reed-Solomon coding, in this embodiment, the redundant data blocks are stored on multiple storage nodes to ensure the availability of data. When a storage node fails, the system can recover the data through the redundant data blocks on other nodes. , and redundant data blocks Distribute to multiple storage nodes. Assume that the system contains N storage nodes, map each data block to N nodes, and distribute them reasonably according to the node load and bandwidth.

[0073] For example, suppose the storage node , data block and redundant data blocks Will be mapped to multiple nodes. In order to achieve redundant storage of data, the following mapping method can be used: Block Assign to Node .

[0074] Redundant data blocks Assign to Node .

[0075] In this way, data blocks and redundant data blocks are distributed among multiple nodes, ensuring that when a node fails, it can be recovered through the remaining data blocks. Generally, the number of redundant blocks m is set according to the system's fault tolerance requirements and failure probability.

[0076] Redundant coding can also improve the scalability of the system. When the system is expanded, new storage nodes can be implemented by introducing new redundant data blocks. By properly selecting the number and allocation method of redundant blocks, the fault tolerance of the system can be improved without significantly increasing the amount of data migration. Specifically, when the system is expanded, redundant blocks can be allocated to new nodes and regenerated through Reed-Solomon coding.

[0077] For example, suppose the current system contains N nodes and uses m redundant blocks. When the system needs to be expanded to N +1 When adding a new node, some redundant data blocks can be mapped to the newly added node. In this way, data availability can be guaranteed during the expansion process and data migration and redistribution can be minimized.

[0078] In some embodiments, redundant coding is not limited to fault tolerance of storage nodes, but can also be used to optimize the efficiency of data transmission. For example, in the process of cross-regional data transmission, redundant coding can effectively reduce the risk of data loss and reduce the delay of data recovery through redundant data blocks. In addition, by adjusting the number of redundant blocks, it can flexibly adapt to different loads and storage requirements and improve the overall performance of the system.

[0079] Through the application of redundant coding, especially Reed-Solomon coding, this embodiment can realize efficient fault tolerance mechanism and data recovery capability in the distributed data storage system. This technical solution improves the reliability of the system and maintains high data availability in scenarios such as node failure and system expansion. It is an important technical support for building a highly fault-tolerant and highly reliable distributed storage system.

[0080] In this embodiment, it is proposed to adjust the load distribution strategy of the storage node through the game theory model, and to adjust the load in combination with the optimal control method, so as to achieve load balancing and optimal resource allocation of the distributed storage system. The goal of this step is to enable the system to dynamically adjust the load distribution between storage nodes according to the load status of the nodes and the network conditions during operation, thereby improving the overall response speed and throughput of the system and reducing the overload risk of a single node.

[0081] Game theory is a method for analyzing multiple participants, which can effectively describe how multiple storage nodes compete and coordinate for resources. In a distributed storage system, each storage node can be regarded as a "player", and the load distribution strategy is the choice of each storage node in the game. Specifically, the game theory model can be used to predict and optimize the strategies of each node in the process of resource competition, so as to achieve the global optimal load distribution.

[0082] In the game theory model, the goal of each storage node is to optimize its own resource allocation as much as possible to maximize its own performance. In order to achieve load balancing, nodes need to cooperate and negotiate at every moment based on the current load status and request volume. In order to quantify the load status of the node, a utility function of each node can be defined to reflect its load status. Assume that the node The load is , whose utility function It can be expressed as: ; in, is a utility function, usually a monotonically decreasing function, indicating that when the load increases, the utility of the node decreases. For example, the following form of utility function can be selected: ; in, and is a constant that controls the impact of node load on utility. Through this utility function, the system can quantify the load pressure of each node and then conduct game analysis.

[0083] In the game theory model, the load distribution decision of a storage node depends not only on its own load status, but also on the load status of other nodes. The goal of each storage node is to minimize its own load while ensuring load balance for the entire system. This can be described by the Nash equilibrium of the following game: ; in, Represents a storage node and The cost function of the load between nodes. Generally speaking, there is a cost for load transmission between nodes, and when the load of a node is too high, it may affect the efficiency of data interaction with other nodes. This mutually influential relationship.

[0084] In order to ensure that the game theory model can effectively optimize the load distribution between nodes in a dynamic environment, the optimal control theory is further introduced to optimize the load adjustment. The optimal control theory aims to adjust the control strategy based on the current system state (such as the load state of each node) and the future state prediction so that the system reaches the global optimum. The optimal control method is usually optimized based on the dynamic system model and the cost function.

[0085] In this embodiment, the optimal control method can be used to dynamically adjust the load distribution of storage nodes to achieve system load balancing. Assume that the load state of the system can be represented by the vector It indicates that the optimal control objective is to minimize the total load of the system while taking into account the cost of load transmission.

[0086] Specifically, the optimal control problem can be expressed as the following optimization problem: ; in, and is a weight parameter, which reflects the cost of a single node load and the load transmission between nodes, and T is the optimization time range. The goal of this optimization problem is to adjust the node load , making the load distribution of the system as balanced as possible while minimizing the load transmission cost.

[0087] To solve this optimization problem, the Lagrange multiplier method can be used to introduce system constraints (for example, the load of a node cannot exceed its maximum carrying capacity), and the optimal strategy for load distribution can be obtained through dynamic programming methods.

[0088] During the dynamic load adjustment process, the system continuously monitors the real-time load status of each node and adjusts it through the optimal control method. For example, when the load of a node is close to its maximum carrying capacity, the system will dynamically adjust the load distribution of other nodes according to the current load status and request volume to avoid overloading of a single node. The load adjustment process can be achieved through the following steps: Status monitoring: The system periodically obtains the load status of each node .

[0089] Optimal control solution: Based on the load status of the node, the optimal load distribution strategy is calculated through the optimal control method .

[0090] Load distribution adjustment: Based on the calculation results, adjust the load distribution between nodes to avoid excessive or low load on a single node.

[0091] Feedback adjustment: After the load distribution is adjusted, continue to monitor the load status of the node. If the load deviation is too large, re-enter the optimal control optimization process.

[0092] In some embodiments, the load distribution strategy can be combined with a machine learning algorithm for adaptive optimization to further improve the system's load prediction and regulation capabilities. For example, a prediction model is trained using historical load data to predict future node load trends, so that pre-adjustments can be made before the load changes. In addition, by introducing more complex game models (such as multi-stage game models), the system can make collaborative adjustments at multiple times to achieve load balancing over a long time scale.

[0093] By combining the game theory model with the optimal control method, this embodiment can achieve dynamic load balancing in the distributed storage system, ensure that the load of each node is optimally distributed, and improve the operating efficiency and stability of the system.

[0094] In response to S6, in this embodiment, it is proposed to combine the consistent hashing algorithm and the high-order graph model to minimize data migration when the system is expanded. The purpose of this step is to improve the efficiency and stability of the system by reducing the amount of data migration when the storage nodes are expanded or reconfigured in the distributed data storage system.

[0095] In a distributed storage system, expansion usually means adding new storage nodes, which will lead to the reallocation of some data blocks. In order to maintain system performance and avoid network load and delay caused by data migration, this embodiment optimizes data migration during system expansion by combining a high-order graph model with a consistent hashing algorithm to minimize the amount of data migration.

[0096] The consistent hashing algorithm is a classic method for solving the problem of data redistribution in distributed systems. Its basic idea is to map storage nodes and data blocks to a hash ring. When the system expands and new storage nodes are added, only some of the data blocks mapped to the newly added nodes need to be migrated. In this way, the amount of data migration can be minimized, full redistribution can be avoided, and the efficiency of the expansion process can be improved.

[0097] In this embodiment, specifically, the new storage node will be mapped to the position on the hash ring through the consistent hashing algorithm. The mapping relationship between the data block and the storage node is determined by the hash function, and the new storage node is only responsible for taking over the data blocks in the adjacent area on the hash ring. This mapping method can ensure that as the system expands, only a small amount of data needs to be migrated, while most data blocks remain unchanged, thereby reducing the overhead caused by data migration.

[0098] In the consistent hashing ring, suppose there are N storage nodes, data blocks is mapped to a hash value , if the hash value Falling on the node Within the responsible interval, the data block Stored on the node When adding a new node When joining, it only takes over some of the data blocks adjacent to its hash value on the hash ring. Therefore, the amount of data migration is: ; Where N is the number of storage nodes in the system, and the original data volume is the amount of data stored in the original storage node. In this way, the addition of a new node only requires the migration of a small amount of data, which greatly reduces the migration overhead.

[0099] The high-order graph model is used to describe the relationship between storage nodes and data blocks and the data flow between nodes. For each storage node, the edges in the graph model represent the storage location of the data block and the transmission of query requests. The node load status, the distribution of storage blocks, and the distribution of query requests can all be modeled through the edges and nodes in the graph.

[0100] During the expansion process, the introduction of high-order graph models can help analyze how to optimize the distribution of data blocks between storage nodes after new storage nodes are added. In particular, the shortest cutting path between different nodes and data blocks can be analyzed through the minimum cut theorem to determine which data blocks have the lowest migration cost and which data blocks should be migrated to new nodes.

[0101] In actual applications, when a new storage node is added to the system, the load of each storage node can be re-evaluated through a high-order graph model. By calculating the load difference of each storage node, the system can determine the optimal data block migration plan to ensure load balancing and minimize the migration cost.

[0102] Specifically, assume that the set of storage nodes in the system is , the data block set is , each data block The corresponding storage node is , the optimal strategy for data block migration can be described by the following minimum cut theorem: ; in, For data blocks From the storage node Migrate to By calculating the cut cost between each pair of nodes, the system can choose the nodes with the lowest migration cost to migrate data, thereby ensuring load balancing and minimizing the amount of data migration during the expansion process.

[0103] In the process of system expansion, in addition to minimizing data migration through consistent hashing algorithms, it is also necessary to monitor and adjust the expanded load in real time through high-order graph models. Specifically, the system can use load balancing algorithms to dynamically adjust the load distribution of each node based on the load status of each node. By introducing load balancing strategies and optimization of high-order graph models, resource contention and load imbalance problems during the expansion process can be further reduced.

[0104] For example, the system can use the shortest path algorithm and minimum cut theorem in graph theory to analyze the distribution of data blocks and query requests in real time, calculate the most appropriate data migration plan, and ensure the stability and efficiency of data transmission during the expansion process.

[0105] In some embodiments, in addition to using a combination of consistent hashing algorithms and high-order graph models, a virtual node mechanism can be introduced to further optimize the data migration process. The introduction of virtual nodes can distribute the load of actual storage nodes to multiple virtual nodes, thereby balancing the load between nodes and preventing a certain node from becoming a bottleneck.

[0106] In addition, as the system continues to expand, the system's storage scale and the complexity of load distribution may increase. Therefore, the use of machine learning algorithms for load prediction and optimization may further improve the efficiency and accuracy of the expansion process. For example, by training models to predict system load status, load distribution and data migration can be performed in advance, which can reduce the system pressure caused by sudden loads.

[0107] Through the above technical solution, this embodiment can combine the consistent hashing algorithm and the high-order graph model during the expansion of the distributed data storage system, minimize data migration, improve the scalability and stability of the system, and thus achieve efficient utilization of resources and optimization of system performance.

[0108] In general, the present invention provides a data index construction method for distributed data storage, which optimizes the allocation of storage nodes and data blocks, load balancing and data migration processes by combining advanced technologies such as high-order graph theory, consistent hashing algorithm, redundant coding and game theory. Specifically, the load of storage nodes and query requests is optimized by constructing a hypergraph model and the minimum cut theorem, the consistent hashing algorithm is used to reduce data migration and improve scalability, redundant coding is used to ensure data security, and game theory and optimal control theory are combined to optimize load distribution. This method can not only effectively handle data distribution and query efficiency problems in distributed storage systems, but also minimize data migration during system expansion and improve system stability and performance.

[0109] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a data index for distributed data storage, characterized in that: The following steps are involved: A hypergraph model of a distributed storage system is constructed by high-order graph theory, wherein the hypergraph model includes multiple storage nodes, multiple data blocks, and multiple query requests; Load balancing optimization of storage nodes and query requests based on the minimum cut theorem in the high-order graph model; Map data blocks and storage nodes to the consistent hash ring according to the consistent hash algorithm, and reduce data migration through the virtual node mechanism; A redundant encoding method is used to redundantly store data blocks on multiple storage nodes; The load distribution strategy of storage nodes is adjusted through game theory model, and the load adjustment is optimized by using optimal control method; When the system expands, the consistent hashing algorithm and high-order graph model are combined to minimize data migration.

2. The method for constructing a data index for distributed data storage according to claim 1, characterized in that: Each hyperedge in the high-order graph theory model represents the dependency relationship between multiple data blocks and multiple storage nodes or query requests, and the cutting cost of the hyperedge is optimized through the minimum cut theorem to ensure the shortest query path and achieve load balancing.

3. The method for constructing a data index for distributed data storage according to claim 1, characterized in that: In the consistent hashing algorithm, storage nodes and data blocks are mapped to a consistent hashing ring through hash values, each storage node is mapped to the hashing ring through multiple virtual nodes, and the load of the node is optimized through the virtual node mechanism.

4. The method for constructing a data index for distributed data storage according to claim 1, characterized in that: The redundant coding adopts Reed-Solomon coding, stores each data block in multiple storage nodes, and further reduces the data storage overhead through data compression method.

5. The method for constructing a data index for distributed data storage according to claim 1, characterized in that: The game theory model adjusts the load distribution strategy of the storage nodes through the Nash equilibrium principle, so that each storage node selects the optimal load distribution solution while considering the loads of other nodes.

6. The method for constructing a data index for distributed data storage according to claim 1, characterized in that: The optimal control method comprises the following steps: Define the system load state variables , represents the load status of each storage node at time t. Assuming that the system contains N storage nodes, the load status variable It represents the load status of node i at time t. The update of load status depends on the number of requests processed by the node, the amount of data stored, and the query load factor. Specifically: ； in, is the computing power of the storage node, is the query request volume of node i, The amount of data stored for node i; Set the system's load optimization objective function J, which describes the load distribution of all nodes and minimizes the load imbalance of the entire system by adjusting the node load. The load optimization objective function J represents the cumulative load cost within the time window T, specifically: ； Among them, the integral symbol is the accumulation of the entire system load cost in the time interval [0,T], is the load cost function of the node, reflecting the increase in cost as the node load increases, and T is the optimization time window; Solve the optimal control strategy for load distribution through optimal control theory , Where is the load adjustment amount of storage node i at time t, and the control goal is to dynamically adjust , so that the system load is minimized within a given time window and the load balancing constraints are met: ； The constraints include: ； in, is the total system load, which represents the sum of the loads of all storage nodes. is the load limit of a single node, and the constraint The load of each storage node i must be less than or equal to the maximum load limit ， Indicates that the constraint condition applies to every storage node i; Based on the Lagrange multiplier method or variational method, by solving the optimal control problem in the above formula, the optimal load adjustment strategy for each storage node i is obtained , so that the system can achieve load balancing on a global scale and minimize the overall load cost; According to the optimal control strategy solved , dynamically adjust the load distribution of each storage node, so that each storage node performs corresponding load adjustment under the constraints of load balancing and performance optimization, thereby achieving the global optimal load distribution of the system.

7. The method for constructing a data index for distributed data storage according to claim 1, characterized in that: When the system is expanded, the consistent hashing algorithm and the high-order graph model are combined to minimize data migration, including the following steps: The data mapping on the hash ring is adjusted through the consistent hashing algorithm, combined with the load balancing strategy of the high-order graph model to minimize the amount of data migration between nodes during the expansion process.

8. The method for constructing a data index for distributed data storage according to claim 7, characterized in that: The load balancing optimization adopts a graph cutting and reorganization strategy based on high-order graph theory. When the node load is uneven, the load distribution is optimized by adjusting the dependency between the data blocks and the storage nodes.

Images