Multi-database deployment method based on server cluster

Abstract

The invention relates to the technical field of computers, and discloses a multi-database deployment method based on a server cluster, which comprises the following steps of: distributing transaction state information for each database node, and realizing data consistency among the nodes in the database cluster by utilizing a consistency operator; a data synchronization path is optimized according to the topological structure between the nodes, and an optimal path is selected to reduce data transmission delay; the synchronization data volume is reduced through an increment synchronization and redundancy elimination method; designing and implementing a fault recovery strategy to ensure that the normal operation of the system is rapidly recovered when the database node has a fault; and dynamically adjusting a synchronization strategy and a consistency level of the database cluster according to a system load condition. By adopting the technical scheme based on transaction state vector allocation and consistency operator mapping, the technical effect of efficiently realizing data consistency in a distributed database environment is achieved. The problems that the consistency maintenance efficiency is low and data conflicts are prone to occurring in a high-concurrency scene are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-database deployment method based on a server cluster. Background Art

[0002] With the rapid development of big data and cloud computing technologies, the deployment of multiple databases in distributed systems is becoming more and more widespread. By storing and processing data together through multiple database nodes, distributed management of large-scale data can be achieved, improving the concurrent processing capabilities of the system.

[0003] In the prior art, data consistency between multiple database nodes usually relies on fixed rules or is implemented based on consistency protocols (such as the two-phase commit protocol). However, fixed consistency maintenance rules lack the ability to respond to dynamic loads and real-time transaction status. When the load differences between nodes are significant, this consistency maintenance method often leads to low transaction processing efficiency and even causes system performance bottlenecks; and some existing distributed consistency protocols (such as the two-phase commit protocol) need to frequently lock resources and wait for confirmation, resulting in serious performance problems in high-concurrency scenarios. This results in fixed rules and high protocol overhead, and due to the complexity of transactions in a distributed environment, it is difficult for the prior art to strike a balance between consistency and performance, especially in high-load or large-scale cluster scenarios. This defect is particularly prominent. For this reason, those skilled in the art have proposed a multi-database deployment method based on a server cluster to solve the above problems. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a multi-database deployment method based on a server cluster, which solves the problem that data consistency among multiple database nodes usually relies on fixed rules or is based on a consistency protocol.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a multi-database deployment method based on a server cluster, comprising the following steps: Assign transaction status information to each database node and use consistency operators to achieve data consistency among nodes in the database cluster; Optimize the data synchronization path according to the topological structure between nodes and select the optimal path to reduce data transmission delay; Reduce the amount of synchronized data through incremental synchronization and redundancy elimination methods; Design and implement fault recovery strategies to ensure rapid recovery of the system when a database node fails; Dynamically adjust the synchronization strategy and consistency level of the database cluster according to the system load.

[0006] Preferably, the transaction status information of the database node is represented by a high-dimensional vector, which specifically includes the following steps: The transaction status information of each database node is represented by a multidimensional vector, where each dimension of the vector corresponds to a specific transaction status attribute, including transaction commit status, isolation level, and data modification status; Combine the transaction states of all database nodes into a high-dimensional state space, with the state vector of each node as a point in the space; The transaction state vector of the database node is dynamically updated as the transaction is executed; After the transaction is committed, the state vector of the database node is synchronized and adjusted according to the consistency operator.

[0007] Preferably, the consistency operator maps the transaction state of each database node by defining a multi-scale consistency model, which specifically includes the following steps: Abstract the transaction status information of each database node into a state point in a high-dimensional space; To achieve data consistency between database nodes, a mapping rule is defined to map the transaction status of each database node to a unified global status; Based on the transaction state space, the global consistency state is calculated using the consistency mapping rules; According to the global consistency state, the transaction state of each database node is adjusted at multiple scales; After the transaction operation is completed, the consistency operator is used to perform consistency check on each node in the database cluster.

[0008] Preferably, the optimized data synchronization path is calculated by a shortest path algorithm based on synchronization delay weights. The path selection is weighted according to the synchronization delay between nodes, and the path with the smallest transmission delay is preferentially selected. The cost function of the path selection is calculated by the following formula: ,in, For path The synchronization cost, For Node To Node The weight between To represent all nodes in the system and The incremental data volume To sum, For synchronization delay, choose the synchronization path with the minimum cost As the optimal data synchronization path.

[0009] Preferably, the shortest path algorithm uses the Dijkstra algorithm, which selects the optimal path according to the synchronization delay between database nodes. The specific steps are: Based on the synchronization delay between each pair of nodes Calculate the initial distance and form the edge weight of the graph; Starting from the source node, the path with the minimum synchronization delay is recursively selected through the shortest path selection algorithm until all nodes are traversed to obtain the optimal data transmission path.

[0010] Preferably, the data synchronization method adopts an incremental synchronization mode, synchronizes only the changed data blocks, evaluates data redundancy by the Shannon entropy method in information theory, and compresses redundant data by using Huffman coding to reduce the bandwidth required for synchronization, specifically comprising the following steps: Calculate a unique identifier for each database node's storage data block to identify whether the data block has changed; The detected changed data block set is used as the incremental synchronization data set; The Shannon entropy method is used to evaluate the redundancy of each data block. The data block with a lower Shannon entropy value has a higher redundancy and is suitable for further compression. Select the optimal data transmission path based on the topological relationship and synchronization delay between database nodes; Synchronize the compressed incremental data to the target node through the selected path; After data synchronization is completed, the integrity and accuracy of the synchronized data are verified by recalculating and comparing the hash value of the target node.

[0011] Preferably, the fault recovery strategy is designed based on a Markov decision process model, and by analyzing the transition probability matrix of the system state, an optimal recovery path is selected to restore the system to a normal operating state when a database node fails. Specifically, the following steps are included: Divide the possible operating state of each node in the database cluster into multiple discrete states; Based on historical monitoring data or real-time observation results, the probability of the system transferring from one state to another is calculated to form a transition probability matrix; Define reward or penalty values ​​for each state transition to evaluate the pros and cons of the system state transition; The optimal recovery strategy is calculated using a Markov decision process model, which selects the recovery path by maximizing the long-term utility of the system; When a failure occurs, repair operations are performed in real time according to the optimal recovery strategy, including data resynchronization, node restart or task transfer; After the repair operation is completed, verify the system status.

[0012] Preferably, the optimal recovery strategy calculates the value of each state through a value iteration algorithm, and the specific steps include: The state is calculated by the following recursive formula The value of : ,in, For all possible operations To optimize, Status Take Action After the immediate return, is the discount factor, From the state Transfer to state The probability of is the current state, indicating the specific state of the system. The target state, which means from the current state The next state after the transition, For all possible target states The sum of; by iteratively calculating the maximum value path, the optimal recovery operation is selected.

[0013] Preferably, the method dynamically adjusts the consistency level according to the system load, and uses a nonlinear activation function to automatically adjust the consistency to balance performance and consistency requirements, specifically comprising the following steps: Collect real-time operating indicators of each database node in the system; Divide the consistency level into multiple levels; Through historical load data and system performance analysis, set upper and lower load thresholds for each node; Use nonlinear activation functions to smooth the real-time load and calculate activation values ​​to adjust the consistency level; Dynamically assign consistency levels to each database node based on the calculated activation value; After dynamically adjusting the consistency level, monitor the system operation status to detect whether anomalies or performance bottlenecks are caused by the consistency adjustment.

[0014] Multi-database deployment system based on server cluster, including: Multiple database nodes: Multiple independent database nodes jointly store and process data, ensuring distributed data storage and supporting parallel operations; A server cluster consists of multiple servers. The cluster management system coordinates the work of these servers, provides computing and storage resources, and achieves load balancing and high availability. The data synchronization module is responsible for synchronizing data between database nodes. It adopts an incremental synchronization strategy and only synchronizes changed data blocks to improve synchronization efficiency and reduce bandwidth usage.

[0015] The present invention provides a multi-database deployment method based on a server cluster. It has the following beneficial effects: 1. The present invention adopts a technical solution based on transaction state vector allocation and consistency operator mapping to achieve the technical effect of efficiently realizing data consistency in a distributed database environment. Compared with the consistency management solution that relies on fixed synchronization rules or manual intervention in the prior art, it solves the problem of low consistency maintenance efficiency and easy data conflicts in high concurrency scenarios.

[0016] 2. The present invention adopts incremental synchronization and redundancy elimination technical solutions, which detects data block changes and synchronizes only the changed parts, and combines the Shannon entropy method and Huffman coding to compress redundant data blocks, thereby achieving the technical effect of significantly reducing the amount of synchronized data and network bandwidth occupancy. Compared with the method of fully synchronizing data in the prior art, it solves the problems of large bandwidth occupancy and low transmission efficiency.

[0017] 3. The present invention achieves the technical effect of quickly restoring the system to a normal state when a node fails by designing a fault recovery strategy based on the Markov decision process (MDP) model and using a value iteration algorithm to generate an optimal recovery path. Compared with the fault recovery strategy in the prior art that relies on a fixed recovery scheme or a single redundant design, it solves the shortcomings of slow recovery speed and low resource utilization.

[0018] 4. The present invention adopts a dynamic consistency adjustment technical solution based on load scoring and nonlinear activation function, which achieves the technical effect of balancing performance and consistency under different load conditions. Compared with the solutions in the prior art with fixed or manually adjusted consistency strategies, it solves the problems of poor performance and inflexible adjustment in high-load or low-load scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flow chart of the steps of the multi-database deployment method based on a server cluster of the present invention; Figure 2 This is a structural diagram of the multi-database deployment system based on server clusters of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Please refer to the attached Figure 1The embodiment of the present invention provides a multi-database deployment method based on a server cluster, comprising the following steps: S1. Allocate transaction status information to each database node and use consistency operators to achieve data consistency among nodes in the database cluster. Specifically, the goal of step S1 is to achieve data consistency management among multiple database nodes by allocating transaction status information to each database node and combining consistency operators to perform global consistency mapping of transaction status. In the multi-database deployment method based on server clusters, the accurate allocation of transaction status information and the application of consistency operators are the basis for achieving cluster consistency and transaction correctness. This step provides methodological and structural technical support to ensure the efficient operation of database clusters in distributed environments.

[0022] Specifically, this step combines the real-time status and operation load between database nodes to model, calculate, and dynamically adjust transaction information, thereby optimizing data consistency.

[0023] In this embodiment, the allocation of transaction status information specifically includes: As an option, the transaction state information of each database node consists of core attributes such as transaction commit state, isolation level, and data modification state. For example, these attributes are organized into a high-dimensional transaction state vector , indicating a node The current state of affairs. Specifically: Transaction commit status : Used to identify whether the current transaction has been committed. A value of 0 indicates uncommitted, and a value of 1 indicates committed.

[0024] Isolation Level : Indicates the concurrency control strategy of the transaction, supporting the following isolation levels: READ UNCOMMITTED READ COMMITTED Repeatable Read SERIALIZABLE.

[0025] Data modification status : Indicates whether the transaction contains data modification operations. A value of 0 indicates no modification, and a value of 1 indicates modification.

[0026] In a possible implementation, the expression of the transaction state vector is as follows: It can be understood that the transaction state vector is updated in real time by dynamically monitoring the running state of the node to reflect the latest transaction status of the node.

[0027] In this embodiment, the implementation of the consistency operator includes: It should be noted that the consistency operator is used to achieve data consistency between multiple database nodes by mapping transaction status information to a global consistency state. Specifically: In some embodiments, the consistency operator first assigns a weight to each node ,The weights are dynamically adjusted based on factors such as the node’s load, network latency, and importance.

[0028] For example, the global consistency state The calculation method is: ; in, is the number of database nodes, For Node The weight of It is the transaction status vector of the node.

[0029] As an option, the consistency operator can also implement fine-grained consistency adjustments based on a multi-scale model. For example, when the node load is high, only the key nodes are constrained for consistency; when the load is low, all nodes are constrained for global consistency.

[0030] In this embodiment, the dynamic consistency check includes: During the transaction execution process, the consistency operator verifies in real time whether the transaction status of each node matches the global consistency status. Specifically: If the transaction status of a node deviates from the global consistency state, the system will trigger the consistency recovery mechanism to correct the deviation through transaction compensation or rollback operations.

[0031] In a possible implementation, the consistency check can be implemented by calculating the degree of deviation between the node state vector and the global state. If the deviation value exceeds a set threshold, a corresponding recovery operation is performed.

[0032] It should be noted that this consistency verification mechanism ensures the transaction correctness of the database cluster in a high-concurrency environment, while reducing the risk of data conflicts caused by consistency issues.

[0033] Through the detailed description of the above implementation method, those skilled in the art can easily reproduce the technical solution of this step according to the definition and allocation of transaction status information, the implementation mechanism of the consistency operator and the logic of dynamic consistency verification.

[0034] S2, optimize the data synchronization path according to the topological structure between nodes, and select the optimal path to reduce data transmission delay; Specifically, the purpose of step S2 is to optimize the data synchronization path by analyzing the topological structure between database nodes, combining the network delay and bandwidth between nodes, and selecting the transmission path with the minimum delay to improve the efficiency and accuracy of data transmission. It should be noted that the optimization of the data synchronization path is the core technical means to achieve efficient data transmission, which is of great significance for ensuring the consistency and performance of the database cluster.

[0035] Specifically, this step uses a graph model to model the connection relationship of database nodes, and combines the shortest path algorithm to dynamically calculate the optimal synchronization path between nodes. The implementation of topology optimization combines real-time monitoring and dynamic adjustment mechanisms to adapt to changes in system load and network conditions.

[0036] In this embodiment, topology modeling includes: As an option, the connections between database nodes can be represented by a weighted undirected graph represents, where: It is a collection of database nodes, each of which represents an independent database instance; is the edge between nodes, indicating the network connection between nodes; Edge weight Representation Node and nodes The synchronization delay between

[0037] It should be noted that network delay It is determined by factors such as bandwidth, physical distance, and transmission error rate. In one possible implementation, the delay weight can be calculated according to the following formula: ; in, For Node To Node The delay weight between is the weight coefficient, indicating the importance of network delay in the comprehensive cost; is the weight coefficient, indicating the importance of network bandwidth in the comprehensive cost; is the weight coefficient, which indicates the importance of the transmission error rate in the comprehensive cost; For Node To Node The network delay between , usually measured in milliseconds (ms); For Node To Node The network bandwidth between the two, usually measured in megabits per second (Mbps); For Node and nodes The transmission error rate between .

[0038] It can be understood that the above topology modeling provides basic data support for the optimization of data synchronization paths and ensures the accuracy of path selection.

[0039] The path selection is weighted according to the synchronization delay between nodes, and the path with the smallest transmission delay is preferred. The cost function of path selection is calculated by the following formula:

[0040] in, For path The synchronization cost, For Node To Node The weight between To represent all nodes in the system and The incremental data volume To sum, For synchronization delay, choose the synchronization path with the minimum cost As the optimal data synchronization path.

[0041] In this embodiment, the optimal path selection includes: Specifically, based on the above topological structure, the shortest path from the source node to the target node is calculated by the Dijkstra algorithm. Exemplarily, the execution process of the Dijkstra algorithm is as follows: First, set the initial path length for each node, the path length of the source node is set to 0, and the path lengths of other nodes are set to infinity; Then, select the node with the shortest path length from the unvisited nodes and mark it as visited; Next, update the path lengths of all adjacent nodes of the node. If the path length through the node is shorter than the current value, update its path length; Finally, repeat the above steps until all nodes are visited.

[0042] Alternatively, the path length update is calculated as: ; in, The target node The current path length, For the current node The path length, For Node and The weight between Representation Node To Node The edge weights between arrive Path cost, Indicates the current path length and from arrive Path length to the source node The smaller value of .

[0043] It should be noted that the calculation result of the Dijkstra algorithm is a set of shortest paths from the source node to the target node, which can ensure that the delay of data synchronization is minimized.

[0044] In this embodiment, the dynamic adjustment of path optimization includes: In some embodiments, in order to adapt to changes in system load and network conditions, a real-time monitoring mechanism can be introduced to dynamically adjust the synchronization path. Specifically: Monitor network latency and bandwidth between nodes in real time, and periodically update edge weights in the topology structure; When network conditions change significantly, the shortest path calculation is re-executed to dynamically adjust the synchronization path.

[0045] Exemplarily, when the delay or bandwidth of a path exceeds a set threshold, the system will select a suboptimal path as an alternative path to ensure the continuity of data synchronization.

[0046] Through the detailed description of the above implementation, those skilled in the art can easily reproduce the technical solution of this step based on topology modeling, optimal path selection and dynamic adjustment mechanism. The above method has significant advantages in improving data transmission efficiency, reducing latency and adapting to network changes.

[0047] S3, reduce the amount of synchronized data through incremental synchronization and redundancy elimination methods; Specifically, the goal of step S3 is to reduce the amount of data synchronization between database nodes and improve data transmission efficiency through incremental synchronization and data redundancy elimination technology. It should be noted that the optimization of data synchronization is of great significance for reducing network bandwidth usage and improving the overall performance of the database cluster.

[0048] Specifically, this step adopts an incremental synchronization strategy based on change detection, synchronizing only the changed data blocks, reducing the data transmission requirements from the source. At the same time, combined with redundancy elimination methods (such as Shannon entropy analysis and Huffman coding technology based on information theory), the amount of synchronized data is further compressed to achieve the purpose of efficient transmission.

[0049] In this embodiment, the implementation of incremental synchronization includes: Incremental synchronization uses change detection to identify data blocks that need to be synchronized. Specifically: By calculating the unique identifier of each data block (such as a hash value), the new and old versions of the data block are compared. If the hash values ​​of the two are inconsistent, the data block is marked as changed data.

[0050] For example, suppose a node's data block Update to the latest version , incremental synchronization first calculates the hash value of both and .like , it is determined that the data block has changed.

[0051] It can be understood that the above method can avoid repeated transmission of unchanged data, thereby significantly reducing the amount of data synchronization.

[0052] As an option, the incrementally synchronized data blocks can be sorted by priority, such as synchronizing data blocks with high transaction dependencies or data blocks that are sensitive to synchronization delays first, to ensure the consistency and real-time requirements of the system.

[0053] In this embodiment, the redundancy elimination method includes: It should be noted that redundancy elimination is an important means to further optimize the amount of synchronized data. In the present invention, the Shannon entropy method based on information theory is used to evaluate data redundancy, and data blocks with high redundancy are compressed. Specifically: Use the Shannon entropy formula to calculate the amount of information in each data block:

[0054] in, is a random variable Shannon entropy (information entropy), represents a discrete random variable, Indicates the symbol in the data block The probability of occurrence, Representing probability The base 2 logarithm of Represents a random variable The number of possible values ​​of , that is, the total number of events.

[0055] As an option, Huffman coding is used to compress data blocks with high redundancy. Huffman coding constructs an optimal prefix tree based on symbol frequency and assigns shorter codes to high-frequency symbols to achieve efficient data compression.

[0056] For example, suppose a data block The symbol distribution is , the result of Huffman coding is , the compression ratio can reach the ideal value.

[0057] It can be understood that the redundancy elimination method further reduces the amount of data transmission based on incremental synchronization, effectively improving the network utilization of the system.

[0058] In this embodiment, data synchronization execution includes: In a possible implementation, after completing redundancy elimination, the incremental data is transmitted to the target node through the optimal synchronization path. Specifically: During data transmission, streaming transmission is adopted to transmit data in blocks step by step to reduce memory usage and delay during transmission.

[0059] At the destination node, the received data blocks are decoded and merged with the existing database to ensure data consistency.

[0060] It should be noted that in order to improve the robustness of the transmission process, the system will perform real-time verification of the transmission process. If data loss or transmission errors are found, the system will re-request the transmission of the missing data blocks until the synchronization is completed.

[0061] Through the detailed description of the above implementation, those skilled in the art can easily reproduce the technical solution of this step based on the change detection method of incremental synchronization, Shannon entropy analysis and Huffman coding technology for redundancy elimination, and the optimization mechanism of data transmission. The above method has significant advantages in reducing the amount of data synchronization and improving data transmission efficiency.

[0062] S4. Design and implement a fault recovery strategy to ensure that the system can be quickly restored to normal operation when a database node fails; Specifically, the purpose of step S4 is to ensure that the database node can quickly recover to a normal operating state when a failure occurs by designing and implementing a fault recovery strategy, so as to improve the reliability and availability of the system. It should be noted that the key to the fault recovery strategy lies in the modeling of the system state, the generation of the optimal recovery path, and the execution of dynamic repair. These links together ensure the efficient recovery capability of the system.

[0063] Specifically, the present invention uses Markov decision process (MDP) to model the fault recovery strategy and generates the optimal recovery path through value iteration algorithm. When a fault occurs, the system dynamically performs the repair operation according to the generated recovery path and verifies and optimizes the repair result.

[0064] In this embodiment, system state modeling includes: As an option, the operating state of the system can be divided into multiple discrete states, including normal state, fault state and recovery state. Specifically: The normal state means that the database node is running normally and can process transactions and synchronize data; The failure state means that the database node stops running or its performance is significantly degraded due to hardware or network problems; The recovery state means that the database node is recovering from the failure and some of its functions may have been restored but further optimization is still needed.

[0065] In some embodiments, the transition of the system state is described by a state transition matrix. Specifically, the state transition probability , indicating that the system is in state arrive The state transition probability matrix can be generated based on historical data statistics or real-time monitoring information.

[0066] It should be noted that state modeling provides a data basis for the generation of the optimal recovery path, ensuring the scientificity and efficiency of the recovery strategy.

[0067] In this embodiment, the optimal recovery path generation includes: Specifically, the Markov decision process (MDP) model is used to calculate the value of each system state through the value iteration algorithm, and the optimal recovery path from the fault state to the normal state is generated. For example, the core calculation formula of the value iteration algorithm is: ;in, Indicates status the value of For all possible operations Optimize Indicates from the state By operation Immediate rewards for moving to other states; Indicates status Transfer to probability; is the discount factor, which indicates the impact weight of future rewards; is the current state, indicating the specific state of the system; The target state, which means from the current state The next state after the transfer; For all possible target states The sum of .

[0068] It can be understood that through the value iteration algorithm, the optimal value of each state can be calculated and the optimal recovery path under the fault state can be determined.

[0069] As an option, when calculating the optimal recovery path, the value function can be dynamically adjusted based on the system's resource usage and operation cost. For example, if a node's repair operation consumes a lot of resources, its reward value can be reduced to reduce resource usage.

[0070] In this embodiment, the dynamic repair execution includes: In one possible implementation, after detecting a node failure, the system will gradually perform repair operations based on the generated optimal recovery path. Specifically: In case of hardware failure, you can restart or migrate the node; For data inconsistency failures, you can perform data resynchronization or transaction rollback operations; In case of network failure, the backup path can be switched to ensure the continuity of data transmission.

[0071] It should be noted that during the dynamic repair process, the system will monitor the effect of the repair operation in real time. For example, by detecting the health status and transaction processing capacity of the node, it can determine whether the current repair has achieved the expected goal. If the repair effect is not ideal, the recovery path will be dynamically adjusted and the repair plan will be regenerated.

[0072] In this embodiment, the repair result verification includes: After the repair operation is completed, the system status is verified to ensure that the system has been fully restored to normal operation. Specifically: By comparing the system status before and after the repair, verify whether all faulty nodes have returned to normal; Use consistency check tools to check whether new data inconsistencies are introduced during the repair process; If it is found that there are still abnormal nodes, repeat the repair operation until the system is fully restored.

[0073] It should be noted that the repair result verification can effectively avoid system performance degradation or potential failure hazards caused by incomplete recovery.

[0074] Through the detailed description of the above implementation, those skilled in the art can easily reproduce the technical solution of this step based on system state modeling, optimal recovery path generation, dynamic repair execution and repair result verification. The above method has significant advantages in improving fault recovery speed and system reliability.

[0075] S5. Dynamically adjust the synchronization strategy and consistency level of the database cluster according to the system load.

[0076] Specifically, the goal of step S5 is to monitor the system load in real time and dynamically adjust the synchronization strategy and consistency level of the database cluster according to the load changes, so as to improve the overall performance while ensuring the consistency of the system. It should be noted that the system load directly affects the processing capacity of the database node and the network bandwidth utilization. By dynamically adjusting the strategy, a balance can be achieved between performance and consistency.

[0077] Specifically, this step includes three core links: load monitoring and analysis, consistency level adjustment, and synchronization strategy optimization. The above links work closely together to achieve efficient operation of the system under different load conditions.

[0078] In this embodiment, load monitoring and analysis include: In some embodiments, the system collects the operating status information of the database node in real time through a monitoring tool, including but not limited to the following indicators: CPU usage, used to evaluate computing resource usage; Memory usage, used to evaluate the storage buffer capacity of the node; Disk I / O rate, used to evaluate data read and write performance; Network bandwidth utilization, used to evaluate data transmission capacity.

[0079] It should be noted that in order to comprehensively evaluate the load of the node, the above indicators can be weighted to generate a load score. : ;in, Representation Node The overall load rating of Representation Node The CPU usage range is , Representation Node Memory usage, the value range is , Representation Node Disk I / O usage, the value range is , Representation Node The network bandwidth utilization rate is in the range of , , , , are the weight coefficients of CPU, memory, disk I / O, and network bandwidth, respectively, and meet the following conditions: .

[0080] For example, if the node The CPU utilization is 70%, the memory utilization is 80%, the disk I / O is 60MB / s, and the network bandwidth utilization is 50%. The load score of the node can be obtained by combining the weight calculation.

[0081] In this embodiment, the consistency level adjustment includes: As an option, the consistency level is dynamically adjusted according to the load score, including two typical modes: strong consistency and eventual consistency. Specifically: Strong consistency requires all nodes to keep data completely consistent when the transaction is completed, but it will increase network latency; Eventual consistency allows node data to be out of sync for a short period of time, thereby improving throughput and responsiveness.

[0082] In one possible implementation, the dynamic adjustment of the consistency level can be achieved through a nonlinear activation function. For example, a Sigmoid function is used to score the load. To perform smoothing:

[0083] in, Representation Node The activation function output value is in the range of , Representation Node The load rating, represents the steepness coefficient of the activation function, represents the load score threshold, Represents the base of natural logarithms, approximately equal to 2.718.

[0084] when When it is close to 0, the system prefers eventual consistency; when When it approaches 1, the system switches to strong consistency mode.

[0085] It can be understood that the above method realizes dynamic adjustment of the consistency level through the activation function, avoiding performance jitter caused by frequent switching.

[0086] In this embodiment, the synchronization strategy optimization includes: Specifically, adjust the data synchronization strategy according to the consistency level, including the following optimization measures: In strong consistency mode, full synchronization is used to ensure the real-time consistency of data on all nodes; In the eventual consistency mode, an incremental synchronization strategy is adopted to synchronize only the changed data blocks to reduce synchronization overhead.

[0087] As an option, the synchronization frequency can also be dynamically adjusted based on the network bandwidth of the node. For example: When network bandwidth is sufficient, increase the synchronization frequency to reduce the time that data is out of sync; When network bandwidth is limited, reduce the synchronization frequency to reduce bandwidth usage.

[0088] It should be noted that the optimization of the synchronization strategy can maximize the system's operating efficiency without sacrificing consistency.

[0089] Through the detailed description of the above implementation, those skilled in the art can easily reproduce the implementation process of this step based on the technical solutions of load monitoring and analysis, consistency level adjustment and synchronization strategy optimization. The above method has significant advantages in dynamically adjusting the balance between database cluster performance and consistency.

[0090] Please see attached Figure 2 ,The present invention also provides a multi-database deployment system based on a server cluster, including: multiple database nodes, multiple independent database nodes jointly store and process data, ensure distributed data storage and support parallel operations; A server cluster consists of multiple servers. The cluster management system coordinates the work of these servers, provides computing and storage resources, and achieves load balancing and high availability. The data synchronization module is responsible for synchronizing data between database nodes. It adopts an incremental synchronization strategy and only synchronizes changed data blocks to improve synchronization efficiency and reduce bandwidth usage. The fault recovery module, when a system failure occurs, quickly evaluates the system status and transition probability through the Markov decision process model and value iteration algorithm, selects the optimal recovery path, ensures rapid system recovery and reduces downtime.

[0091] Specifically, the multi-database deployment system based on the server cluster solves the problems of data consistency, efficient synchronization and fault recovery among multiple database nodes by designing multiple modules to work together, thus ensuring the high availability and high performance of the system. The following is a detailed description and expansion of each module: Multiple database nodes, each node independently stores part of the data and can independently process transaction operations. Specifically, the main functions of the database node include: Distributed data storage: The global data set is divided into multiple data shards, and each database node is responsible for storing a specific data shard. As an option, the sharding strategy can be based on the primary key range or hash value of the data to achieve a balanced storage load.

[0092] Parallel transaction processing: Each database node executes transaction operations independently, supports high concurrent access, and coordinates consistency with other nodes through transaction state vectors (such as commit status, isolation level, and modification status).

[0093] It should be noted that database nodes communicate with each other through high-speed network connections to support operations such as data synchronization and fault recovery. In one possible implementation, the communication protocol between nodes uses a lightweight transport protocol (such as gRPC) to reduce communication overhead and improve response speed.

[0094] The server cluster consists of multiple high-performance servers, which are centrally scheduled and managed through the cluster management system. Specifically, the functions of the server cluster include: Computing resource allocation: Dynamically allocate computing resources, such as CPU cores and memory usage, based on the load of the database node.

[0095] Storage resource scheduling: Manages storage resources in the server cluster, ensures balanced data distribution, and provides redundant storage to improve reliability.

[0096] As an option, the cluster management system monitors the load of each server in real time through the load balancing module and distributes tasks according to the load balancing algorithm. Exemplarily, the load balancing algorithm may adopt a weighted round-robin or minimum connection number strategy.

[0097] It is understandable that the high availability of server clusters is achieved through redundant design and failover mechanisms. For example, when a server fails, the cluster management system can automatically switch to a backup server to avoid affecting the overall operation of the system.

[0098] The data synchronization module is responsible for synchronizing data between database nodes to achieve consistency and data integrity. Specifically, the data synchronization module adopts an incremental synchronization strategy to synchronize only the changed data blocks, thereby significantly reducing the amount of data transmission. Its implementation methods include: Change detection: Detect whether the data has changed by calculating the hash value of the data block. If the hash value is different, mark the data block as a data block that needs to be synchronized.

[0099] Redundancy elimination: Redundancy elimination is performed on the data blocks to be synchronized. For example, the Shannon entropy method is used to evaluate the redundancy of the data blocks, and Huffman coding is applied to compress the data blocks with high redundancy, thereby further reducing the amount of synchronized data.

[0100] Synchronization path optimization: Select the optimal data synchronization path based on the network topology and transmission delay between nodes. Specifically, the shortest path between nodes can be calculated based on the Dijkstra algorithm to reduce synchronization delay and improve transmission efficiency.

[0101] It should be noted that, in order to ensure the accuracy of the synchronized data, the data synchronization module verifies the data integrity of the target node after synchronization is completed. For example, by recalculating the hash value of the data block of the target node and comparing it with the source node, verifying whether the data is consistent.

[0102] Fault recovery module When a system failure occurs, the fault recovery module of this system uses the Markov decision process (MDP) model and value iteration algorithm to quickly evaluate the system status and generate the optimal recovery path. Specifically, the functions of the fault recovery module include: System state modeling: The system state is divided into normal state, fault state and recovery state, and a state transition probability matrix is ​​generated based on historical data. The state transition probability represents the possibility of transitioning from one state to another and is used to guide the generation of recovery paths.

[0103] Optimal recovery path generation: The value of each state is calculated through the value iteration algorithm, and the optimal recovery path from the fault state to the normal state is generated. Exemplarily, the calculation formula of the value function is: ; Among them, among them, Indicates status the value of For all possible operations Optimize Indicates from the state By operation Immediate rewards for moving to other states; Indicates status Transfer to probability; is the discount factor, which indicates the impact weight of future rewards; is the current state, indicating the specific state of the system; The target state, which means from the current state The next state after the transfer; For all possible target states The sum of .

[0104] Dynamic recovery operations: Based on the optimal recovery path, recovery operations are performed step by step, such as node restart, data resynchronization, or task switching.

[0105] As an option, to improve the efficiency of fault recovery, the system can monitor the recovery effect and dynamically adjust the path while performing the recovery operation. For example, when the recovery effect of a node is not ideal, an alternative solution can be selected by recalculating the path.

[0106] Through the above detailed description, those skilled in the art can easily reproduce the implementation process of the system based on the technical solutions of database node design, server cluster management, data synchronization and fault recovery module provided by the present invention. The present invention has significant advantages in improving the performance, consistency and high availability of database clusters.

[0107] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-database deployment method based on a server cluster, characterized in that: The following steps are involved: Assign transaction status information to each database node and use consistency operators to achieve data consistency among nodes in the database cluster; Optimize the data synchronization path according to the topological structure between nodes and select the optimal path to reduce data transmission delay; Reduce the amount of synchronized data through incremental synchronization and redundancy elimination methods; Design and implement fault recovery strategies to ensure rapid recovery of the system when a database node fails; Dynamically adjust the synchronization strategy and consistency level of the database cluster according to the system load.

2. The multi-database deployment method based on server cluster according to claim 1, characterized in that: The transaction status information of the database node is represented by a high-dimensional vector, which specifically includes the following steps: The transaction status information of each database node is represented by a multidimensional vector, where each dimension of the vector corresponds to a specific transaction status attribute, including transaction commit status, isolation level, and data modification status; Combine the transaction states of all database nodes into a high-dimensional state space, with the state vector of each node as a point in the space; The transaction state vector of the database node is dynamically updated as the transaction is executed; After the transaction is committed, the state vector of the database node is synchronized and adjusted according to the consistency operator.

3. The multi-database deployment method based on server cluster according to claim 1, characterized in that: The consistency operator maps the transaction status of each database node by defining a multi-scale consistency model, which specifically includes the following steps: Abstract the transaction status information of each database node into a state point in a high-dimensional space; To achieve data consistency between database nodes, a mapping rule is defined to map the transaction status of each database node to a unified global status; Based on the transaction state space, the global consistency state is calculated using the consistency mapping rules; According to the global consistency state, the transaction state of each database node is adjusted at multiple scales; After the transaction operation is completed, the consistency operator is used to perform consistency check on each node in the database cluster.

4. The multi-database deployment method based on server cluster according to claim 1, characterized in that: The optimized data synchronization path is calculated by the shortest path algorithm based on synchronization delay weight. The path selection is weighted according to the synchronization delay between nodes, and the path with the smallest transmission delay is preferentially selected. The cost function of the path selection is calculated by the following formula: ; in, For path The synchronization cost, For Node To Node The weight between To represent all nodes in the system and The incremental data volume To sum, For synchronization delay, choose the synchronization path with the minimum cost As the optimal data synchronization path.

5. The multi-database deployment method based on server cluster according to claim 4 is characterized in that: The shortest path algorithm uses the Dijkstra algorithm, which selects the optimal path according to the synchronization delay between database nodes. The specific steps are: Based on the synchronization delay between each pair of nodes Calculate the initial distance and form the edge weight of the graph; Starting from the source node, the path with the minimum synchronization delay is recursively selected through the shortest path selection algorithm until all nodes are traversed to obtain the optimal data transmission path.

6. The multi-database deployment method based on server cluster according to claim 1, characterized in that: The data synchronization method adopts an incremental synchronization mode, synchronizes only the changed data blocks, evaluates data redundancy by the Shannon entropy method in information theory, and compresses redundant data by using Huffman coding to reduce the bandwidth required for synchronization. Specifically, the method includes the following steps: Calculate a unique identifier for each database node's storage data block to identify whether the data block has changed; The detected changed data block set is used as the incremental synchronization data set; The Shannon entropy method is used to evaluate the redundancy of each data block. The data block with a lower Shannon entropy value has a higher redundancy and is suitable for further compression. Select the optimal data transmission path based on the topological relationship and synchronization delay between database nodes; Synchronize the compressed incremental data to the target node through the selected path; After data synchronization is completed, the integrity and accuracy of the synchronized data are verified by recalculating and comparing the hash value of the target node.

7. The multi-database deployment method based on server cluster according to claim 1, characterized in that: The fault recovery strategy is designed based on the Markov decision process model. By analyzing the transition probability matrix of the system state, the optimal recovery path is selected to restore the system to a normal operating state when a database node fails. Specifically, the following steps are included: Divide the possible operating state of each node in the database cluster into multiple discrete states; Based on historical monitoring data or real-time observation results, the probability of the system transferring from one state to another is calculated to form a transition probability matrix; Define reward or penalty values ​​for each state transition to evaluate the pros and cons of the system state transition; The optimal recovery strategy is calculated using a Markov decision process model, which selects the recovery path by maximizing the long-term utility of the system; When a failure occurs, repair operations are performed in real time according to the optimal recovery strategy, including data resynchronization, node restart or task transfer; After the repair operation is completed, verify the system status.

8. The method for deploying multiple databases based on a server cluster according to claim 7, characterized in that: The optimal recovery strategy calculates the value of each state through a value iteration algorithm, and the specific steps include: The state is calculated by the following recursive formula The value of : ； in, Status Take Action After the immediate return, is the discount factor, From the state Transfer to state probability; The optimal recovery operation is selected by iteratively calculating the maximum value path.

9. The method for deploying multiple databases based on a server cluster according to claim 1, characterized in that: The method dynamically adjusts the consistency level according to the system load and uses a nonlinear activation function to automatically adjust the consistency to balance performance and consistency requirements. Specifically, the method includes the following steps: Collect real-time operating indicators of each database node in the system; Divide the consistency level into multiple levels; Through historical load data and system performance analysis, set upper and lower load thresholds for each node; Use a nonlinear activation function to smooth the real-time load and calculate the activation value to adjust the consistency level; Dynamically assign consistency levels to each database node based on the calculated activation value; After dynamically adjusting the consistency level, monitor the system operation status to detect whether anomalies or performance bottlenecks are caused by the consistency adjustment.

10. A multi-database deployment system based on a server cluster, applied to a multi-database deployment method based on a server cluster as claimed in any one of claims 1 to 9, characterized in that: include: Multiple database nodes: Multiple independent database nodes jointly store and process data, ensuring distributed data storage and supporting parallel operations; A server cluster consists of multiple servers. The cluster management system coordinates the work of these servers, provides computing and storage resources, and achieves load balancing and high availability. The data synchronization module is responsible for synchronizing data between database nodes. It adopts an incremental synchronization strategy and only synchronizes changed data blocks to improve synchronization efficiency and reduce bandwidth usage. The fault recovery module, when a system failure occurs, quickly evaluates the system status and transition probability through the Markov decision process model and value iteration algorithm, selects the optimal recovery path, ensures rapid system recovery and reduces downtime.

Images