Multi-database Deployment Method Based on Server Cluster

Abstract

The present invention relates to the field of computer technology, and discloses a multi-database deployment method based on a server cluster, including the following steps: allocating transaction status information to each database node, and using a consistency operator to achieve data consistency among nodes in the database cluster; optimizing the data synchronization path according to the topological structure among nodes, and selecting an optimal path to reduce data transmission latency; reducing the amount of synchronized data through an incremental synchronization and redundancy elimination method; designing and implementing a fault recovery strategy to ensure that the system can quickly resume normal operation when a database node fails; dynamically adjusting the synchronization strategy and consistency level of the database cluster according to the system load conditions. By adopting the technical solution based on the allocation of transaction state vectors and the mapping of consistency operators, the technical effect of efficiently achieving data consistency in a distributed database environment is achieved. The problems of low consistency maintenance efficiency and easy occurrence of data conflicts in high-concurrency scenarios are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a multi-database deployment method based on a server cluster. Background Art

[0002] With the rapid development of big data and cloud computing technologies, the application of multi-database deployment in distributed systems has become increasingly widespread. By jointly storing and processing data through multiple database nodes, distributed management of large-scale data can be achieved, and the concurrent processing capacity of the system can be improved.

[0003] In the prior art, data consistency between multiple database nodes usually relies on fixed rules or is achieved based on a consistency protocol (such as the two-phase commit protocol). However, the fixed consistency maintenance rules lack the ability to respond to dynamic loads and real-time transaction states. When the load difference between nodes is significant, this consistency maintenance method often leads to low transaction processing efficiency and even causes system performance bottlenecks; moreover, some existing distributed consistency protocols (such as the two-phase commit protocol) need to frequently lock resources and wait for confirmation, resulting in serious performance problems in high-concurrency scenarios. This leads to rule fixation and large protocol overheads, and due to the complexity of transactions in a distributed environment, it is difficult for the prior art to achieve a balance between consistency and performance, especially in high-load or large-scale cluster scenarios, where this defect is particularly prominent. For this reason, those skilled in the art have proposed a multi-database deployment method based on a server cluster to solve the above problems. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the present invention provides a multi-database deployment method based on a server cluster, which solves the problem that data consistency between multiple database nodes usually relies on fixed rules or a consistency protocol.

[0005] To achieve the above objectives, the present invention is realized through the following technical solutions: A multi-database deployment method based on a server cluster, including the following steps:

[0006] Assign transaction status information to each database node, and use a consistency operator to achieve data consistency among nodes in the database cluster;

[0007] Optimize the data synchronization path according to the topological structure between nodes, and select the optimal path to reduce data transmission latency;

[0008] Reduce the amount of synchronized data through incremental synchronization and redundancy elimination methods;

[0009] Design and implement a fault recovery strategy to ensure that the system can quickly resume normal operation when a database node fails;

[0010] Dynamically adjust the synchronization strategy and consistency level of the database cluster according to the system load situation.

[0011] Preferably, the transaction status information of the database node is represented by a high-dimensional vector, and the specific steps are as follows:

[0012] The transaction status information of each database node is represented by a multi-dimensional vector, and each dimension of the vector corresponds to a specific transaction status attribute, specifically including the transaction commit status, isolation level, and data modification status;

[0013] Combine the transaction statuses of all database nodes into a high-dimensional state space, and the state vector of each node is used as a point in the space;

[0014] The transaction status vector of the database node will be dynamically updated as the transaction is executed;

[0015] After the transaction is committed, synchronously adjust the state vector of the database node according to the consistency operator.

[0016] Preferably, the consistency operator maps the transaction status of each database node by defining a multi-scale consistency model, and the specific steps are as follows:

[0017] Abstract the transaction status information of each database node into a state point in a high-dimensional space;

[0018] To achieve data consistency between database nodes, define a mapping rule to map the transaction statuses of each database node to a unified global state;

[0019] Based on the transaction status space, calculate the global consistency state using the consistency mapping rule;

[0020] According to the global consistency state, perform multi-scale adjustment on the transaction statuses of each database node;

[0021] After the transaction operation is completed, perform consistency verification on each node in the database cluster through the consistency operator.

[0022] Preferably, the optimized data synchronization path is calculated by a shortest path algorithm based on synchronization delay weights. The selection of the path is weighted according to the synchronization delay between nodes, and the path with the minimum transmission delay is preferably selected. The cost function for path selection is calculated by the following formula: , where is the synchronization cost of path is the weight between node and node to node is the sum over all node pairs in the system in the system and the incremental data volume are summed up For the synchronization delay, select the synchronization path with the minimum cost as the optimal data synchronization path

[0023] Preferably, the shortest path algorithm uses the Dijkstra algorithm, which selects the optimal path according to the synchronization delay between database nodes. The specific steps are as follows:

[0024] According to the synchronization delay between each pair of nodes Calculate the initial distance to form the edge weights of the graph

[0025] Starting from the source node, recursively select the path with the minimum synchronization delay through the shortest path selection algorithm until all nodes are traversed to obtain the optimal data transmission path

[0026] Preferably, the data synchronization method adopts the incremental synchronization method, only synchronizes the changed data blocks, evaluates the data redundancy by the Shannon entropy method in information theory, and compresses the redundant data using Huffman coding to reduce the bandwidth required for synchronization. The specific steps include the following:

[0027] Calculate the unique identifier for the stored data blocks of each database node to identify whether the data blocks have changed

[0028] Take the detected set of changed data blocks as the incremental synchronization data set

[0029] Use the Shannon entropy method to evaluate the redundancy of each data block. The lower the Shannon entropy value of the data block, the higher the redundancy, and it is suitable for further compression

[0030] According to the topological relationship and synchronization delay between database nodes, select the optimal data transmission path

[0031] Synchronize the compressed incremental data to the target node through the selected path

[0032] After the data synchronization is completed, verify the integrity and accuracy of the synchronized data by recalculating and comparing the hash values of the target node

[0033] Preferably, the fault recovery strategy is designed based on the Markov decision process model. By analyzing the transition probability matrix of the system state, select the optimal recovery path to restore the system to the normal operation state when a database node fails. The specific steps include the following:

[0034] Divide the possible operating states of each node in the database cluster into multiple discrete states

[0035] Calculate the probability of the system transitioning from one state to another based on historical monitoring data or real-time observation results to form a transition probability matrix;

[0036] Define a reward or penalty value for each state transition to evaluate the quality of the system state transition;

[0037] Use the Markov decision process model to calculate the optimal recovery strategy, which selects the recovery path by maximizing the long-term utility of the system;

[0038] When a failure occurs, perform repair operations in real-time according to the optimal recovery strategy, including data resynchronization, node restart, or task transfer;

[0039] After completing the repair operation, verify the system state.

[0040] Preferably, the optimal recovery strategy calculates the value of each state through the value iteration algorithm, and the specific steps include:

[0041] Calculate the value of state through the following recurrence formula : , where is to optimize all possible operations , is the immediate reward after taking action in state , is the discount factor, is the probability of transitioning from state to state , is the current state, representing the specific state of the system, is the target state, representing the next state after transitioning from the current state , represents the sum over all possible target states ; Select the optimal recovery operation by iteratively calculating the maximum value path.

[0042] Preferably, the method dynamically adjusts the consistency level according to the system load condition, and uses a non-linear activation function to automatically adjust the consistency to balance the performance and consistency requirements. The specific steps include:

[0043] Collect the real-time operation metrics of each database node in the system;

[0044] Divide the consistency level into multiple levels;

[0045] Set the load upper and lower limit thresholds for each node through historical load data and system performance analysis;

[0046] Use a non - linear activation function to smooth the real - time load and calculate the activation value for adjusting the consistency level;

[0047] According to the calculated activation value, dynamically allocate the consistency level for each database node;

[0048] After dynamically adjusting the consistency level, monitor the system running state to detect whether exceptions or performance bottlenecks are caused by the consistency adjustment.

[0049] A multi - database deployment system based on a server cluster, including:

[0050] Multiple database nodes. Multiple independent database nodes jointly store and process data, ensuring distributed data storage and supporting parallel operations;

[0051] A server cluster, composed of multiple servers, coordinates the work of these servers through a cluster management system, provides computing and storage resources, and achieves load balancing and high availability;

[0052] A data synchronization module, responsible for synchronizing data between database nodes, adopts an incremental synchronization strategy, and only synchronizes the changed data blocks to improve the synchronization efficiency and reduce bandwidth occupancy.

[0053] The present invention provides a multi - database deployment method based on a server cluster. It has the following beneficial effects:

[0054] 1. The present invention adopts a technical solution based on transaction state vector allocation and consistency operator mapping, achieving the technical effect of efficiently realizing data consistency in a distributed database environment. Compared with the consistency management solutions that rely on fixed synchronization rules or manual intervention in the prior art, it solves the problems of low consistency maintenance efficiency and easy data conflicts in high - concurrency scenarios.

[0055] 2. The present invention adopts an incremental synchronization and redundancy elimination technical solution. By detecting data block changes and only synchronizing the changed parts, and combining the Shannon entropy method and Huffman coding to compress redundant data blocks, it achieves the technical effect of significantly reducing the amount of synchronized data and network bandwidth occupancy. Compared with the method of full - volume synchronizing data in the prior art, it solves the problems of large bandwidth occupancy and low transmission efficiency.

[0056] 3. The present invention designs a fault recovery strategy based on the Markov decision process (MDP) model, and uses the value iteration algorithm to generate an optimal recovery path, achieving the technical effect of quickly restoring the system to the normal state when a node fails. Compared with the fault recovery strategies that rely on fixed recovery schemes or single - redundancy designs in the prior art, it solves the deficiencies of slow recovery speed and low resource utilization.

[0057] 4. The present invention adopts a dynamic consistency adjustment technical solution based on load scoring and non-linear activation functions, achieving the technical effect of balancing performance and consistency under different load conditions. Compared with the prior art solutions with fixed or manually adjusted consistency strategies, it solves the problems of poor performance and inflexible adjustment in high-load or low-load scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flowchart of the steps of the multi-database deployment method based on a server cluster according to the present invention;

[0059] Figure 2 It is a structural diagram of the multi-database deployment system based on a server cluster according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0061] Please refer to the attached Figure 1 , the embodiments of the present invention provide a multi-database deployment method based on a server cluster, including the following steps:

[0062] S1. Assign transaction status information to each database node, and use a consistency operator to achieve data consistency among nodes in the database cluster;

[0063] Specifically, the goal of step S1 is to achieve data consistency management among multiple database nodes by assigning transaction status information to each database node and combining a consistency operator to perform a global consistency mapping on the transaction status; in the multi-database deployment method based on a server cluster, the accurate assignment of transaction status information and the application of a consistency operator are the basis for achieving cluster consistency and transaction correctness; this step provides methodological and structural technical support to ensure the efficient operation of the database cluster in a distributed environment.

[0064] Specifically, this step combines the real-time status and operation load among database nodes to model, calculate, and dynamically adjust transaction information, thereby achieving data consistency optimization.

[0065] In this embodiment, the assignment of transaction status information specifically includes:

[0066] As an option, the transaction status information of each database node consists of core attributes such as transaction commit status, isolation level, and data modification status. Exemplarily, these attributes are organized into a high-dimensional transaction status vector indicates the node 's current transaction status. Specifically:

[0067] Transaction commit status : Used to identify whether the current transaction has been committed. A value of 0 indicates not committed, and a value of 1 indicates committed.

[0068] Isolation level : Represents the concurrency control strategy of the transaction and supports the following isolation levels:

[0069] Read uncommitted (READ UNCOMMITTED)

[0070] Read committed (READ COMMITTED)

[0071] Repeatable read (REPEATABLE READ)

[0072] Serializable (SERIALIZABLE).

[0073] Data modification status : Indicates whether the transaction contains data modification operations. A value of 0 indicates no modification, and a value of 1 indicates modification.

[0074] In one possible implementation, the expression of the transaction status vector is as follows: ; It can be understood that the transaction status vector is updated in real time by dynamically monitoring the running status of the node to reflect the latest transaction situation of the node.

[0075] In this embodiment, the implementation of the consistency operator includes:

[0076] It should be noted that the consistency operator is used to achieve data consistency between multiple database nodes by mapping transaction status information to a global consistency state . Specifically:

[0077] In some embodiments, the consistency operator first assigns weights to each node , and the weights are dynamically adjusted according to factors such as the load, network latency, and importance of the node.

[0078] Exemplarily, the calculation method of the global consistency state is: ;

[0079] Among them, is the number of database nodes, is the weight of node , is the transaction status vector of the node.

[0080] As an option, the consistency operator can also perform fine-grained consistency adjustment based on a multi-scale model. For example, when the node load is high, only critical nodes are subject to consistency constraints; when the load is low, global consistency constraints are applied to all nodes.

[0081] In this embodiment, the dynamic consistency check includes:

[0082] During the execution of a transaction, the consistency operator is used to continuously verify in real time whether the transaction status of each node matches the global consistency status. Specifically:

[0083] If the transaction status of a certain node deviates from the global consistency status, the system will trigger a consistency recovery mechanism to correct the deviation through transaction compensation or rollback operations.

[0084] In a possible implementation, the consistency check can be achieved by calculating the degree of deviation between the node state vector and the global state. If the deviation value exceeds the set threshold, the corresponding recovery operation is executed.

[0085] It should be noted that this consistency check mechanism ensures the transaction correctness of the database cluster in a high-concurrency environment and reduces the risk of data conflicts caused by consistency problems.

[0086] Through the detailed description of the above implementation manner, those skilled in the art can easily reproduce the technical solution of this step according to the definition and allocation of transaction status information, the implementation mechanism of the consistency operator, and the logic of dynamic consistency check.

[0087] S2. Optimize the data synchronization path according to the topological structure between nodes, and select the optimal path to reduce data transmission latency;

[0088] Specifically, the purpose of step S2 is to optimize the data synchronization path by analyzing the topological structure between database nodes, combined with the network latency and bandwidth between nodes, and select the transmission path with the minimum latency to improve the efficiency and accuracy of data transmission. It should be noted that the optimization of the data synchronization path is the core technical means to achieve efficient data transmission and is of great significance for ensuring the consistency and performance of the database cluster.

[0089] Specifically, this step uses a graph model to model the connection relationship of database nodes and dynamically calculates the optimal synchronization path between nodes in combination with the shortest path algorithm. The implementation of topological structure optimization combines real-time monitoring and dynamic adjustment mechanisms and can adapt to changes in system load and network conditions.

[0090] In this embodiment, the topological structure modeling includes:

[0091] As an option, the connection relationship between database nodes can be represented by a weighted undirected graph denotes, where:

[0092] is a set of database nodes, and each node represents an independent database instance;

[0093] is the edge between nodes, representing the network connection between nodes;

[0094] The weight of the edge represents node and node the synchronization delay between them.

[0095] It should be noted that the network delay is jointly determined by factors such as bandwidth, physical distance, and transmission error rate. In a possible implementation, the delay weight can be calculated according to the following formula: ;

[0096] where is the delay weight from node to node ; is the weight coefficient, representing the importance of network delay in the comprehensive cost; is the weight coefficient, representing the importance of network bandwidth in the comprehensive cost; is the weight coefficient, representing the importance of transmission error rate in the comprehensive cost; is the network delay from node to node , usually in milliseconds (ms); is the network bandwidth from node to node , usually in megabits per second (Mbps); is the transmission error rate between node and node , and the value range is .

[0097] It can be understood that the above topological structure modeling provides basic data support for the optimization of the data synchronization path, ensuring the accuracy of path selection.

[0098] The path selection is weighted according to the synchronization delay between nodes, and the path with the minimum transmission delay is preferentially selected. The cost function of path selection is calculated by the following formula:

[0099]

[0100] where is the synchronization cost of path , is node The weight between nodes, represents the sum of the incremental data volumes for all node pairs in the system and in the system. To sum up, for the synchronization delay, select the synchronization path with the minimum cost as the optimal data synchronization path.

[0101] In this embodiment, the optimal path selection includes:

[0102] Specifically, based on the above topological structure, calculate the shortest path from the source node to the target node through the Dijkstra algorithm. Exemplarily, the execution process of the Dijkstra algorithm is as follows:

[0103] First, set the initial path length for each node. The path length of the source node is set to 0, and the path lengths of other nodes are set to infinity;

[0104] Then, select the node with the shortest path length from the unvisited nodes and mark it as visited;

[0105] Next, update the path lengths of all adjacent nodes of this node. If the path length through this node is shorter than the current value, update its path length;

[0106] Finally, repeat the above steps until all nodes are visited.

[0107] As an option, the calculation method for updating the path length is: ;

[0108] where is the current path length of the target node , is the path length of the current node , is the weight between node and , represents the edge weight between node and node , usually used to measure the path cost from to , represents taking the smaller value of the current path length and the path length from to to the source node .

[0109] It should be noted that the calculation result of the Dijkstra algorithm is the set of the shortest paths from the source node to the target node, which can ensure the minimization of the data synchronization delay.

[0110] In this embodiment, the dynamic adjustment of path optimization includes:

[0111] In some embodiments, to adapt to the changes in system load and network conditions, a real-time monitoring mechanism can be introduced to dynamically adjust the synchronization path. Specifically:

[0112] Real-time monitor the network delay and bandwidth between nodes, and periodically update the edge weights in the topology structure;

[0113] When the network condition changes significantly, re-execute the shortest path calculation and dynamically adjust the synchronization path.

[0114] Exemplarily, when the delay or bandwidth of a certain path exceeds the set threshold, the system will select a sub-optimal path as an alternative path to ensure the continuity of data synchronization.

[0115] Through the detailed description of the above implementation manners, those skilled in the art can easily reproduce the technical solution of this step based on the topology structure modeling, optimal path selection, and dynamic adjustment mechanism. The above method has significant advantages in improving data transmission efficiency, reducing delay, and adapting to network changes.

[0116] S3. Reduce the amount of synchronized data through incremental synchronization and redundancy elimination methods;

[0117] Specifically, the goal of step S3 is to reduce the amount of data synchronization between database nodes through incremental synchronization methods and data redundancy elimination technologies, and improve data transmission efficiency. It should be noted that the optimization of the amount of data synchronization is of great significance for reducing network bandwidth occupancy and improving the overall performance of the database cluster.

[0118] Specifically, this step adopts an incremental synchronization strategy based on change detection, only synchronizes the changed data blocks, and reduces the data transmission requirements from the source. At the same time, combined with redundancy elimination methods (such as Shannon entropy analysis and Huffman coding technology based on information theory), further compress the amount of synchronized data to achieve the purpose of efficient transmission.

[0119] In this embodiment, the implementation of incremental synchronization includes:

[0120] Incremental synchronization identifies the data blocks that need to be synchronized through change detection. Specifically:

[0121] By calculating the unique identifier (such as a hash value) of each data block, comparing the data blocks of the new and old versions, if the hash values of the two are inconsistent, then mark this data block as changed data.

[0122] Exemplarily, assume that the data block of a certain node is updated to a new version , and incremental synchronization first calculates the hash values of the two and . If , it is determined that the data block has changed.

[0123] It can be understood that the above method can avoid the repeated transmission of unchanged data, thus significantly reducing the amount of data synchronization.

[0124] As an option, the data blocks for incremental synchronization can be sorted by priority. For example, data blocks with high transaction dependency or sensitive to synchronization latency are synchronized first to ensure the consistency and real-time requirements of the system.

[0125] In this embodiment, the redundancy elimination method includes:

[0126] It should be noted that redundancy elimination is an important means to further optimize the amount of synchronized data. In the present invention, the Shannon entropy method based on information theory is used to evaluate the data redundancy degree, and the data blocks with high redundancy degree are compressed. Specifically:

[0127] Use the Shannon entropy formula to calculate the information amount of each data block:

[0128]

[0129] where is the Shannon entropy (information entropy) of the random variable , represents a discrete random variable, represents the probability of the symbol appearing in the data block, represents the probability with base 2 logarithm, represents the random variable and the number of possible values of

[0130] As an option, for data blocks with high redundancy degree, Huffman coding is used for compression. Huffman coding constructs an optimal prefix tree based on symbol frequencies, assigns shorter codes to high-frequency symbols, and realizes efficient data compression.

[0131] Exemplarily, assume that the symbol distribution of a certain data block is , and the result of Huffman coding is , and the compression ratio can reach the ideal value.

[0132] It can be understood that the redundancy elimination method further reduces the amount of data transmission on the basis of incremental synchronization, effectively improving the network utilization rate of the system.

[0133] In this embodiment, data synchronization execution includes:

[0134] In a possible implementation, after the incremental data completes redundancy elimination, it is transmitted to the target node through the optimal synchronization path. Specifically:

[0135] During the data transmission process, the streaming transmission method is adopted to gradually transmit the data in blocks to reduce the memory occupation and latency during the transmission process.

[0136] At the target node, the received data block will be decoded and merged with the existing database to ensure data consistency.

[0137] It should be noted that in order to improve the robustness of the transmission process, the system will perform real-time verification on the transmission process. If data loss or transmission error is found, the missing data block will be requested to be transmitted again until the synchronization is completed.

[0138] Through the detailed description of the above implementation, those skilled in the art can easily reproduce the technical solution of this step based on the change detection method of incremental synchronization, the Shannon entropy analysis and Huffman coding technology of redundancy elimination, and the optimization mechanism of data transmission. The above method has significant advantages in reducing the amount of data synchronization and improving the data transmission efficiency.

[0139] S4. Design and implement a fault recovery strategy to ensure that the system can quickly resume normal operation when a database node fails;

[0140] Specifically, the purpose of step S4 is to ensure that the database node can quickly resume to the normal operation state when a failure occurs by designing and implementing a fault recovery strategy, so as to improve the reliability and availability of the system. It should be noted that the key to the fault recovery strategy lies in the modeling of the system state, the generation of the optimal recovery path, and the execution of dynamic repair, which jointly ensure the efficient recovery ability of the system.

[0141] Specifically, the present invention uses the Markov decision process (MDP) to model the fault recovery strategy, and generates the optimal recovery path through the value iteration algorithm. When a failure occurs, the system dynamically executes the repair operation according to the generated recovery path, and verifies and optimizes the repair result.

[0142] In this embodiment, the system state modeling includes:

[0143] As an option, the operating state of the system can be divided into multiple discrete states, including the normal state, the fault state, and the recovery state, etc. Specifically:

[0144] The normal state indicates that the database node is running normally and can process transactions and synchronize data;

[0145] The failure state indicates that the database node has stopped running or its performance has significantly degraded due to hardware or network problems;

[0146] The recovery state indicates that the database node is being repaired from a failure, and some of its functions may have been restored but still need further optimization.

[0147] In some embodiments, the transition of the system state is described by a state transition matrix. Specifically, the state transition probability , represents the probability that the system transitions from state to . The state transition probability matrix can be generated based on historical data statistics or real-time monitoring information.

[0148] It should be noted that state modeling provides a data basis for the generation of the optimal recovery path, ensuring the scientificity and efficiency of the recovery strategy.

[0149] In this embodiment, the generation of the optimal recovery path includes:

[0150] Specifically, using the Markov decision process (MDP) model, the value of each system state is calculated through the value iteration algorithm, and the optimal recovery path from the failure state to the normal state is generated. Exemplarily, the core calculation formula of the value iteration algorithm is: ; where represents the value of state ; is to optimize all possible operations ; represents the immediate reward for transitioning from state to other states through operation ; represents the probability that state transitions to ; is the discount factor, representing the influence weight of future rewards; is the current state, representing the specific state of the system; is the target state, representing the next state after transitioning from the current state ; represents the sum over all possible target states .

[0151] It can be understood that through the value iteration algorithm, the optimal value of each state can be calculated, and the optimal recovery path in the failure state can be determined.

[0152] As an option, when calculating the optimal recovery path, the value function can be dynamically adjusted by combining the resource usage of the system and the operation cost. For example, if the repair operation of a certain node consumes high resources, its reward value can be reduced to reduce resource usage.

[0153] In this embodiment, the dynamic repair execution includes:

[0154] In a possible implementation, after the system detects a node failure, it will gradually execute the repair operation according to the generated optimal recovery path. Specifically:

[0155] For hardware failures, node restart or migration operations can be performed;

[0156] For data inconsistency failures, data resynchronization or transaction rollback operations can be performed;

[0157] For network failures, switch to the alternate path to ensure the continuity of data transmission.

[0158] It should be noted that during the dynamic repair process, the system will monitor the effect of the repair operation in real time. For example, by detecting the health status of the node and the transaction processing ability, it is judged whether the current repair reaches the expected goal. If the repair effect is not ideal, the recovery path is dynamically adjusted and a new repair plan is generated.

[0159] In this embodiment, the repair result verification includes:

[0160] After the repair operation is completed, the system status is verified to ensure that the system has been fully restored to the normal operating state. Specifically:

[0161] By comparing the system status before and after the repair, verify whether all failed nodes have returned to normal;

[0162] Through the consistency verification tool, check whether new data inconsistency problems are introduced during the repair process;

[0163] If it is found that there are still abnormal node states, repeat the repair operation until the system is fully restored.

[0164] It should be noted that the repair result verification can effectively avoid the system performance degradation or potential fault hazards caused by incomplete recovery.

[0165] Through the detailed description of the above implementation manners, those skilled in the art can easily reproduce the technical solution of this step based on system state modeling, optimal recovery path generation, dynamic repair execution, and repair result verification. The above method has significant advantages in improving the fault recovery speed and system reliability.

[0166] S5. Dynamically adjust the synchronization strategy and consistency level of the database cluster according to the system load situation.

[0167] Specifically, the goal of step S5 is to dynamically adjust the synchronization strategy and consistency level of the database cluster by monitoring the system load in real time and according to the load changes, so as to improve the overall performance while ensuring the system consistency. It should be noted that the system load directly affects the processing capacity of the database nodes and the network bandwidth utilization rate. By dynamically adjusting the strategy, a balance can be achieved between performance and consistency.

[0168] Specifically, this step includes three core links: load monitoring and analysis, consistency level adjustment, and synchronization strategy optimization. The above links cooperate closely to jointly achieve the efficient operation of the system under different load conditions.

[0169] In this embodiment, the load monitoring and analysis include:

[0170] In some embodiments, the system collects the running state information of the database nodes in real time through monitoring tools, including but not limited to the following indicators:

[0171] CPU occupancy rate, used to evaluate the usage of computing resources;

[0172] Memory usage rate, used to evaluate the storage buffer capacity of the node;

[0173] Disk I / O rate, used to evaluate the data read and write performance;

[0174] Network bandwidth utilization rate, used to evaluate the data transmission capacity.

[0175] It should be noted that in order to comprehensively evaluate the load situation of the nodes, the above indicators can be weighted and calculated to generate a load score :

[0176] ; where represents the comprehensive load score of node , represents the CPU usage rate of node , and the value range is , represents the memory usage rate of node , and the value range is , represents the disk I / O usage rate of node , and the value range is , represents the network bandwidth utilization rate of node , and the value range is , , ​ , are the weight coefficients of CPU, memory, disk I / O, and network bandwidth respectively, and satisfy the following conditions: .

[0177] Exemplarily, if the CPU utilization rate of node is 70%, the memory usage rate is 80%, the disk I / O is 60 MB / s, and the network bandwidth occupancy rate is 50%, then the load score of this node can be obtained by combining the weight calculation.

[0178] In this embodiment, the adjustment of the consistency level includes:

[0179] As an option, dynamically adjust the consistency level according to the load score, including two typical modes: strong consistency and eventual consistency. Specifically:

[0180] Strong consistency requires that all nodes maintain complete data consistency when the transaction is completed, but it will increase network latency;

[0181] Eventual consistency allows node data to be out of sync for a short period of time, thereby improving throughput and response speed.

[0182] In a possible implementation, the dynamic adjustment of the consistency level can be achieved through a non-linear activation function. For example, use the Sigmoid function to smooth the load score as follows:

[0183]

[0184] where, represents the output value of the activation function of node , and its value range is , represents the load score of node , represents the steepness coefficient of the activation function, represents the threshold of the load score, represents the base of the natural logarithm, approximately equal to 2.718.

[0185] When is close to 0, the system preferentially selects eventual consistency;

[0186] When is close to 1, the system switches to the strong consistency mode.

[0187] It can be understood that the above method realizes the dynamic adjustment of the consistency level through the activation function, avoiding the performance jitter caused by frequent switching.

[0188] In this embodiment, the optimization of the synchronization strategy includes:

[0189] Specifically, the data synchronization strategy is adjusted according to the selection of the consistency level, including the following optimization measures:

[0190] In the strong consistency mode, the full synchronization method is adopted to ensure the real-time consistency of data on all nodes;

[0191] In the eventual consistency mode, the incremental synchronization strategy is adopted to synchronize only the changed data blocks to reduce the synchronization overhead.

[0192] As an option, the synchronization frequency can also be dynamically adjusted according to the network bandwidth of the nodes. For example:

[0193] When the network bandwidth is abundant, increase the synchronization frequency to reduce the time of data desynchronization;

[0194] When the network bandwidth is limited, reduce the synchronization frequency to reduce the bandwidth occupancy.

[0195] It should be noted that the optimization of the synchronization strategy can maximize the operating efficiency of the system without sacrificing consistency.

[0196] Through the detailed description of the above implementation manners, those skilled in the art can easily reproduce the implementation process of this step based on the technical solutions of load monitoring and analysis, consistency level adjustment, and synchronization strategy optimization. The above method has significant advantages in dynamically adjusting the balance between the performance and consistency of the database cluster.

[0197] Please refer to the attached Figure 2 , the present invention also provides a multi-database deployment system based on a server cluster, including: multiple database nodes, and multiple independent database nodes jointly store and process data to ensure distributed storage of data and support parallel operations;

[0198] A server cluster, composed of multiple servers, coordinates the work of these servers through a cluster management system, provides computing and storage resources, and realizes load balancing and high availability;

[0199] A data synchronization module, responsible for synchronizing data between database nodes, adopts an incremental synchronization strategy to synchronize only the changed data blocks to improve the synchronization efficiency and reduce the bandwidth occupancy;

[0200] A fault recovery module, when a system failure occurs, quickly evaluates the system state and transition probability through a Markov decision process model and a value iteration algorithm, selects the optimal recovery path, and ensures the rapid recovery of the system and reduces the downtime.

[0201] Specifically, the multi-database deployment system based on a server cluster solves the problems of data consistency, efficient synchronization, and fault recovery among multiple database nodes by designing multiple modules to work together, ensuring the high availability and high performance of the system. The following is a detailed description and extension of each module:

[0202] Multiple database nodes, each node independently stores part of the data and can independently process transaction operations. Specifically, the main functions of the database nodes include:

[0203] Distributed data storage: Divide the global data set into multiple data shards, and each database node is responsible for storing a specific data shard. As an option, the sharding strategy can be based on the primary key range or hash value of the data for partitioning, thus achieving balanced storage load.

[0204] Parallel transaction processing: Each database node independently executes transaction operations, supports high-concurrency access, and coordinates consistency with other nodes through a transaction status vector (such as commit status, isolation level, and modification status).

[0205] It should be noted that the database nodes communicate through a high-speed network connection to support operations such as data synchronization and fault recovery. In a possible implementation, the communication protocol between nodes adopts a lightweight transport protocol (such as gRPC) to reduce communication overhead and improve response speed.

[0206] The server cluster consists of multiple high-performance servers, and unified scheduling and resource management are achieved through a cluster management system. Specifically, the functions of the server cluster include:

[0207] Computing resource allocation: Dynamically allocate computing resources according to the load conditions of the database nodes, such as the number of CPU cores and memory usage.

[0208] Storage resource scheduling: Manage the storage resources in the server cluster to ensure the balance of data distribution and provide redundant storage to enhance reliability.

[0209] As an option, the cluster management system monitors the load conditions of each server in real time through a load balancing module and allocates tasks according to the load balancing algorithm. Exemplarily, the load balancing algorithm can adopt the weighted round-robin or least-connections strategy.

[0210] It can be understood that the high availability of the server cluster is achieved through redundant design and a fault switching mechanism. For example, when a certain server fails, the cluster management system can automatically switch to a standby server to avoid affecting the overall operation of the system.

[0211] The data synchronization module is responsible for synchronizing data between database nodes to achieve consistency and data integrity. Specifically, the data synchronization module adopts an incremental synchronization strategy, only synchronizing the data blocks that have changed, thus significantly reducing the amount of data transmission. Its implementation methods include:

[0212] Change detection: Detect whether the data has changed by calculating the hash value of the data block. If the hash values are different, mark the data block as the data block to be synchronized.

[0213] Redundancy elimination: Perform redundancy elimination operations on the data blocks to be synchronized. For example, use the Shannon entropy method to evaluate the redundancy of the data blocks, and apply Huffman coding to compress the data blocks with higher redundancy, thereby further reducing the amount of synchronized data.

[0214] Synchronization path optimization: Select the optimal data synchronization path according to the network topology and transmission delay between nodes. Specifically, the shortest path between nodes can be calculated based on the Dijkstra algorithm to reduce the synchronization delay and improve the transmission efficiency.

[0215] It should be noted that to ensure the accuracy of the synchronized data, the data synchronization module verifies the data integrity of the target node after synchronization. For example, verify whether the data is consistent by recalculating the hash value of the data block of the target node and comparing it with the source node.

[0216] Fault recovery module

[0217] When a fault occurs in the system, the fault recovery module of this system quickly evaluates the system state and generates an optimal recovery path through the Markov decision process (MDP) model and the value iteration algorithm. Specifically, the functions of the fault recovery module include:

[0218] System state modeling: Divide the system state into normal state, fault state, and recovery state, and generate a state transition probability matrix based on historical data. The state transition probability represents the possibility of transitioning from one state to another state, and is used to guide the generation of the recovery path.

[0219] Optimal recovery path generation: Calculate the value of each state through the value iteration algorithm, and generate an optimal recovery path from the fault state to the normal state. Exemplarily, the calculation formula of the value function is: ;

[0220] Among them, among them, represents the value of state ; is optimized for all possible operations ; represents the immediate reward for transitioning from state to other states through operation ; The probability of indicating the state Transfer to ; The probability is the discount factor, indicating the influence weight of future rewards; is the current state, indicating the specific state in which the system is; The target state, indicating the next state after transfer from the current state ; Indicates the sum over all possible target states ;

[0221] Dynamic recovery operation: According to the optimal recovery path, gradually execute recovery operations, such as node restart, data resynchronization, or task switching.

[0222] As an option, to improve the efficiency of fault recovery, the system can monitor the recovery effect and dynamically adjust the path while executing the recovery operation. For example, when the recovery operation effect of a certain node is not ideal, an alternative solution can be selected by recalculating the path.

[0223] Through the above detailed description, those skilled in the art can easily reproduce the implementation process of the system based on the technical solutions of the database node design, server cluster management, data synchronization, and fault recovery modules provided by the present invention. The present invention has significant advantages in improving the performance, consistency, and high availability of the database cluster.

[0224] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-database deployment method based on a server cluster, characterized in that: The following steps are involved: Assign transaction status information to each database node and use consistency operators to achieve data consistency among nodes in the database cluster; Optimize the data synchronization path according to the topological structure between nodes and select the optimal path to reduce data transmission delay; Reduce the amount of synchronized data through incremental synchronization and redundancy elimination methods; Design and implement fault recovery strategies to ensure rapid recovery of the system when a database node fails; Dynamically adjust the synchronization strategy and consistency level of the database cluster according to the system load; The transaction status information of the database node is represented by a high-dimensional vector, which specifically includes the following steps: The transaction status information of each database node is represented by a multidimensional vector, where each dimension of the vector corresponds to a specific transaction status attribute, including transaction commit status, isolation level, and data modification status; Combine the transaction states of all database nodes into a high-dimensional state space, with the state vector of each node as a point in the space; The transaction state vector of the database node is dynamically updated as the transaction is executed; After the transaction is committed, the state vector of the database node is synchronously adjusted according to the consistency operator; The consistency operator is used to achieve data consistency between multiple database nodes by mapping transaction state information into a global consistency state C.

2. The multi-database deployment method based on server cluster according to claim 1, characterized in that: The consistency operator maps the transaction status of each database node by defining a multi-scale consistency model, which specifically includes the following steps: Abstract the transaction status information of each database node into a state point in a high-dimensional space; To achieve data consistency between database nodes, a mapping rule is defined to map the transaction status of each database node to a unified global status; Based on the transaction state space, the global consistency state is calculated using the consistency mapping rules; According to the global consistency state, the transaction state of each database node is adjusted at multiple scales; After the transaction operation is completed, the consistency operator is used to check the consistency of each node in the database cluster.

3. The multi-database deployment method based on server cluster according to claim 1, characterized in that: The optimized data synchronization path is calculated by the shortest path algorithm based on synchronization delay weight. The path selection is weighted according to the synchronization delay between nodes, and the path with the smallest transmission delay is preferentially selected. The cost function of the path selection is calculated by the following formula: Among them, C sync (p) is the synchronization cost of path p, W ij is the weight between node i and node j, ∑ i，j ΔT ij is the incremental data volume ΔT for all pairs of nodes i and j in the system ij Summing, ΔT ij To reduce the synchronization delay, the synchronization path p with the minimum cost is selected as the optimal data synchronization path.

4. The multi-database deployment method based on server cluster according to claim 3 is characterized in that: The shortest path algorithm uses the Dijkstra algorithm, which selects the optimal path according to the synchronization delay between database nodes. The specific steps are: According to the synchronization delay ΔT between each pair of nodes ij Calculate the initial distance and form the edge weight of the graph; Starting from the source node, the path with the minimum synchronization delay is recursively selected through the shortest path selection algorithm until all nodes are traversed to obtain the optimal data synchronization path.

5. The multi-database deployment method based on server cluster according to claim 1, characterized in that: The data synchronization method adopts incremental synchronization, synchronizing only the changed data blocks, and evaluates data redundancy through the Shannon entropy method in information theory, and uses Huffman coding to compress redundant data to reduce the bandwidth required for synchronization. The specific steps include: Calculate a unique identifier for each database node's storage data block to identify whether the data block has changed; The detected changed data block set is used as the incremental synchronization data set; The Shannon entropy method is used to evaluate the redundancy of each data block. The data block with a lower Shannon entropy value has a higher redundancy and is suitable for further compression. Select the optimal data transmission path based on the topological relationship and synchronization delay between database nodes; Synchronize the compressed incremental data to the target node through the selected path; After data synchronization is completed, the integrity and accuracy of the synchronized data are verified by recalculating and comparing the hash value of the target node.

6. The multi-database deployment method based on server cluster according to claim 1, characterized in that: The fault recovery strategy is designed based on the Markov decision process model. By analyzing the transition probability matrix of the system state, the optimal recovery path is selected to restore the system to a normal operating state when a database node fails. Specifically, the following steps are included: Divide the possible operating state of each node in the database cluster into multiple discrete states; Based on historical monitoring data or real-time observation results, the probability of the system transferring from one state to another is calculated to form a transition probability matrix; Define reward or penalty values ​​for each state transition to evaluate the pros and cons of the system state transition; The optimal recovery strategy is calculated using a Markov decision process model, which selects the recovery path by maximizing the long-term utility of the system; When a failure occurs, repair operations are performed in real time according to the optimal recovery strategy, including data resynchronization, node restart or task transfer; After the repair operation is completed, verify the system status.

7. The method for deploying multiple databases based on a server cluster according to claim 6, characterized in that: The optimal recovery strategy calculates the value of each state through a value iteration algorithm, and the specific steps include: The state S is calculated by the following recursive formula i The value of V(S i ): Among them, R(S i , a) is state S i The immediate reward after taking action a, γ is the discount factor, P(S j |S i , a) from state S i Transfer to state S j The probability of The optimal recovery operation is selected by iteratively calculating the maximum value path.

8. The multi-database deployment method based on server cluster according to claim 1, characterized in that: The method dynamically adjusts the consistency level according to the system load and uses a nonlinear activation function to automatically adjust the consistency to balance performance and consistency requirements. Specifically, the method includes the following steps: Collect real-time operating indicators of each database node in the system; Divide the consistency level into multiple levels; Through historical load data and system performance analysis, set upper and lower load thresholds for each node; Use nonlinear activation functions to smooth the real-time load and calculate activation values ​​to adjust the consistency level; Dynamically assign consistency levels to each database node based on the calculated activation value; After dynamically adjusting the consistency level, monitor the system operation status to detect whether anomalies or performance bottlenecks are caused by the consistency adjustment.

9. A multi-database deployment system based on a server cluster, applied to a multi-database deployment method based on a server cluster as claimed in any one of claims 1 to 8, characterized in that: include: Multiple database nodes: Multiple independent database nodes jointly store and process data, ensuring distributed data storage and supporting parallel operations; A server cluster consists of multiple servers. The cluster management system coordinates the work of these servers, provides computing and storage resources, and achieves load balancing and high availability. The data synchronization module is responsible for synchronizing data between database nodes. It adopts an incremental synchronization strategy and only synchronizes changed data blocks to improve synchronization efficiency and reduce bandwidth usage. The fault recovery module, when a system failure occurs, quickly evaluates the system status and transition probability through the Markov decision process model and value iteration algorithm, selects the optimal recovery path, ensures rapid system recovery and reduces downtime.

Images