Distributed block storage system performance optimization methods and systems

Abstract

This application relates to the field of computer technology and discloses a method and system for optimizing the performance of a distributed block storage system. The method includes: responding to user data storage needs, analyzing the access frequency of the data to be stored to dynamically divide it into blocks, generating multiple blocks; dynamically classifying data storage nodes into master nodes and slave nodes according to the load status of each node in the storage system; writing each block of data to the corresponding master node according to the access frequency, and copying the data increments corresponding to each master node to the corresponding slave nodes, thus obtaining updated master nodes and updated slave nodes; responding to user data reading needs, analyzing the access correlation between data, and fusing and reading the corresponding data from the updated slave nodes. This application dynamically divides and hierarchically stores data, which can alleviate the pressure on overloaded master nodes, balance the load between data storage nodes, and optimize system performance.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a method and system for optimizing the performance of distributed block storage systems. Background Technology

[0002] Currently, data storage uses a fixed block size method, resulting in the mixed storage of high-frequency and low-frequency access data, which affects read efficiency. The master-slave node partitioning method uses static configuration, which cannot respond to changes in node load and failures in real time, leading to master node overload or waste of slave node resources. The data replication process uses full replication or fixed ratio replication, which cannot be adjusted to the real-time status of nodes, increasing data storage costs and affecting system performance.

[0003] To address at least one of the aforementioned problems, this application proposes a method and system for optimizing the performance of distributed block storage systems. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this application is to provide a performance optimization method and system for distributed block storage systems, which can effectively solve the problems in the background technology. The specific technical solution of this application is as follows:

[0005] Performance optimization methods for distributed block storage systems include:

[0006] In response to users' data storage needs, the access frequency of the data to be stored is analyzed to dynamically divide it into blocks and generate multiple blocks of data.

[0007] Based on the load status of each node in the storage system, data storage nodes are dynamically classified into master nodes and slave nodes.

[0008] Based on the access frequency, each block of data is written to the corresponding master node, and the data increment corresponding to each master node is copied to the corresponding slave node to obtain the updated master node and the updated slave node.

[0009] In response to users' data reading needs, the system analyzes the access correlation between data and merges and reads the corresponding data from the update slave nodes to optimize the performance of the distributed block storage system.

[0010] Specifically, in response to the user's data storage needs, the access frequency of the data to be stored is analyzed to dynamically divide it into blocks, generating multiple blocks of data, including:

[0011] In response to users' data storage needs, the data type and content of the data to be stored are analyzed.

[0012] Based on the data type, the access frequency of the data is predicted using a preset access frequency prediction model to obtain the predicted access frequency.

[0013] Based on the predicted access frequency and the preset segmentation strategy, the data content is segmented into multiple blocks of data.

[0014] Specifically, the step of dynamically classifying data storage nodes into master nodes and slave nodes based on the load status of each node in the storage system includes:

[0015] Based on the load status of nodes in the storage system, the load score of each node is calculated using a preset node load evaluation model.

[0016] Nodes with load scores less than a preset first load threshold are designated as initial master nodes, and nodes with load scores greater than or equal to the preset first load threshold are designated as initial slave nodes.

[0017] By analyzing the positional relationships between nodes and the consistency of data storage types, at least one initial slave node is assigned to each initial master node, and a master-slave node relationship graph is constructed.

[0018] Based on the real-time status of the data storage nodes, the master-slave node relationship graph is updated through a preset node update mechanism to obtain the real-time master and slave nodes.

[0019] Specifically, the step of updating the master-slave node relationship graph based on the real-time status of the data storage nodes through a preset node update mechanism to obtain the real-time master and slave nodes includes:

[0020] The load score of each node is updated in real time based on the real-time status of the data storage nodes to obtain the updated load score.

[0021] When the update load score of the initial master node is greater than the preset second load threshold, the node with the smallest update load score is selected from the corresponding initial slave nodes and used as the auxiliary master node.

[0022] When the initial master node fails or malfunctions, the node with the lowest update load score among the corresponding initial slave nodes is selected as the replacement master node.

[0023] Based on the auxiliary master node and the replacement master node, the master-slave node relationship graph is updated to obtain the real-time master node and slave node.

[0024] Specifically, based on the access frequency, each block of data is written to the corresponding master node, and the data increments corresponding to each master node are copied to the corresponding slave nodes, resulting in updated master nodes and updated slave nodes, including:

[0025] Based on access frequency, block data is divided into hot data blocks, warm data blocks, and cold data blocks;

[0026] Based on the real-time status of each master node, a comprehensive status score for each master node is calculated using a pre-defined node status evaluation model.

[0027] Combining the comprehensive status score and the type of block data, each block data is matched with a corresponding master node and the corresponding data is written to obtain the updated master node;

[0028] Based on the data write status of the updated master node, the data increment is copied to the corresponding slave node to obtain the updated slave node.

[0029] Specifically, combining the comprehensive status score and the type of block data, each block data is matched with a corresponding master node and the corresponding data is written to obtain the updated master node, including:

[0030] Based on the comprehensive status score, the master nodes are layered to obtain multi-layer master nodes, which include high-level master nodes, middle-level master nodes and low-level master nodes.

[0031] For hot data blocks, they are matched to the corresponding first master node in the high-level master nodes according to the comprehensive status score from high to low.

[0032] For warm data blocks, they are matched to the corresponding second master node in the middle-level master node according to the comprehensive status score from high to low.

[0033] For cold data blocks, they are matched to the corresponding third master node in the lower-level master nodes according to the comprehensive status score from high to low.

[0034] The process continues until a corresponding master node is matched for each block of data. The block data is then written into the corresponding master node to obtain the updated master node.

[0035] Specifically, based on the data write status of the updated master node, the data increment is copied to the corresponding slave node to obtain the updated slave node, including:

[0036] Based on the data writing status of the master node and the status of the slave node, the corresponding slave node is matched through the preset master-slave node mapping relationship;

[0037] The data increment in the updated master node is copied to the corresponding slave node to obtain the updated slave node.

[0038] Specifically, responding to the user's data reading request, analyzing the access correlation between data, and fusing and reading the corresponding data from the update slave node includes:

[0039] In response to the user's data reading request, obtain the target data block to be read;

[0040] By analyzing the access relationships between data, the corresponding updates associated with the target data block are fused and read from the nodes.

[0041] Specifically, the step of analyzing the access correlations between data and fusing and reading the corresponding update from the node associated with the target data block includes:

[0042] Based on the access correlation between data, the data correlation access probability of each update slave node is calculated through a preset data block correlation analysis model;

[0043] The update slave nodes whose data association access probability is greater than a preset association access probability threshold are merged to obtain a merged slave node block;

[0044] Based on the update slave node corresponding to the target data block, and in order of the data association access probability from high to low, a preset number of update slave nodes are selected from the fused slave node block as a candidate access node set.

[0045] Read the corresponding target data block from the set of alternative access nodes.

[0046] A distributed block storage system performance optimization system, used to implement the aforementioned distributed block storage system performance optimization method, including:

[0047] The data partitioning module responds to the user's data storage needs by analyzing the access frequency of the data to be stored to dynamically divide it into blocks and generate multiple blocks of data.

[0048] The data storage node partitioning module dynamically classifies data storage nodes into master nodes and slave nodes based on the load status of each node in the storage system.

[0049] The data writing module writes each block of data to the corresponding master node according to the access frequency, and copies the data increment corresponding to each master node to the corresponding slave node to obtain the updated master node and the updated slave node.

[0050] The data reading module responds to the user's data reading needs, analyzes the access correlation between data, and merges and reads the corresponding data from the update slave node to optimize the performance of the distributed block storage system.

[0051] The beneficial effects of this application are: dynamic block and hierarchical storage of data, and updating the master-slave node relationship according to the real-time status of the system storage nodes, so that high-frequency data is concentrated on high-performance nodes and low-frequency data is allocated to low-load nodes, avoiding resource waste and improving overall storage efficiency; it can relieve the pressure on overloaded master nodes and quickly switch slave nodes to master nodes in case of failure, improving system performance; through incremental replication, separation of hot and cold data and dynamic resource allocation, it can improve system performance while reducing storage costs and network overhead. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the performance optimization method for a distributed block storage system in this application embodiment;

[0053] Figure 2 This is a schematic diagram of the auxiliary master node updating the master-slave node relationship diagram in the embodiments of this application;

[0054] Figure 3 This is a schematic diagram illustrating the replacement of the master node and updating the master-slave node relationship diagram in an embodiment of this application.

[0055] Figure 4 This is a schematic diagram illustrating the updating of data read from the node fusion in an embodiment of this application. Detailed Implementation

[0056] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0057] Hereinafter, the terms "first," "second," and other generic terms are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0058] refer to Figure 1 As shown, the specific implementation method of the distributed block storage system performance optimization method of this application includes:

[0059] S101. In response to the user's data storage needs, analyze the access frequency of the data to be stored to dynamically divide it into blocks and generate multiple blocks of data.

[0060] S102. Based on the load status of each node in the storage system, dynamically distinguish data storage nodes as master nodes and slave nodes;

[0061] S103. Based on the access frequency, write each block of data to the corresponding master node, and copy the data increment corresponding to each master node to the corresponding slave node to obtain the updated master node and the updated slave node.

[0062] S104. In response to the user's data reading request, analyze the access correlation between data, and merge and read the corresponding data from the update slave node to optimize the performance of the distributed block storage system.

[0063] This embodiment dynamically segments data and distinguishes storage nodes in the system by analyzing data access frequency, and stores the segmented data to the corresponding nodes. Through data write and fused read operations, it effectively improves the read and write performance of the distributed block storage system. This method can reduce data access latency by storing high-frequency and low-frequency data on high-performance and low-performance nodes respectively, making full use of the performance differences of various types of nodes, improving the utilization rate of storage and computing resources, and thus optimizing the overall performance of the distributed block storage system.

[0064] In this embodiment, upon receiving a user's data storage request, the data is dynamically segmented based on its data type and access frequency. Data with similar access frequencies are grouped together, and data with different access frequencies are allocated data blocks of different sizes. This dynamic segmentation method fully considers the impact of different data access frequencies on storage system performance, avoiding the interference problem caused by mixing high-frequency and low-frequency data in traditional methods. By distinguishing between storing high-frequency and low-frequency data, interference with high-frequency data access is reduced, thereby improving the system's speed in locating and retrieving high-frequency data and enhancing data access efficiency.

[0065] Specifically, during data read and write operations, the state of the system's data storage nodes changes accordingly. Based on real-time changes in node load, master and slave nodes are dynamically assigned. This dynamic node partitioning mechanism adapts to changes in node load over time and with varying data access patterns, overcoming the uneven load distribution and resource waste caused by traditional static master-slave partitioning. This allows each node to more evenly handle storage and access tasks, improving system stability and reliability. Furthermore, when a master node fails, the system can quickly switch to a slave node, enhancing the system's fault response capabilities and overall reliability.

[0066] Furthermore, data is written to the matching master node based on its access frequency, and the incremental data on the master node is synchronized to the corresponding slave node. By assigning appropriate master nodes to data with different access frequencies, the problem of writing high-frequency data to nodes with insufficient performance is avoided, thereby improving data writing efficiency. The incremental replication mechanism reduces the amount of data that needs to be transferred, improving backup and node update speed.

[0067] Specifically, during the data reading phase, upon receiving a user's data reading request, the system merges strongly correlated update slave nodes based on the target data block and the access relationships between data, forming a merged slave node block. Each time data is read, it directly retrieves data from the corresponding merged slave node block. This correlation analysis enables merged reading, effectively reducing the number of reads and time, and improving data retrieval efficiency and overall system performance.

[0068] This application dynamically segments and tiers data for storage, and updates the master-slave node relationship based on the real-time status of the system's storage nodes. This concentrates high-frequency data on high-performance nodes and distributes low-frequency data to low-load nodes, avoiding resource waste and improving overall storage efficiency. It can also alleviate the pressure on overloaded master nodes and quickly switch slave nodes to master nodes in case of failure, improving system performance. Through incremental replication, separation of hot and cold data, and dynamic resource allocation, it improves system performance while reducing storage costs and network overhead.

[0069] Furthermore, in response to users' data storage needs, the access frequency of the data to be stored is analyzed to dynamically divide it into blocks, generating multiple blocks of data, including:

[0070] S201. In response to the user's data storage needs, analyze and obtain the data type and data content of the data to be stored;

[0071] S202. Based on the data type, predict the access frequency of the data using a preset access frequency prediction model to obtain the predicted access frequency.

[0072] S203. Combining the predicted access frequency and the preset segmentation strategy, the data content is segmented into multiple blocks of data.

[0073] In this embodiment, when a user's data storage request is received, the data to be stored is first acquired, and its data type and content are analyzed. Since different types of data typically have different access patterns and frequencies—for example, system configuration files are accessed less frequently, while real-time monitoring data is accessed more frequently—based on the data type, a preset access frequency prediction model is used. This model includes, but is not limited to, a neural network trained on a large amount of historical data. This model can predict the future access frequency of data based on learned patterns. By obtaining the predicted access frequency, the system can identify high-frequency and low-frequency access data in advance, storing high-frequency data on high-performance storage devices and low-frequency data on low-cost devices, thus optimizing storage resource allocation.

[0074] Specifically, by combining predicted access frequency with a preset chunking strategy, data content is chunked. The chunking strategy is formulated based on various factors such as access frequency thresholds, data size, and business requirements. In this embodiment, data is divided into hot data, warm data, and cold data according to the predicted access frequency, and different chunk sizes are used accordingly: smaller chunks, such as 16KB-64KB, are used for hot data to support fast response to frequent access; medium chunks, such as 128KB-512KB, are used for warm data to balance access efficiency and storage cost; and larger chunks, such as 1MB-4MB, are used for cold data to reduce metadata management overhead. By analyzing data types and content, and combining access frequency prediction with reasonable chunking, data layout can be optimized based on actual access characteristics, improving read / write performance and storage efficiency. Furthermore, it can adapt to different types and access patterns of data, enhancing the system's flexibility and scalability.

[0075] Furthermore, based on the load status of each node in the storage system, data storage nodes are dynamically classified into master nodes and slave nodes, including:

[0076] S301. Based on the load status of the nodes in the storage system, calculate the load score of each node using a preset node load evaluation model.

[0077] S302. Nodes with load scores less than a preset first load threshold are designated as initial master nodes, and nodes with load scores greater than or equal to the preset first load threshold are designated as initial slave nodes.

[0078] S303. By analyzing the positional relationships between nodes and the consistency of data storage types, at least one initial slave node is assigned to each initial master node, and a master-slave node relationship graph is constructed.

[0079] S304. Based on the real-time status of the data storage nodes, update the master-slave node relationship diagram through a preset node update mechanism to obtain the real-time master node and slave node.

[0080] In this embodiment, the load status of each data storage node in the system is first analyzed and evaluated using a pre-defined node load evaluation model, and the load score of each node is calculated. Multiple node load indicators are collected using system monitoring tools or dedicated software, including CPU utilization, memory utilization, disk I / O read / write speed, and network bandwidth utilization. The node load evaluation model employs a weighted evaluation model, assigning corresponding weights based on the degree of influence of each indicator on the node load, and calculating the overall load score of the node. This provides a quantitative and intuitive representation of the node load status, facilitating comparison and analysis between nodes.

[0081] For example, let's set the weights for CPU utilization (0.3), memory utilization (0.2), disk I / O read / write speed (0.4), and network bandwidth utilization (0.1). If a node has a CPU utilization of 60%, a memory utilization of 50%, a disk I / O read / write speed of 80MB / s, and a network bandwidth utilization of 30%, then the node's load score would be: 0.3 × 60% + 0.2 × 50% + 0.4 × 80 + 0.1 × 30%. The final score is a dimensionless value, facilitating standardized comparisons.

[0082] Furthermore, based on the obtained node load scores and a preset first load threshold, nodes are distinguished into initial master nodes and slave nodes. The first load threshold can be set according to system performance and computing requirements. If a node's load score is lower than the first load threshold, it is marked as an initial master node; if it is greater than or equal to the first load threshold, it is marked as an initial slave node. Taking into account the availability of node resources, nodes with lower loads serve as master nodes to undertake the main data storage responsibilities, while nodes with higher loads serve as slave nodes for data backup and auxiliary reading. This avoids data concentration on a few nodes, achieves reasonable distribution of storage tasks among heterogeneous load nodes, and improves the overall performance and stability of the system.

[0083] After the initial master-slave node partitioning is completed, at least one slave node is assigned to each master node based on the location relationship between nodes and the consistency of storage data types. During allocation, slave nodes that are physically close to the master node, have matching storage types, and lower loads are prioritized to improve data transmission efficiency and ensure data consistency. Taking network latency and data affinity into account also helps improve system response speed and collaboration efficiency.

[0084] Furthermore, during system operation, the roles and corresponding relationships of master and slave nodes are dynamically adjusted based on the real-time status of the nodes and through a preset node update mechanism. The master-slave node relationship graph is updated in real time, ensuring that it always reflects the actual state of the system. Through real-time role adjustment and relationship optimization, the system can adapt to node load fluctuations, maintain load balance, improve performance and stability, and quickly restore data storage and access capabilities when nodes fail, further enhancing system reliability.

[0085] Furthermore, based on the real-time status of the data storage nodes, the master-slave node relationship graph is updated through a preset node update mechanism to obtain the real-time master and slave nodes, including:

[0086] S401. Based on the real-time status of the data storage nodes, update the load score of each node in real time to obtain the updated load score.

[0087] S402. When the update load score of the initial master node is greater than the preset second load threshold, select the node with the smallest update load score from the corresponding initial slave nodes as the auxiliary master node.

[0088] S403. When the initial master node fails or malfunctions, select the node with the lowest update load score from the corresponding initial slave nodes as the replacement master node.

[0089] S404. Update the master-slave node relationship graph based on the auxiliary master node and the replacement master node to obtain the real-time master node and slave node.

[0090] In this embodiment, key performance indicators of each data storage node are continuously collected using system monitoring tools. The load score is updated in real time based on dynamic changes in node status, providing an updated value reflecting the current load status of each node. Based on accurate load status, node roles and relationship diagrams are adjusted promptly, ensuring continuous, efficient, and stable system operation. The dynamic role allocation mechanism can quickly respond to changes in node load, guaranteeing system performance.

[0091] like Figure 2 As shown, the update load score of each initial master node is compared with a preset second load threshold, which can be set according to system performance and actual needs. If the update load score of an initial master node exceeds the second load threshold, it indicates that its load is too high and it is not suitable to continue as an independent master node. In this case, the initial slave node with the lowest update load score is selected from its initial slave nodes as an auxiliary master node, and the original master node and the auxiliary master node jointly undertake the data writing task. By introducing an auxiliary master node to distribute the load, the overall load distribution of the system can be optimized, and performance and stability can be improved.

[0092] like Figure 3 As shown, in a distributed storage system, the initial master node may fail due to hardware or software failures. Once a failure is detected, the initial slave node with the lowest update load score is selected from the initial slave nodes corresponding to the original master node to replace the failed master node. By quickly replacing the master node, the continuous availability of data can be maximized, system downtime can be shortened, and prioritizing low-load nodes helps improve resource utilization efficiency.

[0093] Furthermore, based on the identified auxiliary master node and replacement master node, the roles and connections of the corresponding nodes in the relationship graph are updated, including establishing connections related to the auxiliary master node and adjusting the correspondence between the replacement master node and the initial slave node. The updated master-slave node relationship graph is stored in the system storage module as the basis for subsequent data storage and access operations. By updating the relationship graph in real time to match the actual system state, data errors or operational anomalies caused by inaccurate master-slave relationships can be avoided, ensuring the correct execution of data storage, backup, and retrieval processes.

[0094] Furthermore, based on the access frequency, each block of data is written to the corresponding master node, and the data increments corresponding to each master node are copied to the corresponding slave nodes, resulting in updated master nodes and updated slave nodes, including:

[0095] S501. Based on the access frequency, divide the block data into hot data blocks, warm data blocks and cold data blocks;

[0096] S502. Based on the real-time status of each master node, calculate the comprehensive status score of each master node through a preset node status evaluation model.

[0097] S503. Combining the comprehensive status score and the type of block data, match each block data to the corresponding master node and write the corresponding data to obtain the updated master node;

[0098] S504. Based on the data write status of the updated master node, the data increment is copied to the corresponding slave node to obtain the updated slave node.

[0099] In this embodiment, the block data is divided into hot data blocks, warm data blocks, and cold data blocks according to the access frequency. Hot data blocks are accessed frequently and require fast response, while cold data blocks are accessed less frequently and have lower response speed requirements. Warm data blocks are between hot data blocks and cold data blocks in terms of access frequency and performance requirements. They are stored in the corresponding data nodes according to the data type, and a corresponding storage node is allocated to each data block.

[0100] Meanwhile, the storage performance and status of master nodes are evaluated through a preset node status evaluation model, and the comprehensive status score of each master node is calculated. The node status evaluation model includes, but is not limited to, a weighted model. Based on the collected master node status indicators, including CPU utilization, memory utilization, disk space remaining and network bandwidth utilization, the model assigns corresponding weights according to the impact of each indicator on node performance, and calculates the comprehensive status score by weighting. The combination of multiple indicators can comprehensively reflect the actual status of master nodes, overcome the limitations of single indicator evaluation, and provide a basis for matching data blocks with nodes, ensuring efficient and stable data writing.

[0101] Specifically, the master nodes are stratified based on the comprehensive status score, and different types of data blocks are matched with nodes of the corresponding level. Each data block is assigned a master node with appropriate performance and the write is performed. This node becomes the update master node. Through this node matching mechanism, hot data blocks are written to high-performance master nodes to ensure access speed, while cold data blocks are stored in lower-performance nodes to save storage costs. At the same time, the data is avoided from being written to a few nodes in a concentrated manner, so as to achieve a relatively balanced load and improve the overall performance of the system.

[0102] Furthermore, after the master node completes the data writing, the incremental data on the master node is copied to the corresponding slave node. By monitoring the write operation of the master node in real time, the data change is recorded to obtain the data increment. Then, according to the preset master-slave node mapping relationship, the incremental data is copied to the corresponding slave node. After the copy is completed, the slave node becomes the update slave node. Through the incremental copy mechanism, data backup can be achieved, ensuring the recoverability and availability of data in the event of a master node failure. The slave node handles part of the read requests to reduce the pressure on the master node. Copying only the changed data reduces the amount of transmission and storage overhead, thus improving system efficiency.

[0103] Furthermore, combining the overall status score and the type of block data, each block data is matched with a corresponding master node and the corresponding data is written to obtain the updated master node, including:

[0104] S601. Based on the comprehensive status score, the master nodes are layered to obtain multi-level master nodes, which include high-level master nodes, middle-level master nodes and low-level master nodes.

[0105] S602. For hot data blocks, match them to the corresponding first master node in the high-level master nodes according to the comprehensive status score from high to low.

[0106] S603. For warm data blocks, match them to the corresponding second master node in the middle-level master node according to the comprehensive status score from high to low.

[0107] S604. For cold data blocks, match them to the corresponding third master node in the lower-level master nodes according to the comprehensive status score from high to low.

[0108] S605. Continue until each block of data is matched with the corresponding master node, write the block data into the corresponding master node, and obtain the updated master node.

[0109] In this embodiment, master nodes are stratified based on the calculated comprehensive status score. According to the actual performance requirements of the system, a stratification threshold is preset. For example, master nodes with a comprehensive status score greater than or equal to 80 are classified as high-level master nodes, master nodes with a comprehensive status score greater than 40 and less than 80 are classified as mid-level master nodes, and master nodes with a comprehensive status score less than or equal to 40 are classified as low-level master nodes. Through the master node stratification mechanism, nodes with different performance are clearly distinguished, so that data can be stored in the most suitable node according to its characteristics, thereby improving the overall performance of the storage system.

[0110] For hot data blocks with high access frequency and strict performance requirements, the high-level master nodes are sorted in descending order of comprehensive status score, and the corresponding high-level master node is selected from the head of the sorted list as the first master node for each hot data block. The high-level master nodes have excellent performance, which can meet the high-speed read and write requirements of hot data blocks and effectively reduce access latency.

[0111] For warm data blocks with moderate access frequency and performance requirements, the intermediate master nodes are sorted in descending order of comprehensive status score, and the corresponding intermediate master node is selected from the head of the sequence as the second master node for each warm data block. The performance of this node is moderate, which can meet the storage needs of warm data blocks, and avoid occupying the resources of higher-level nodes or affecting access performance due to storage in lower-level nodes.

[0112] For cold data blocks with low access frequency and relaxed performance requirements, the lower-level master nodes are arranged in descending order of comprehensive status score, and the corresponding lower-level master node is selected from the front of the sequence as the third master node for each cold data block. The lower-level master nodes have low comprehensive status scores and relatively economical costs. Using them to store cold data can save storage costs while ensuring basic storage needs.

[0113] After matching each data block with its corresponding master node, the data is written to the corresponding node to complete the storage. During the writing process, measures such as verification are taken to ensure data integrity and consistency. After writing, the status information of the master node is updated, including disk usage and load status, thereby updating the master node, realizing persistent data storage, and ensuring its security and availability.

[0114] Furthermore, based on the data write status of the updated master node, the incremental data is copied to the corresponding slave nodes to obtain the updated slave nodes, including:

[0115] S701. Based on the data writing status of the master node and the status of the slave node, match the corresponding slave node through the preset master-slave node mapping relationship;

[0116] S702. Copy the incremental data from the updated master node to the corresponding slave node to obtain the updated slave node.

[0117] In this embodiment, after the master node completes the data writing, the corresponding data increment is synchronized to the corresponding slave node. The system continuously monitors the write operation of the master node, records the amount of data written, time, and data block location, and obtains the status information of the slave node. Based on the preset master-slave node mapping relationship, combined with the data allocation and load status of the slave node, the system selects a slave node with sufficient resources and appropriate load for data replication, so as to avoid replication to resource-constrained or high-load nodes, thereby improving the system resource utilization efficiency.

[0118] Specifically, after determining the corresponding slave node for the master node, the incremental data that has changed in the master node is copied to the selected slave node. Based on the write information recorded by the master node, the incremental data to be copied is identified, and efficient data transmission protocols such as NFS and iSCSI are used to send the incremental data to the corresponding slave node. After receiving the incremental data, the slave node writes it to its local storage device and updates its own status information after the copy is completed, thus becoming the update slave node. Through the incremental copy mechanism, the amount of data transmission and copy time are effectively reduced, the data synchronization efficiency is improved, and the system load is reduced.

[0119] Furthermore, in response to user data retrieval needs, the system analyzes the access relationships between data and merges and reads the corresponding data from the update slave nodes, including:

[0120] S801, In response to the user's data reading request, obtain the target data block to be read;

[0121] S802. By analyzing the access correlation between data, the corresponding update associated with the target data block is fused and read from the node.

[0122] In this embodiment, after a user initiates a data read request to the distributed block storage system, the system responds and determines the target data block to be read, and locates the slave node storing the data block.

[0123] Specifically, based on the access correlation between the target data block and other data, associated data blocks are identified, and their corresponding update slave nodes are determined for fusion. By pre-fetching related data that users may access, data reading speed can be improved. By obtaining relevant data block information in advance based on the correlation characteristics between data accesses, user waiting time is effectively shortened, thereby improving data reading efficiency and overall system response performance.

[0124] Furthermore, by analyzing the access relationships between data, the corresponding updates associated with the target data block are fused and read from the nodes, including:

[0125] S901. Based on the access correlation between data, calculate the data association access probability of each update slave node through the preset data block association analysis model;

[0126] S902. Merge update slave nodes whose data association access probability is greater than the preset association access probability threshold to obtain a merged slave node block;

[0127] S903. Based on the update slave nodes corresponding to the target data block, select a preset number of update slave nodes from the fused slave node block in descending order of data association access probability as a candidate access node set.

[0128] S904. Read the corresponding target data block from the set of alternative access nodes.

[0129] In this embodiment, based on the access correlation between data, a preset data block correlation analysis model, including but not limited to the Apriori algorithm, is used to analyze the data access correlation and calculate the correlation access probability between the data blocks stored in each update slave node and the target data block. This model is trained based on a large amount of historical data and can evaluate the importance of each update slave node's data in the current read operation and the likelihood of it being accessed.

[0130] like Figure 4 As shown, based on the actual system conditions and experience, a threshold for associated access probability is set, and update slave nodes whose data associated access probability exceeds the threshold are selected for fusion. These update slave nodes that are closely associated with the target data block and have a high access probability are combined into a fused slave node block. Through node fusion, nodes that are of great significance to the current read operation can be focused on, thereby improving the targeting of the read.

[0131] Specifically, after forming the fused slave node block, the update slave node containing the target data block is determined, and the update slave nodes in the fused slave node block are sorted from high to low according to the data association access probability. A top node is selected from the sorted update slave nodes according to a preset number to form a candidate access node set. This prioritizes the node with the highest association degree, increasing the probability of successfully obtaining the target data and reducing unnecessary node access.

[0132] Furthermore, by reading the corresponding target data block from the set of candidate access nodes, when a user requests relevant data, the data can be read directly from the set of candidate access nodes. This avoids having to search for the next target data from the nodes again during a full update, thereby improving the data reading speed and reducing reading time and resource consumption. By centrally reading from the nodes, the performance of the distributed block storage system is optimized.

[0133] A performance optimization system for distributed block storage systems, used to implement methods for optimizing the performance of distributed block storage systems, including:

[0134] The data partitioning module responds to the user's data storage needs by analyzing the access frequency of the data to be stored to dynamically divide it into blocks and generate multiple blocks of data.

[0135] The data storage node partitioning module dynamically classifies data storage nodes into master nodes and slave nodes based on the load status of each node in the storage system.

[0136] The data writing module writes each block of data to the corresponding master node according to the access frequency, and copies the data increment corresponding to each master node to the corresponding slave node to obtain the updated master node and updated slave node.

[0137] The data reading module responds to users' data reading needs, analyzes the access correlation between data, and merges and reads the corresponding data from the update slave nodes to optimize the performance of the distributed block storage system.

[0138] In this embodiment, the data partitioning module responds to the user's data storage needs by analyzing and processing the data to be stored, dynamically dividing it into multiple data blocks. Based on the prediction of data access frequency, it optimizes resource allocation during subsequent storage and retrieval processes, thereby improving storage efficiency and data access performance. The data storage node partitioning module dynamically divides nodes into master nodes and slave nodes according to the load status of nodes in the storage system, and constructs and updates the master-slave node relationship graph. By rationally partitioning nodes, the storage system load is balanced, improving system performance and stability; at the same time, using slave nodes as data backups for master nodes enhances data reliability and availability.

[0139] Specifically, the data writing module writes block data to the matching master node based on data access frequency and the real-time status of the master node, and replicates the incremental data to the corresponding slave nodes. By optimizing storage resource allocation, it improves data writing efficiency and reliability. The data reading module responds to user data read requests, analyzes the access correlation between data, performs fused read operations on updated slave nodes, accurately locates slave nodes associated with the target data block, and prioritizes reading data from closely associated nodes, reducing the number of reads and time, improving data read efficiency and accuracy, and thus optimizing the performance of the distributed block storage system.

[0140] The above description is merely a preferred embodiment of this application. The scope of protection of this application is not limited to the above embodiments. All technical solutions falling within the scope of this application's concept are within the scope of protection of this application. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of this application should also be considered within the scope of protection of this application.

Claims

1. A performance optimization method for a distributed block storage system, characterized in that, include: In response to users' data storage needs, the access frequency of the data to be stored is analyzed to dynamically divide it into blocks and generate multiple blocks of data. Based on the load status of each node in the storage system, data storage nodes are dynamically classified into master nodes and slave nodes. Based on the access frequency, each block of data is written to the corresponding master node, and the data increment corresponding to each master node is copied to the corresponding slave node to obtain the updated master node and the updated slave node. In response to the user's data reading request, obtain the target data block to be read; Based on the access correlation between data, the data correlation access probability of each update slave node is calculated through a preset data block correlation analysis model. The access correlation reflects the correlation access probability between data and is used to evaluate the importance of each update slave node's data in the current read operation and the likelihood of it being accessed. The update slave nodes whose data association access probability is greater than a preset association access probability threshold are merged to obtain a merged slave node block; Based on the update slave node corresponding to the target data block, and in order of the data association access probability from high to low, a preset number of update slave nodes are selected from the fused slave node block as a candidate access node set. The corresponding target data block is read from the set of candidate access nodes to optimize the performance of the distributed block storage system.

2. The performance optimization method for a distributed block storage system according to claim 1, characterized in that, In response to the user's data storage needs, the access frequency of the data to be stored is analyzed to dynamically divide it into blocks, generating multiple blocks of data, including: In response to users' data storage needs, the data type and content of the data to be stored are analyzed. Based on the data type, the access frequency of the data is predicted using a preset access frequency prediction model to obtain the predicted access frequency. Based on the predicted access frequency and the preset segmentation strategy, the data content is segmented into multiple blocks of data.

3. The performance optimization method for a distributed block storage system according to claim 1, characterized in that, The method of dynamically classifying data storage nodes into master nodes and slave nodes based on the load status of each node in the storage system includes: Based on the load status of nodes in the storage system, the load score of each node is calculated using a preset node load evaluation model. Nodes with load scores less than a preset first load threshold are designated as initial master nodes, and nodes with load scores greater than or equal to the preset first load threshold are designated as initial slave nodes. By analyzing the positional relationships between nodes and the consistency of data storage types, at least one initial slave node is assigned to each initial master node, and a master-slave node relationship graph is constructed. Based on the real-time status of the data storage nodes, the master-slave node relationship graph is updated through a preset node update mechanism to obtain the real-time master and slave nodes.

4. The performance optimization method for a distributed block storage system according to claim 3, characterized in that, The step of updating the master-slave node relationship graph according to the real-time status of the data storage nodes through a preset node update mechanism to obtain the real-time master and slave nodes includes: The load score of each node is updated in real time based on the real-time status of the data storage nodes to obtain the updated load score. When the update load score of the initial master node is greater than the preset second load threshold, the node with the smallest update load score is selected from the corresponding initial slave nodes and used as the auxiliary master node. When the initial master node fails or malfunctions, the node with the lowest update load score among the corresponding initial slave nodes is selected as the replacement master node. Based on the auxiliary master node and the replacement master node, the master-slave node relationship graph is updated to obtain the real-time master node and slave node.

5. The performance optimization method for a distributed block storage system according to claim 1, characterized in that, Based on the access frequency, each block of data is written to the corresponding master node, and the data increment corresponding to each master node is copied to the corresponding slave node, resulting in updated master nodes and updated slave nodes, including: Based on access frequency, block data is divided into hot data blocks, warm data blocks, and cold data blocks; Based on the real-time status of each master node, a comprehensive status score for each master node is calculated using a pre-defined node status evaluation model. Combining the comprehensive status score and the type of block data, each block data is matched with a corresponding master node and the corresponding data is written to obtain the updated master node; Based on the data write status of the updated master node, the data increment is copied to the corresponding slave node to obtain the updated slave node.

6. The performance optimization method for a distributed block storage system according to claim 5, characterized in that, Combining the overall status score and the type of block data, each block data is matched to a corresponding master node and the corresponding data is written to obtain the updated master node, including: Based on the comprehensive status score, the master nodes are layered to obtain multi-layer master nodes, which include high-level master nodes, middle-level master nodes and low-level master nodes. For hot data blocks, they are matched to the corresponding first master node in the higher-level master nodes according to the comprehensive status score from high to low. For warm data blocks, they are matched to the corresponding second master node in the middle-level master node according to the comprehensive status score from high to low. For cold data blocks, they are matched to the corresponding third master node in the lower-level master nodes according to the comprehensive status score from high to low. The process continues until a corresponding master node is matched for each block of data. The block data is then written into the corresponding master node to obtain the updated master node.

7. The performance optimization method for a distributed block storage system according to claim 5, characterized in that, Based on the data write status of the updated master node, the incremental data is copied to the corresponding slave node to obtain the updated slave node, including: Based on the data writing status of the master node and the status of the slave node, the corresponding slave node is matched through the preset master-slave node mapping relationship; The incremental data in the updated master node is copied to the corresponding slave node to obtain the updated slave node.

8. A performance optimization system for distributed block storage systems, characterized in that, A method for optimizing the performance of a distributed block storage system as described in any one of claims 1 to 7, comprising: The data partitioning module responds to the user's data storage needs by analyzing the access frequency of the data to be stored to dynamically divide it into blocks and generate multiple blocks of data. The data storage node partitioning module dynamically classifies data storage nodes into master nodes and slave nodes based on the load status of each node in the storage system. The data writing module writes each block of data to the corresponding master node according to the access frequency, and copies the data increment corresponding to each master node to the corresponding slave node to obtain the updated master node and the updated slave node. The data reading module, in response to the user's data reading request, acquires the target data block to be read; based on the access correlation between data, it calculates the data association access probability of each update slave node through a preset data block association analysis model. The access correlation reflects the association access probability between data and is used to evaluate the importance and access probability of each update slave node's data in the current read operation; update slave nodes with data association access probabilities greater than a preset association access probability threshold are merged to obtain a merged slave node block; based on the update slave nodes corresponding to the target data block, a preset number of update slave nodes are selected from the merged slave node block in descending order of data association access probability as a candidate access node set; the corresponding target data block is read from the candidate access node set to optimize the performance of the distributed block storage system.

Images