An efficient distributed storage method, system, device and medium

Abstract

The present application relates to an efficient distributed storage method, system, device and medium, belonging to the field of distributed storage technology. The storage method includes: receiving a data storage request from a business system; parsing the data storage request and outputting a data unit containing a storage policy tag; the storage policy tag includes a data sharding granularity level and a fault tolerance level; splitting the data unit into data shards of corresponding sizes according to the data sharding granularity level; determining the number of replicas according to the fault tolerance level, generating a replica distribution strategy based on the distributed cluster topology, and specifying a master replica node and a slave replica node list; writing the data shards to the master replica node according to the replica distribution strategy, and synchronously initiating a replica copy operation to the slave replica node; receiving a storage confirmation signal from the master replica node, and returning a storage success response to the business system. The present application can improve the data storage reliability of the business system.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to an efficient distributed storage method, system, device and medium. Background Art

[0002] Distributed storage systems have been widely used in various large-scale data management applications in recent years, particularly in cloud computing and big data processing. With the rapid growth of data volumes, achieving efficient storage, fast access, and high-availability management of large-scale data has become a key issue in technological development. The design of distributed storage systems typically involves distributing large amounts of data across multiple physical storage nodes while ensuring redundant backup, fault tolerance, and load balancing within the system. These challenges stem from the complex balance between data storage reliability, access performance, and fault recovery capabilities.

[0003] In traditional storage systems, data is typically stored on a single node or a small number of nodes. This storage approach often leads to bottlenecks when faced with large amounts of data and frequent access. For example, a storage node failure can result in data loss or service interruption. Furthermore, the read and write speeds and load capacity of a single storage node are limited, making it unable to meet the demands of large-scale, highly concurrent access scenarios. To address these issues, distributed storage systems have emerged. By distributing data across multiple nodes and creating multiple copies of each data, they provide enhanced fault tolerance and scalability.

[0004] However, current distributed storage systems fail to fully consider the differences between different data types and business needs, and often lack the ability to dynamically adjust in actual applications, resulting in the system's storage efficiency, fault tolerance, and access performance not being fully utilized, thereby affecting the data storage reliability of the business system. Summary of the Invention

[0005] In order to improve the data storage reliability of business systems, the present application provides an efficient distributed storage method, system, device and medium.

[0006] In a first aspect, the present application provides an efficient distributed storage method, which adopts the following technical solutions:

[0007] An efficient distributed storage method, comprising:

[0008] Receive data storage requests from business systems;

[0009] Parsing the data storage request and outputting a data unit including a storage policy tag; the storage policy tag includes a data sharding granularity level and a fault tolerance level;

[0010] Splitting the data unit into data fragments of corresponding sizes according to the data fragmentation granularity level;

[0011] Determine the number of replicas according to the fault tolerance level, generate a replica distribution strategy based on the distributed cluster topology, and specify a list of master replica nodes and slave replica nodes;

[0012] According to the replica distribution strategy, the data shards are written to the master replica node, and a replica replication operation is synchronously initiated to the slave replica node;

[0013] Receive the storage confirmation signal from the master and replica nodes, and return a storage success response to the business system.

[0014] By adopting the above technical solutions, based on refined storage strategies and dynamic adjustment mechanisms, significant advantages are achieved in improving the availability, fault tolerance, and performance of distributed storage systems. First, by analyzing the content of storage requests and parsing storage policy tags, the system can automatically adjust storage strategies based on different data requirements, ensuring that the storage method of each data unit can be optimized. Second, through flexible sharding strategies and replica distribution strategies, the system can efficiently manage data in a large-scale distributed environment, ensuring high data availability and fast access capabilities. Finally, through replica replication and load monitoring mechanisms, the storage system can achieve dynamic load balancing and high fault tolerance, ensuring data integrity and continuous service availability even in extreme situations.

[0015] Optionally, the steps of determining the number of replicas according to the fault tolerance level, generating a replica distribution strategy based on the distributed cluster topology, and specifying a list of master replica nodes and slave replica nodes include:

[0016] Get the availability zone identifier of the data unit's business request source;

[0017] Querying a preset replica quantity mapping table according to the fault tolerance level of the data unit, and outputting a replica quantity value;

[0018] Based on the distributed cluster topology data and the availability zone identifier of the business request source, select the candidate node that meets the first network delay threshold and has the highest health score as the master replica node, and output the master replica node identifier;

[0019] According to the replica quantity value and the master replica node identifier, select candidate nodes in different availability zones from the master replica node and meeting a second network delay threshold as slave replica nodes, and output a list of slave replica nodes;

[0020] The master replica node identifier and the slave replica node list are bound to the data shard identifier, and a replica distribution strategy is generated and stored in a distributed configuration center.

[0021] By adopting the above technical solution, based on a flexible replication strategy generation method, combined with the fault tolerance requirements of business data units and network topology, the system intelligently selects a list of master and slave replica nodes, ensuring high data availability, redundancy, and access efficiency. By considering network latency, health scores, and the geographic distribution of nodes, the system optimizes the load balancing and fault tolerance capabilities of the distributed storage system. The generated replica distribution strategy file ensures consistent and synchronized configuration across all nodes, thereby improving system reliability and performance.

[0022] Optionally, the step of selecting a candidate node that meets a first network latency threshold and has a highest health score as a primary replica node based on the distributed cluster topology data and the availability zone identifier of the service request source includes:

[0023] Obtain the availability zone location and historical failure rate of each candidate node from the distributed cluster topology data;

[0024] Collect the load indicators of each candidate node in real time;

[0025] Calculate a health score for each candidate node based on the load metric and historical failure rate;

[0026] Based on the available zone identifier of the service request source, candidate nodes that are in the same zone as the available zone of the service request source and that meet a first network delay threshold are screened to obtain a first candidate node set;

[0027] The first candidate node set is sorted according to the health score, and the candidate node with the highest health score is selected as the primary replica node.

[0028] By employing the aforementioned technical solution, combined with distributed cluster topology data, real-time load monitoring, historical failure rates, and network latency, the system comprehensively evaluates the health of each candidate node and intelligently selects primary and secondary nodes. By accurately screening low-latency nodes in the same region and sorting them according to health scores, the system ensures the selection of a high-performing and stable primary and secondary node, maximizing data access efficiency and the system's fault tolerance. Ultimately, this solution implements an efficient replica distribution strategy, ensuring high availability, low latency, and high reliability for the distributed storage system.

[0029] Optionally, the step of selecting, based on the replica quantity value and the master replica node identifier, a candidate node in a different availability zone from the master replica node and meeting a second network delay threshold as a slave replica node includes:

[0030] Based on the primary and replica node identifiers, candidate nodes in different availability zones from the primary and replica nodes are screened to obtain a second candidate node set;

[0031] Calculating the network delay between each candidate node in the second candidate node set and the primary replica node;

[0032] Screening candidate nodes whose network delays meet a second network delay threshold and sorting them in ascending order according to the network delays;

[0033] According to the replica number value N, the first N-1 candidate nodes are selected as slave replica nodes.

[0034] By employing the above technical solutions, we accurately select nodes in different availability zones from the primary replica node and optimize the selection process for secondary replica nodes based on network latency and the number of replicas required. First, we ensure that secondary replica nodes are distributed across different availability zones, thereby improving the system's fault tolerance. Second, by calculating and ranking the network latency between nodes, we ensure that the node with the lowest latency is selected as the secondary replica node, thereby improving data access speed and synchronization efficiency.

[0035] Optionally, according to the replica distribution strategy, the step of writing the data shard to the master replica node and synchronously initiating a replica copy operation to the slave replica node includes:

[0036] Parse the replica distribution strategy, extract the master replica node identifier, slave replica node list and synchronization protocol type;

[0037] Send a data shard write request to the master replica node, and receive a write timestamp and data check code returned by the master replica node;

[0038] Synchronously initiating a data replication request from the replica node list; wherein the data replication request includes the data shard content, the data check code, and the write timestamp;

[0039] Collect the response status of each slave replica node and determine the data consistency result based on the synchronization protocol type and the number of successfully responding replicas;

[0040] According to the failed node list in the data consistency result, the replica data is retried for replication or the replica distribution strategy is triggered to update.

[0041] By employing the aforementioned technical solution and parsing the replica distribution policy file, the system obtains the node information and synchronization protocol type for the write request. Data is then written to and synchronized to the slave replica nodes via the master replica node, ensuring redundant data storage and high availability. During data synchronization, the system determines consistency based on the response status and handles it according to the different synchronization protocol types. For failed nodes, the system implements a retry mechanism and, if necessary, updates the replica distribution policy to address replica node failures or network issues. This overall solution ensures the stability, reliability, and data consistency of the distributed storage system through efficient data writing, replication, and consistency verification mechanisms.

[0042] Optionally, after the step of writing the data shard to the master replica node and synchronously initiating a replica copy operation to the slave replica node according to the replica distribution strategy, the step further includes:

[0043] Collecting load indicators of the master replica node and the slave replica node in real time;

[0044] Calculate load scores based on pre-built load assessment models;

[0045] Determine whether the load score of the master and replica nodes is lower than a score threshold for a consecutive preset number of times;

[0046] If so, select the slave replica node with the highest load score and meeting the topology constraints from the list of slave replica nodes as the new master replica node;

[0047] Incrementally synchronize the data shards from the original master replica node to the new master replica node, and update the replica distribution strategy after the synchronization is completed.

[0048] The above technical solution effectively improves the load balancing, fault tolerance, and overall performance of the distributed storage system. By collecting and evaluating the load of each node in real time, the system automatically migrates the primary and replica nodes when overloaded, ensuring high system availability and avoiding the impact of single points of failure on service. Incremental synchronization and topology update strategies ensure data consistency and efficient migration.

[0049] In a second aspect, the present application provides an efficient distributed storage system, which adopts the following technical solutions:

[0050] An efficient distributed storage system, comprising:

[0051] A receiving module, used to receive data storage requests from business systems;

[0052] A parsing module, configured to parse the data storage request and output a data unit including a storage policy tag; the storage policy tag includes a data sharding granularity level and a fault tolerance level;

[0053] A data splitting module, configured to split the data unit into data fragments of corresponding sizes according to the data fragmentation granularity level;

[0054] A replica distribution strategy generation module is used to determine the number of replicas according to the fault tolerance level, generate a replica distribution strategy based on the distributed cluster topology, and specify a list of master replica nodes and slave replica nodes;

[0055] A data storage module, configured to write the data shards into the master replica node and synchronously initiate a replica replication operation to the slave replica node according to the replica distribution strategy;

[0056] The storage success response module is used to receive the storage confirmation signal of the master and replica nodes and return a storage success response to the business system.

[0057] Optionally, the storage system further includes:

[0058] A load index collection module is used to collect the load index of the master replica node and the slave replica node in real time;

[0059] A load score calculation module, used to calculate a load score based on a pre-built load assessment model;

[0060] a judgment module, configured to judge whether the load score of the master and replica nodes is lower than the score threshold for a consecutive preset number of times; if so, output a first judgment result;

[0061] a node screening module, configured to select, in response to the first judgment result, a slave replica node with the highest load score and satisfying the topology constraint from the slave replica node list as a new master replica node;

[0062] The data incremental synchronization module is used to incrementally synchronize data shards from the original master replica node to the new master replica node;

[0063] The strategy update module is used to update the replica distribution strategy after synchronization is completed.

[0064] In a third aspect, the present application provides a computer device that adopts the following technical solution:

[0065] A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to the first aspect.

[0066] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0067] A computer-readable storage medium stores a computer program capable of being loaded by a processor and executing any one of the methods in the first aspect.

[0068] In summary, the present application includes at least one of the following beneficial technical effects: by combining the parsing of data storage requests, data sharding, determination of the number of replicas based on the fault tolerance level, and generation of a replica distribution strategy for the distributed cluster topology, the efficiency and reliability of distributed storage are optimized. Through a reasonable master and slave replica node selection and synchronization mechanism, high data availability and fast access are ensured, while the system's load balancing capabilities are improved. The present application can automatically adjust storage strategies to suit different business needs, enhance the flexibility and fault tolerance of the storage system, effectively reduce the risk of data loss, and improve the stability and efficiency of storage operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a first flow chart of an efficient distributed storage method according to one of the embodiments of the present application.

[0070] Figure 2 This is a second flow chart of an efficient distributed storage method according to one of the embodiments of the present application.

[0071] Figure 3 This is a third flow chart of an efficient distributed storage method according to one of the embodiments of the present application.

[0072] Figure 4 This is a fourth flow chart of an efficient distributed storage method according to one of the embodiments of the present application.

[0073] Figure 5 This is the fifth flow chart of an efficient distributed storage method in one of the embodiments of the present application.

[0074] Figure 6 This is the sixth flow chart of an efficient distributed storage method according to one of the embodiments of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-6 It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0076] The embodiments of the present application disclose an efficient distributed storage method.

[0077] Reference Figure 1 , an efficient distributed storage method, the storage method includes:

[0078] Step S101: receiving a data storage request from a business system;

[0079] In a distributed storage system, the system needs to store data generated by business systems. Data storage requests typically include specific business requirements, including the data type (such as text, images, and videos), its criticality, and storage requirements. The storage system's receiving operation essentially connects to a request interface, processing the externally input storage instructions and data content. This is the most fundamental step in the entire storage process, ensuring that data can enter the storage process and prepare for subsequent storage processing.

[0080] For example, if it's an online e-commerce platform, the requests received in this step might involve storing order data, customer data, or transaction logs. The business system might also specify the importance of certain data, such as "normal order data" or "high-priority financial data." These requests might originate from different modules, such as the order management module or the user management module. After the system receives these requests, the subsequent processing flow relies on parsing and understanding the request content to ensure that the data is properly allocated, stored, and managed.

[0081] Step S102: Parse the data storage request and output a data unit including a storage policy tag; wherein the storage policy tag includes a data sharding granularity level and a fault tolerance level;

[0082] Specifically, after receiving a data storage request, the storage system needs to parse the request content. The key to this step is determining the data storage policy based on the specific content of the request, defining the basic properties of data storage based on different data types and criticality levels, and outputting data units with storage policy tags. This process involves two key operations: first, parsing the "data type" and "criticality level" in the data request, and then generating a storage policy tag based on the parsed results. The tag mainly includes the data sharding granularity and fault tolerance level.

[0083] The data sharding granularity level refers to the granularity of the data when it is split into several independent parts, and there are usually two options: larger granularity and smaller granularity. Larger granularity sharding means that each data unit is split into relatively large blocks, which is suitable for scenarios that require overall operations; while smaller granularity sharding is suitable for scenarios with frequent access and modification because it can improve the parallel processing capabilities and flexibility of the storage system. The fault tolerance level refers to the redundancy requirement of the data. Generally, the higher the fault tolerance level, the more copies of the data. For example, for important data, the storage system may require at least 3 copies to ensure that if one copy is lost, it can still be recovered from other copies.

[0084] Specifically, when parsing a request to store order data, the storage system might discover that the requested data type is "order data" and is of high importance. Therefore, the system might generate a "high fault tolerance" and "medium granularity" tag. This means that the order data will be split into multiple shards and stored in three copies to ensure high data availability.

[0085] Step S103: split the data unit into data fragments of corresponding sizes according to the data fragmentation granularity level;

[0086] Among them, after receiving and parsing the data storage request, the system begins to split the data unit according to the storage policy label. Sharding is a common operation in distributed storage, which splits a large data unit into several smaller shards for easy storage, management and access. The granularity of the shard is determined according to the shard granularity level defined in the previous step. For larger data units, the storage system will decide the splitting method based on the granularity level. For scenarios that require higher concurrent access, the shard granularity may be selected to be smaller to ensure that each small shard can operate in parallel on multiple nodes, thereby improving the throughput and access speed of the storage system. For some data with less access and a single data type, a larger granularity may be used for sharding to reduce the management overhead brought by sharding.

[0087] For example, when a storage system receives a request for order data, defined as "medium granularity" based on business requirements, it might divide each order into multiple data blocks, such as the order header, order items, and payment information, and store them as separate shards. Each shard can be accessed independently, increasing storage and access flexibility. Furthermore, the shard design can be optimized based on the data's usage scenario; for example, frequently accessed shards might be assigned to higher-performance storage nodes.

[0088] As you can see, this sharding approach not only improves concurrent access capabilities but also optimizes disk space usage in large-scale data storage, increasing data retrieval efficiency and storage resource utilization. Through fine-grained sharding, the system can flexibly control data allocation and access, avoiding the performance bottlenecks that can arise from storing a single large block of data.

[0089] Step S104: determining the number of replicas according to the fault tolerance level, generating a replica distribution strategy based on the distributed cluster topology, and specifying a list of master replica nodes and slave replica nodes;

[0090] The storage system determines the number of replicas based on the data's fault tolerance level and generates a replica distribution strategy based on the distributed cluster's topology. The number of replicas typically depends on the required fault tolerance level; for example, a higher fault tolerance level might require multiple replicas. The replica distribution strategy takes into account the storage cluster's node topology, the geographic location between nodes, and network latency to ensure even distribution of replicas and minimize the risk of single points of failure.

[0091] Specifically, generating a replica distribution strategy is a complex process. The system must not only ensure that replicas meet redundancy requirements but also select appropriate nodes to store the master and slave replicas based on the cluster topology. The master replica is typically stored on a node with high access frequency or low latency, while slave replicas can be distributed across other nodes, typically with a reasonable latency to the master node. This ensures that even if a replica node fails, the remaining replicas maintain data integrity and availability.

[0092] For example, if an e-commerce platform's order data requires high availability, the storage system might create three replicas for each data shard. One replica is stored on the primary replica node, while the other two replicas are stored on appropriate secondary replica nodes based on the cluster topology to ensure redundancy and low latency. This rational replica distribution strategy ensures high availability, data redundancy, and fault tolerance. This optimized replica distribution not only ensures data consistency but also improves system reliability and resilience.

[0093] Step S105: write the data shards to the master replica node according to the replica distribution strategy, and synchronously initiate a replica replication operation to the slave replica node;

[0094] Based on the replica distribution strategy, the system writes data shards to the master replica node and simultaneously initiates data replication requests to the slave replica nodes. First, the master replica node receives the data shards and immediately writes them to the local storage engine. Writing to the master replica node is typically synchronous to ensure that the data is up-to-date on the master replica. Next, the system sends replication requests to all slave replica nodes in parallel, accelerating data replication.

[0095] As you can understand, through replication, the system can synchronize and redundantly store data across different nodes, ensuring that even if a replica node fails, other replicas can still maintain data integrity. This synchronous replication mechanism ensures data consistency and avoids inconsistencies during data updates.

[0096] Step S106: Receive the storage confirmation signal from the master and replica nodes, and return a storage success response to the business system.

[0097] Replication operations are typically performed concurrently to ensure data consistency across replica nodes. Slave replica nodes immediately store data upon receipt, and the master replica node returns a storage confirmation signal confirming the success of the data storage operation. The system then returns a storage success response to the business system, indicating that the data has been securely stored.

[0098] The above-mentioned implementation method, based on refined storage strategies and dynamic adjustment mechanisms, has significant advantages in improving the availability, fault tolerance, and performance of distributed storage systems. First, by analyzing the storage request content and parsing the storage policy tags, the system can automatically adjust the storage strategy according to different data requirements to ensure that the storage method of each data unit can be optimized. Secondly, through flexible sharding strategies and replica distribution strategies, the system can efficiently manage data in a large-scale distributed environment, ensuring high data availability and fast access capabilities. Finally, through replica replication and load monitoring mechanisms, the storage system can achieve dynamic load balancing and high fault tolerance, ensuring data integrity and continuous service availability even in extreme situations.

[0099] Reference Figure 2 As an implementation method of step S104, the steps of determining the number of replicas according to the fault tolerance level, generating a replica distribution strategy based on the distributed cluster topology, and specifying a list of master replica nodes and slave replica nodes include:

[0100] Step S201, obtaining the available zone identifier of the service request source of the data unit;

[0101] Specifically, in distributed storage systems, data storage and access efficiency are often affected by the location of the request source. To ensure efficient data access within the availability zone of the request source, we first need to obtain the availability zone identifier for the business data unit's business request. This identifier typically comes from the region of the user requesting the data or the deployment location of the application service. This identifier helps the system prioritize geographic locations during subsequent replica selection and data migration, thereby optimizing data availability and access speed.

[0102] For example, assume that the business data unit of an e-commerce platform is order data, which comes from user A, and user A is located in availability zone AZ1. The system will record this availability zone identifier as the basis for subsequent replication strategy generation.

[0103] Step S202: query a preset replica quantity mapping table according to the fault tolerance level of the data unit and output a replica quantity value;

[0104] To ensure high data availability and fault tolerance, the system determines the number of replicas based on the data unit's fault tolerance level tag. Higher fault tolerance levels require more replicas. Fault tolerance levels are typically mapped to a replica count table in advance, which determines the number of replicas for each data unit. This process helps the system dynamically generate a replication strategy based on business needs, ensuring service availability in the face of node failures.

[0105] For example, assuming that the fault tolerance level of the data unit order data is "high", in the replica quantity mapping table, the number of replicas corresponding to the high fault tolerance level is 3, then the output replica quantity value N is 3.

[0106] Step S203: Based on the distributed cluster topology data and the available zone identifier of the service request source, select the candidate node that meets the first network delay threshold and has the highest health score as the primary and secondary nodes, and output the primary and secondary node identifiers;

[0107] The selection of primary and replica nodes is crucial for ensuring data access efficiency and system load balancing. Based on the cluster topology and the availability zone identifier of the service request's source, the system first selects nodes within the same zone that meet certain network latency requirements (i.e., nodes with network latency below a first network latency threshold). Then, based on a comprehensive health score (including factors such as CPU utilization and storage latency), the node with the highest score is selected as the primary and replica node. This health score calculation ensures that the selected node has high performance and stability, capable of carrying the primary and replica load.

[0108] For example, assume a service request originates in availability zone AZ1. The topology shows that there are nodes A and B in availability zone AZ1, with a network latency of 10ms for node A and 30ms for node B. The system first selects nodes with a latency of ≤ 20ms and then selects node A as the primary replica node based on its health score (assuming node A has a health score of 90 and node B has a score of 80).

[0109] Step S204: Based on the replica quantity value and the primary replica node identifier, select candidate nodes that are in different availability zones from the primary replica node and meet the second network delay threshold as secondary replica nodes, and output a list of secondary replica nodes;

[0110] The selection of slave replica nodes aims to ensure data redundancy and high availability, and they must be deployed in different availability zones to avoid data loss caused by simultaneous failures of multiple nodes in the same zone. Based on the output replica count value N, the system selects N-1 eligible slave replica nodes. The screening criteria include different availability zones and a secondary network latency threshold. This ensures that the geographic distribution of replica nodes is different from that of the primary replica node to reduce the impact of regional failures, while also ensuring that the network latency of slave replica nodes does not exceed a certain threshold, thereby ensuring efficient data access.

[0111] For example, assume the number of replicas is 3, and master node A is located in availability zone AZ1. The system selects nodes that meet the following conditions: Node B is located in availability zone AZ2 and has a network latency of 15ms, which meets the requirement. Node C is located in availability zone AZ3 and has a network latency of 25ms, which also meets the requirement. Therefore, the output list of slave nodes is ["AZ2-NodeB","AZ3-NodeC"].

[0112] As you can see, by intelligently selecting replica nodes, the system ensures geographical distribution of replicas, enhances the system's fault tolerance, and optimizes data access speed and network efficiency. Multi-AZ deployment avoids single points of failure and improves data availability.

[0113] Step S205: Bind the master replica node identifier and the slave replica node list to the data shard identifier, generate a replica distribution strategy, and store it in a distributed configuration center.

[0114] After replica node selection, the system binds the master replica node ID and the list of slave replica nodes to the data shard ID to generate a replica distribution policy file. This policy file contains specific information about the replica nodes and serves as the basis for subsequent storage system execution. Ultimately, the replica distribution policy file is stored in a distributed configuration center, such as ZooKeeper, to ensure that all nodes in the system are synchronized with the latest replica distribution policy.

[0115] In this implementation, a flexible replication strategy generation method, combined with the fault tolerance requirements of business data units and network topology, intelligently selects a list of master and slave replica nodes, ensuring high data availability, redundancy, and access efficiency. By considering network latency, health scores, and the geographic distribution of nodes, the system optimizes the load balancing and fault tolerance capabilities of the distributed storage system. The generated replica distribution strategy file ensures consistent and synchronized configuration across all nodes, thereby improving system reliability and performance.

[0116] Reference Figure 3 As an implementation method of step S203, the step of selecting a candidate node that meets the first network delay threshold and has the highest health score as the primary replica node based on the distributed cluster topology data and the availability zone identifier of the service request source includes:

[0117] Step S301: Obtain the available zone location and historical failure rate of each candidate node from the distributed cluster topology data;

[0118] In a distributed storage system, each node's availability zone location and historical failure rate are important factors influencing node selection. Availability zone location determines whether a node is physically located in the same region as other nodes, while historical failure rates reflect a node's reliability. In this step, the system extracts this information from the distributed cluster's topology data. Historical failure rates can be obtained from monitoring data, including the number of node downtimes and the frequency of network outages. Based on this information, the system can assess each node's suitability for fault-tolerant design.

[0119] Step S302: collecting load indicators of each candidate node in real time;

[0120] The node's load status directly affects its request processing capability and response speed. Load metrics typically include CPU usage, memory utilization, and disk read / write speeds. In this step, the system collects load information for each candidate node in real time to assess the node's current status. Highly loaded nodes may not be able to provide stable services and should be excluded or have their priority lowered as primary or replica nodes.

[0121] Step S303, calculating the health score of each candidate node based on the load index and historical failure rate;

[0122] The health score is a key indicator for assessing a node's current health (e.g., load and resource usage) within a distributed system. The system uses a weighted formula to calculate the health score, combining the node's load metric and historical failure rate. The load metric typically carries a larger weight, as a node's real-time performance directly impacts its ability to effectively provide services. The historical failure rate, reflecting the node's reliability, typically carries a smaller weight. Nodes with high health scores exhibit better performance and lower failure rates, making them preferred for primary and replica nodes.

[0123] In one embodiment of the present application, the health score calculation formula is configured as: Health Score = (100 - Real-time CPU Usage) × 0.7 + (100 - Historical Failure Rate × 100) × 0.3. By comprehensively considering load and historical failure rates, the system can generate a scientific health score, enabling the selected nodes to have better processing capabilities and lower failure rates, thereby improving the reliability of the primary and replica nodes.

[0124] Step S304: Based on the available zone identifier of the service request source, candidate nodes that are in the same zone as the available zone of the service request source and that meet a first network delay threshold are screened to obtain a first candidate node set.

[0125] To ensure efficient data access, the system selects nodes in the same availability zone as the source of the business request as primary and replica nodes. This significantly reduces network latency and improves data access speed. First, the system selects nodes in the same availability zone as the source of the business request. Next, it further selects nodes with network latency within a set threshold to ensure that request response times meet business requirements.

[0126] For example, assume a service request originates in availability zone AZ1, nodes A and B are both located in AZ1, and node C is located in AZ2. Assume the latency from AZ1 to node A is 10ms, node B is 30ms, and node C is 50ms. The system first selects nodes A and B in the same zone AZ1. Then, based on the latency threshold, it selects node A with a latency of ≤20ms. Therefore, node A is placed in the first candidate node set.

[0127] It can be understood that by selecting a node in the same availability zone as the source of the business request and with a network latency that meets the requirements, the system effectively reduces network latency and improves the user's data access experience.

[0128] Step S305 , sorting the first candidate node set according to the health score, and selecting the candidate node with the highest health score as the primary replica node.

[0129] Specifically, after selecting a set of candidate nodes that meet network latency requirements, the system ranks them based on their health scores. Nodes with higher health scores demonstrate better performance and are more stable in processing business requests. Ultimately, the node with the highest health score is selected as the primary and replica nodes, ensuring high availability and performance of the system.

[0130] The above implementation combines distributed cluster topology data, real-time load monitoring, historical failure rates, and network latency to comprehensively evaluate the health of each candidate node and intelligently select primary and replica nodes. By accurately screening low-latency nodes in the same region and sorting them according to health scores, the system ensures the selection of a high-performing and stable primary and replica node, maximizing data access efficiency and system fault tolerance. Ultimately, this solution implements an efficient replica distribution strategy, ensuring high availability, low latency, and high reliability for the distributed storage system.

[0131] Reference Figure 4 As an implementation of step S204, the step of selecting a candidate node in a different availability zone from the primary replica node and meeting a second network delay threshold as a secondary replica node according to the number of replicas and the primary replica node identifier includes:

[0132] Step S401: Based on the primary and secondary node identifiers, candidate nodes in different availability zones from the primary and secondary nodes are screened to obtain a second set of candidate nodes.

[0133] Specifically, in a distributed storage system, replica nodes must be selected in different availability zones from the primary replica node to avoid the risk of all replicas failing due to a single availability zone failure. Therefore, the system first selects nodes in different availability zones from the primary replica node based on the primary replica node ID, constructing a second set of candidate nodes. The goal of this step is to distribute replica nodes across multiple availability zones, improving the system's fault tolerance and data availability.

[0134] For example, assuming the primary replica node is AZ1-NodeA, the system filters out nodes in different availability zones from AZ1 based on the topology data, such as AZ2-NodeB, AZ3-NodeC, etc. The filtered candidate node set is: ["AZ2-NodeB", "AZ3-NodeC"].

[0135] Step S402, calculating the network delay between each candidate node in the second candidate node set and the primary replica node;

[0136] Network latency is a key factor affecting data access efficiency, especially in distributed systems. The lower the network latency between replica nodes and the primary replica node, the more efficient data synchronization and access. Therefore, the system calculates the network latency between each candidate node and the primary replica node. This latency calculation helps select replica nodes with lower network latency, thereby improving data access speed and overall system performance.

[0137] For example, assuming that the network delays between the primary replica node AZ1-NodeA and the candidate nodes AZ2-NodeB and AZ3-NodeC are 15ms and 30ms respectively, the system will record and calculate these delay values ​​as the basis for subsequent node screening.

[0138] Step S403, screening candidate nodes whose network delays meet a second network delay threshold and sorting them in ascending order according to the network delays;

[0139] To ensure data synchronization and access efficiency, the system sets a second network latency threshold. Only candidate nodes with network latency less than or equal to this threshold are selected as slave replica nodes. After selecting nodes that meet the latency requirement, the system sorts them in ascending order by latency, ensuring that nodes with the lowest latency are prioritized as slave replica nodes. This sorting process helps optimize node selection, minimizing overall system latency.

[0140] For example, assume the network delays of candidate nodes AZ2-NodeB and AZ3-NodeC are 15ms and 30ms, respectively. The system sets the second network delay threshold to 20ms. The selected nodes are AZ2-NodeB (delay 15ms) and AZ3-NodeC (delay 30ms). After sorting in ascending order, AZ2-NodeB will be ranked first.

[0141] Step S404: Select the first N-1 candidate nodes as slave replica nodes according to the replica number value N.

[0142] Based on the replica count value N, the system selects the first N-1 nodes with the lowest latency as slave replicas, ensuring that the number of replica nodes meets the business fault tolerance requirements. N-1 nodes are selected because the master replica node already exists, and the remaining number of replica nodes must meet the replica count requirement. This step ensures the geographic distribution of replica nodes, latency requirements, and fault tolerance, ensuring data redundancy and availability.

[0143] For example, assume there are three replicas (i.e., N=3). After latency filtering and sorting, the latency of node AZ2-NodeB is 15ms, and the latency of node AZ3-NodeC is 30ms. Since there are three replicas, the system needs to select two slave replica nodes. In this example, the system selects nodes AZ2-NodeB and AZ3-NodeC as slave replica nodes.

[0144] It can be understood that by selecting the optimal slave replica node based on the replica quantity value, the system ensures that the quantity and quality of replicas meet the high availability requirements of the business and ensures that the latency of replica distribution is as low as possible, thereby improving the stability and performance of the system.

[0145] The above implementation precisely selects nodes in different availability zones from the primary replica node and optimizes the selection of secondary replica nodes based on network latency and the required number of replicas. First, secondary replica nodes are ensured to be distributed across different availability zones, thereby improving the system's fault tolerance. Second, by calculating and ranking the network latency between nodes, the node with the lowest latency is selected as the secondary replica node, thereby improving data access speed and synchronization efficiency.

[0146] Reference Figure 5 As an implementation method of step S105, the steps of writing the data shards to the primary replica node and synchronously initiating the replica copy operation to the secondary replica node according to the replica distribution strategy include:

[0147] Step S501: parse the replica distribution strategy, extract the master replica node identifier, slave replica node list, and synchronization protocol type;

[0148] In a distributed storage system, the replica distribution strategy file contains key information about data distribution, node selection, and synchronization rules. This step parses the replica distribution strategy file to extract the master replica node identifier, the slave replica node list, and the synchronization protocol type. The master replica node is responsible for receiving write requests and recording data, while the slave replica nodes are responsible for data replication and redundancy. The synchronization protocol type defines the data synchronization method between replica nodes, such as strong synchronization or weak synchronization.

[0149] For example, by parsing the file, the system extracts that the master replica node identifier is AZ1-NodeA, the slave replica node list is ["AZ2-NodeB", "AZ3-NodeC"], and the synchronization protocol type is "strong".

[0150] Step S502: Send a data shard write request to the primary replica node, and receive a write timestamp and data check code returned by the primary replica node;

[0151] When a business request requires writing data to a distributed storage system, the system first sends the data shards to the primary replica node. The primary replica node writes the data to the local storage engine and generates a data checksum (such as a hash value) to verify data integrity. It also records the physical timestamp of the write. The primary replica node then returns a write timestamp and data checksum to confirm the successful data write. This information is crucial for subsequent data replication and consistency verification.

[0152] Step S503: Synchronously initiate a data replication request from the replica node list; wherein the data replication request includes the data shard content, data checksum, and write timestamp;

[0153] To ensure high data availability and redundancy, the master replica node replicates data to slave replica nodes. The system then sends data replication requests to all slave replica nodes based on the slave node list in the replica distribution strategy. The data replication request includes the data shard content, write timestamp, and data checksum to ensure the consistency and correctness of the data replicas. This step ensures that after the data is written to the master replica node, the replica nodes synchronously receive and store the same data.

[0154] For example, assuming that the data verification code returned by the master replica node is "abc123" and the timestamp is "2025-06-04T10:00:00Z", the system will initiate a replication request to AZ2-NodeB and AZ3-NodeC. The request includes: data shard content, verification code "abc123" and timestamp "2025-06-04T10:00:00Z".

[0155] Step S504: Collect the response status of each slave replica node and determine the data consistency result based on the synchronization protocol type and the number of successfully responding replicas;

[0156] After a data replication request is sent to a slave replica node, the system waits for a response from the replica node and collects the response status of each slave replica node. If the response is successful, it means that the replica node has successfully received and stored the data; if the response fails, it means that a problem occurred during the data synchronization process. Depending on the type of synchronization protocol, the system determines the data consistency result based on the number of nodes that successfully responded. In strong synchronization mode, the system requires a majority of replica nodes to successfully respond in order to consider the data consistency verification passed; in weak synchronization mode, the system may allow a small number of replica nodes to fail.

[0157] Specifically, the steps of determining the data consistency result based on the synchronization protocol type and the number of successful response replicas include: if the synchronization protocol type is strong synchronization mode, counting the number of successful response nodes; when the number of successful response nodes exceeds one-half of the total number of replica nodes, marking the data consistency result as submitted, and generating a status mapping table containing the failed node identifier.

[0158] For example, assume the synchronization protocol type is "strong," and replica nodes AZ2-NodeB and AZ3-NodeC respond successfully and fail, respectively. The system counts the number of successful response nodes and finds only one. Because the number of successful response nodes is insufficient, the system determines that data consistency verification has failed.

[0159] Step S505 : executing a replica data retry replication or triggering a replica distribution strategy update according to the failed node list in the data consistency result.

[0160] If some slave replica nodes fail to synchronize data successfully, the system will record these failed nodes in the failed node list. Based on the consistency result, the system will decide whether to perform data replication retry or trigger the replica distribution strategy update. Data retry uses the exponential backoff algorithm to calculate the retry delay time: Delay time = basic delay × (2 重试次数 ) to avoid excessive system load from multiple retries in a short period of time. If data synchronization fails after multiple retries, the system will call the replica distribution strategy generation module to update the replica distribution strategy, replace the failed slave replica node, and broadcast the new replica distribution strategy to all nodes.

[0161] For example, assuming that the replica node AZ3-NodeC still fails to successfully receive data after multiple retries, the system will call the replica distribution strategy generation module, select the new slave replica node AZ4-NodeD to replace AZ3-NodeC, update the replica distribution strategy and broadcast it to all nodes.

[0162] In the above implementation, the replica distribution strategy file is parsed, and the system obtains the node information and synchronization protocol type of the write request. Then, the data is written to the master replica node and synchronized to the slave replica node to ensure redundant storage and high availability of the data. During the data synchronization process, the system determines the consistency result based on the response status and handles it according to different synchronization protocol types. For failed nodes, the system adopts a retry mechanism and updates the replica distribution strategy when necessary to deal with replica node failures or network problems. The overall solution ensures the stability, reliability and data consistency of the distributed storage system through efficient data writing, replication and consistency verification mechanisms.

[0163] Reference Figure 6 As a further implementation method of the efficient distributed storage method, according to the replica distribution strategy, after writing the data shards to the primary replica node and synchronously initiating the replica copy operation to the secondary replica node, the method further includes:

[0164] Step S601: collecting load indicators of the master replica node and the slave replica node in real time;

[0165] Load metrics include CPU utilization, network I / O throughput, and storage latency. In distributed storage systems, load monitoring is a critical component for ensuring efficient system operation. By collecting load metrics from the master and all slave nodes in real time, the system can fully understand the current operating status of each node, enabling timely response and optimization decisions.

[0166] Specifically, CPU utilization reflects the utilization of a node's processing power. Excessively high CPU utilization indicates a potential computational bottleneck, impacting data processing and response speeds. Network I / O throughput indicates the speed at which data is transmitted between a node and other nodes, directly impacting the efficiency of data synchronization and access. A high-throughput network can respond to data requests more quickly. Storage latency refers to the time required for a node to read or write data. High storage latency typically indicates an overloaded node or insufficient storage performance, potentially leading to longer system response times and impacting data access speeds.

[0167] Step S602, calculating a load score based on a pre-built load assessment model;

[0168] The load score is calculated based on a pre-set load assessment model, which uses collected load metrics to comprehensively score each node. The scoring formula takes into account several key factors: lower CPU utilization indicates a healthier node, higher throughput enables faster response to data requests, and lower storage latency indicates a more responsive node.

[0169] Specifically, the load assessment model can calculate the load score using a pre-built weighted formula. The specific formula is:

[0170] Score=(0.5×(100-cpu_usage))+(0.3×io_throughput / 300)+(0.2×(50-storage_latency));

[0171] In the above formula, Score is the load score, cpu_usage is the CPU usage, io_throughput is the network IO throughput, and storage_latency is the storage latency.

[0172] Step S603: Determine whether the load score of the primary and replica nodes is lower than the score threshold for a preset number of consecutive times; if so, jump to step S604; if not, do not perform any operation;

[0173] Specifically, in a distributed storage system, the load status of the master replica node is crucial to the performance of the entire system. If the load score of the master replica node is lower than the set threshold for multiple consecutive times, it may indicate that the node is overloaded and can no longer provide services efficiently. Therefore, the system will check the load score of the master replica node according to the set number of consecutive times. If the consecutive detection results are all lower than the threshold, the master replica migration process will be triggered. Setting the threshold to "80 points" means that nodes with higher loads will be considered to be at high risk, while points below 60 indicate that migration is needed as soon as possible.

[0174] For example, assuming that the load scores of the master-replica node A are 50, 55, and 58, and the threshold is set to 60 points, then the load score of node A is lower than 60 points in three consecutive monitorings, which will trigger the master-replica migration process.

[0175] Step S604: Select a slave replica node with the highest load score and that meets the topology constraints from the slave replica node list as the new master replica node;

[0176] Once the load score of a primary replica node is determined to be below a threshold for a preset number of times, the system selects a new primary replica node based on the current replica group's load score and topological constraints. This selection prioritizes the secondary replica node with the highest load score, ensuring that the new primary replica node has a low load and is capable of assuming primary replica responsibilities. Furthermore, topological constraints, such as network latency requirements and availability zone constraints, must be met to ensure that the new primary replica node provides good access performance and system stability.

[0177] For example, assuming there are two slave replica nodes B and C in the replica group, the load score of node B is 85, the score of node C is 95, and the delay between node C and the business source availability zone is less than 25ms, the system selects node C as the new master replica node.

[0178] As you can see, by dynamically selecting the slave replica node with the least load and that meets the topology constraints, the system can balance the load and ensure that system performance is not affected after the master replica migration. In addition, this approach can improve the system's load balancing capabilities and prevent a single node from becoming a bottleneck.

[0179] Step S605: Incrementally synchronize the data shards from the original master replica node to the new master replica node, and update the replica distribution strategy after the synchronization is completed.

[0180] The data migration process must be incremental, meaning only the data that has changed since the last synchronization is synchronized, rather than the entire dataset. This significantly reduces data migration time and bandwidth consumption, improving migration efficiency. After synchronization is complete, the system updates the replica distribution topology, clarifies the location of the new master replica node, and notifies all replica nodes of the new storage policy.

[0181] For example, suppose that during the migration of order data from the original master replica node A, its incremental data only contains newly added order information. The original master replica node A synchronizes this newly added data to the new master replica node C. Upon completion, the replica distribution strategy is updated. The new replica topology will include node C as the master replica node, and other nodes in the system will be notified to synchronize with the new replication strategy.

[0182] As you can see, the incremental synchronization mechanism ensures efficient migration and avoids the resource waste associated with a full data migration. By promptly updating the replica distribution strategy, we ensure system consistency and data integrity, minimizing service interruptions or performance degradation caused by primary-replica migrations.

[0183] In this implementation, by collecting and evaluating the load of each node in real time, the system can automatically migrate the primary and replica nodes when overloaded, ensuring high system availability and avoiding the impact of single points of failure on services. Incremental synchronization and topology update strategies ensure data consistency and efficient migration.

[0184] The embodiments of the present application also disclose an efficient distributed storage system.

[0185] An efficient distributed storage system, the storage system comprising:

[0186] A receiving module, used to receive data storage requests from business systems;

[0187] A parsing module, configured to parse data storage requests and output data units containing storage policy tags; the storage policy tags include data sharding granularity and fault tolerance levels;

[0188] The data splitting module is used to split the data unit into data fragments of corresponding sizes according to the data fragmentation granularity level;

[0189] The replica distribution strategy generation module is used to determine the number of replicas according to the fault tolerance level, generate a replica distribution strategy based on the distributed cluster topology, and specify the master replica node and slave replica node list;

[0190] The data storage module is used to write data shards into the master replica node according to the replica distribution strategy, and simultaneously initiate the replica replication operation to the slave replica node;

[0191] The storage success response module is used to receive the storage confirmation signal from the master and replica nodes and return the storage success response to the business system.

[0192] As a further embodiment of the efficient distributed storage system, the storage system further includes:

[0193] Load indicator collection module, used to collect load indicators of master and slave nodes in real time;

[0194] A load score calculation module, used to calculate a load score based on a pre-built load assessment model;

[0195] A judgment module is used to judge whether the load score of the master and replica nodes is lower than the score threshold for a preset number of consecutive times; if so, output a first judgment result;

[0196] a node screening module, configured to select, in response to the first judgment result, a slave replica node with the highest load score and satisfying the topology constraint from the slave replica node list as a new master replica node;

[0197] The data incremental synchronization module is used to incrementally synchronize data shards from the original master replica node to the new master replica node;

[0198] The strategy update module is used to update the replica distribution strategy after synchronization is completed.

[0199] The efficient distributed storage system of the embodiment of the present application can implement any of the above-mentioned storage methods, and the specific working process of each module in the storage system can refer to the corresponding process in the above-mentioned method embodiment.

[0200] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for example, the division of a module is merely a logical functional division, and in actual implementation, other division methods may be used, such as combining or integrating multiple modules into another system, or ignoring or not implementing certain features.

[0201] The embodiment of the present application also discloses a computer device.

[0202] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the efficient distributed storage method described above is implemented.

[0203] The embodiment of the present application also discloses a computer-readable storage medium.

[0204] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the above-mentioned efficient distributed storage methods.

[0205] Among them, computer-readable storage media can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0206] It should be noted that, in the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0207] The above are all preferred embodiments of the present application and are not intended to limit the scope of protection of this application. Unless otherwise specified, any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features. In other words, unless otherwise specified, each feature is merely an example of a series of equivalent or similar features.

Claims

1. An efficient distributed storage method, characterized in that: The storage method includes: Receive data storage requests from business systems; Parsing the data storage request and outputting a data unit including a storage policy tag; the storage policy tag includes a data sharding granularity level and a fault tolerance level; Splitting the data unit into data fragments of corresponding sizes according to the data fragmentation granularity level; Determine the number of replicas according to the fault tolerance level, generate a replica distribution strategy based on the distributed cluster topology, and specify a list of master replica nodes and slave replica nodes; According to the replica distribution strategy, the data shards are written to the master replica node, and a replica replication operation is synchronously initiated to the slave replica node; The steps of receiving a storage confirmation signal from the master replica node and returning a storage success response to the business system; determining the number of replicas according to the fault tolerance level, generating a replica distribution strategy based on the distributed cluster topology, and specifying a list of master replica nodes and slave replica nodes include: Obtain the availability zone identifier of the data unit's business request source; the availability zone identifier of the business request source is derived from the user region requesting the data or the deployment location of the application service; Querying a preset replica quantity mapping table according to the fault tolerance level of the data unit, and outputting a replica quantity value; Based on the distributed cluster topology data and the availability zone identifier of the business request source, select the candidate node in the same zone that meets the first network latency threshold and has the highest health score as the primary and replica nodes, and output the primary and replica node identifiers; According to the replica quantity value and the master replica node identifier, select candidate nodes in different availability zones from the master replica node and meeting a second network delay threshold as slave replica nodes, and output a list of slave replica nodes; Binding the master replica node identifier and the slave replica node list to the data shard identifier, generating a replica distribution strategy and storing it in a distributed configuration center; and selecting a candidate node that meets a first network latency threshold and has the highest health score as the master replica node based on the distributed cluster topology data and the availability zone identifier of the service request source, including: Obtain the availability zone location and historical failure rate of each candidate node from the distributed cluster topology data; Collect load indicators of each candidate node in real time; load indicators include CPU usage, memory usage, and disk read and write speed; Calculate a health score for each candidate node based on the load metric and historical failure rate; Based on the available zone identifier of the service request source, candidate nodes that are in the same zone as the available zone of the service request source and that meet a first network delay threshold are screened to obtain a first candidate node set; The steps of sorting the first candidate node set according to the health score and selecting the candidate node with the highest health score as the primary replica node; and screening the candidate node in a different availability zone from the primary replica node and meeting a second network delay threshold as the secondary replica node according to the replica quantity value and the primary replica node identifier include: Based on the primary and replica node identifiers, candidate nodes in different availability zones from the primary and replica nodes are screened to obtain a second candidate node set; Calculating the network delay between each candidate node in the second candidate node set and the primary replica node; Screening candidate nodes whose network delays meet a second network delay threshold and sorting them in ascending order according to the network delays; According to the replica number value N, the first N-1 candidate nodes are selected as slave replica nodes.

2. The efficient distributed storage method according to claim 1, characterized in that: According to the replica distribution strategy, the steps of writing the data shards to the master replica node and synchronously initiating a replica copy operation to the slave replica node include: Parse the replica distribution strategy, extract the master replica node identifier, slave replica node list and synchronization protocol type; Send a data shard write request to the master replica node, and receive a write timestamp and data check code returned by the master replica node; Synchronously initiating a data replication request from the replica node list; wherein the data replication request includes the data shard content, the data check code, and the write timestamp; Collect the response status of each slave replica node and determine the data consistency result based on the synchronization protocol type and the number of successfully responding replicas; According to the failed node list in the data consistency result, the replica data is retried for replication or the replica distribution strategy is triggered to update.

3. An efficient distributed storage method according to any one of claims 1 to 2, characterized in that: After writing the data shards to the master replica node and synchronously initiating a replica copy operation to the slave replica node according to the replica distribution strategy, the method further includes: Collecting load indicators of the master replica node and the slave replica node in real time; Calculate load scores based on pre-built load assessment models; Determine whether the load score of the master and replica nodes is lower than a score threshold for a consecutive preset number of times; If so, select the slave replica node with the highest load score and meeting the topology constraints from the list of slave replica nodes as the new master replica node; Incrementally synchronize the data shards from the original master replica node to the new master replica node, and update the replica distribution strategy after the synchronization is completed.

4. An efficient distributed storage system, characterized in that: For executing an efficient distributed storage method according to any one of claims 1 to 3, the storage system comprises: A receiving module, used to receive data storage requests from business systems; A parsing module, configured to parse the data storage request and output a data unit including a storage policy tag; the storage policy tag includes a data sharding granularity level and a fault tolerance level; A data splitting module, configured to split the data unit into data fragments of corresponding sizes according to the data fragmentation granularity level; A replica distribution strategy generation module is used to determine the number of replicas according to the fault tolerance level, generate a replica distribution strategy based on the distributed cluster topology, and specify a list of master replica nodes and slave replica nodes; A data storage module, configured to write the data shards into the master replica node and synchronously initiate a replica replication operation to the slave replica node according to the replica distribution strategy; The storage success response module is used to receive the storage confirmation signal of the master and replica nodes and return a storage success response to the business system.

5. An efficient distributed storage system according to claim 4, characterized in that: The storage system further includes: A load index collection module is used to collect the load index of the master replica node and the slave replica node in real time; A load score calculation module, used to calculate a load score based on a pre-built load assessment model; a judgment module, configured to judge whether the load score of the master and replica nodes is lower than the score threshold for a consecutive preset number of times; if so, output a first judgment result; a node screening module, configured to select, in response to the first judgment result, a slave replica node with the highest load score and satisfying the topology constraint from the slave replica node list as a new master replica node; The data incremental synchronization module is used to incrementally synchronize data shards from the original master replica node to the new master replica node; The strategy update module is used to update the replica distribution strategy after synchronization is completed.

6. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 3 when executing the program.

7. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 3.

Images