Efficient distributed storage method, system, equipment and medium

Abstract

The invention relates to an efficient distributed storage method, system and device and a medium, and belongs to the technical field of distributed storage. The storage method comprises the steps that a data storage request of a service system is received; analyzing the data storage request, and outputting a data unit containing a storage strategy label; the storage strategy label comprises a data fragment granularity level and a fault-tolerant level; splitting the data unit into data fragments with corresponding sizes according to the granularity level of the data fragments; determining the number of copies according to the fault-tolerant level, generating a copy distribution strategy based on a distributed cluster topological structure, and appointing a master copy node list and a slave copy node list; according to a copy distribution strategy, data fragments are written into a master copy node, and copy replication operation is synchronously initiated to slave copy nodes; and receiving a storage confirmation signal of the main replica node, and returning a storage success response to the service system. The data storage reliability of the business system can be improved.

Description

Technical Field

[0001] This application relates to the field of distributed storage technologies, and in particular, to an efficient distributed storage method, system, device, and medium. Background Art

[0002] In recent years, distributed storage systems have been widely used in various large-scale data management applications. Especially in the fields of cloud computing and big data processing, with the rapid growth of data volume, how to achieve effective storage, fast access, and high-availability management of large-scale data has become a key issue in technological development. The design of a distributed storage system usually involves how to distribute a large amount of data across multiple physical storage nodes while ensuring redundant backup, fault tolerance, and load balancing of the data in the system. These problems mainly stem from the complex balance among data storage reliability, access performance, and fault recovery capabilities.

[0003] In traditional storage systems, data is usually stored on a single node or a few nodes. This storage method often leads to bottleneck phenomena when facing large-scale data and frequent access. For example, when a storage node fails, it may cause data loss or service interruption. In addition, the read / write speed and load capacity of a single storage node are limited and cannot meet the requirements in large-scale, high-concurrency access scenarios. To solve these problems, distributed storage systems emerged. By distributing data across multiple nodes and creating multiple copies of each piece of data, they provide stronger fault tolerance and scalability.

[0004] However, current distributed storage systems do not fully consider the differences in different data types and business requirements and often lack the ability to dynamically adjust during actual application processes, resulting in the inability to fully exert the storage efficiency, fault tolerance, and access performance of the system, thus affecting the data storage reliability of business systems. Summary of the Invention[[ID=X]]

[0005] To improve the data storage reliability of business systems, this application provides an efficient distributed storage method, system, device, and medium.

[0006] In a first aspect, this application provides an efficient distributed storage method, adopting the following technical solution: An efficient distributed storage method, the storage method comprising: Receiving a data storage request from a business system; Parsing the data storage request and outputting a data unit containing a storage policy tag; the storage policy tag includes a data sharding granularity level and a fault tolerance level; According to the data sharding granularity level, splitting the data unit into data shards of corresponding sizes; Determine the number of replicas according to the fault tolerance level, generate a replica distribution strategy based on the distributed cluster topology structure, and specify the list of primary replica nodes and secondary replica nodes; According to the replica distribution strategy, write the data shard into the primary replica node and synchronously initiate a replica replication operation to the secondary replica nodes; Receive the storage confirmation signal from the primary replica node and return a storage success response to the service system.

[0007] By adopting the above technical solution, based on the refined storage strategy and dynamic adjustment mechanism, it has significant advantages in improving the availability, fault tolerance and performance of the distributed storage system. First, by analyzing the content of the storage request and parsing out the storage strategy tag, the system can automatically adjust the storage strategy according to different data requirements to ensure that the storage method of each data unit can be optimized. Second, through the flexible sharding strategy and replica distribution strategy, the system can efficiently manage data in a large-scale distributed environment, ensuring the high availability and fast access ability of the data. Finally, through the replica replication and load monitoring mechanism, the storage system can achieve dynamic load balancing and high fault tolerance, ensuring the integrity of the data and the continuous availability of the service in extreme cases.

[0008] Optionally, the steps of determining the number of replicas according to the fault tolerance level, generating a replica distribution strategy based on the distributed cluster topology structure, and specifying the list of primary replica nodes and secondary replica nodes include: Obtain the available zone identifier of the service request source of the data unit; Query the preset replica number mapping table according to the fault tolerance level of the data unit and output the replica number value; Based on the distributed cluster topology structure data and the available zone identifier of the service request source, filter the candidate nodes that meet the first network delay threshold and have the highest health score as the primary replica node and output the primary replica node identifier; According to the replica number value and the primary replica node identifier, filter the candidate nodes in different available zones from the primary replica node and meeting the second network delay threshold as the secondary replica nodes and output the list of secondary replica nodes; Bind the primary replica node identifier and the list of secondary replica nodes to the data shard identifier, generate a replica distribution strategy and store it in the distributed configuration center.

[0009] By adopting the above technical solution, based on the flexible replica strategy generation method, combined with the fault tolerance requirements of business data units and the network topology, the master replica node and the list of slave replica nodes are intelligently selected, ensuring high availability, redundancy, and access efficiency of the data. By considering network latency, health scores, and the geographical distribution of nodes, the system optimizes the load balancing and fault tolerance of the distributed storage system. The generated replica distribution strategy file ensures the consistency and synchronization of each node's configuration, thereby improving the reliability and performance of the system.

[0010] Optionally, the step of screening candidate nodes that meet the first network latency threshold and have the highest health score as the master replica node based on the distributed cluster topology data and the available zone identifier of the business request source includes: Obtain the available zone location and historical failure rate of each candidate node from the distributed cluster topology data; Real-time collect the load metrics of each candidate node; Calculate the health score of each candidate node based on the load metrics and historical failure rate; Based on the available zone identifier of the business request source, screen candidate nodes in the same zone as the business request source and meeting the first network latency threshold to obtain a first candidate node set; Sort the first candidate node set according to the health score, and select the candidate node with the highest health score as the master replica node.

[0011] By adopting the above technical solution, combined with the distributed cluster topology data, real-time load monitoring, historical failure rate, and network latency, comprehensively evaluate the health status of each candidate node, and intelligently select the master replica node. By accurately screening nodes with low latency in the same zone and sorting according to the health score, the system can ensure the selection of a master replica node with superior performance and stability, maximizing the data access efficiency and the fault tolerance of the system. Finally, this solution realizes an efficient replica distribution strategy, ensuring the high availability, low latency, and high reliability of the distributed storage system.

[0012] Optionally, the step of screening candidate nodes in different available zones from the master replica node and meeting the second network latency threshold as the slave replica nodes according to the replica quantity value and the master replica node identifier includes: Based on the master replica node identifier, screen candidate nodes in different available zones from the master replica node to obtain a second candidate node set; Calculate the network latency between each candidate node in the second candidate node set and the master replica node; Screen candidate nodes with network latency meeting the second network latency threshold and sort them in ascending order according to the network latency; According to the replica quantity value N, select the first N - 1 candidate nodes as the slave replica nodes.

[0013] By adopting the above technical solution, nodes in different availability zones from the primary replica node are accurately screened, and based on the requirements of network latency and the number of replicas, the selection process of the secondary replica nodes is optimized. First, it is ensured that the secondary replica nodes are distributed in different availability zones, thereby improving the fault tolerance of the system; second, by calculating the network latency between nodes and sorting them, it is ensured that the nodes with the lowest latency are selected as secondary replica nodes, thereby improving data access speed and synchronization efficiency.

[0014] Optionally, according to the replica distribution strategy, the steps of writing the data shard into the primary replica node and synchronously initiating a replica replication operation to the secondary replica nodes include: Analyze the replica distribution strategy, and extract the primary replica node identifier, the list of secondary replica nodes, and the synchronization protocol type; Send a data shard writing request to the primary replica node, and receive the writing timestamp and data checksum returned by the primary replica node; Synchronously initiate a data replication request according to the list of secondary replica nodes; wherein, the data replication request includes the data shard content, the data checksum, and the writing timestamp; Collect the response status of each secondary replica node, and determine the data consistency result according to the synchronization protocol type and the number of successfully responded replicas; According to the list of failed nodes in the data consistency result, perform retry replication of replica data or trigger an update of the replica distribution strategy.

[0015] By adopting the above technical solution, by analyzing the replica distribution strategy file, the system obtains the node information and synchronization protocol type of the write request. Then, the data is written through the primary replica node and synchronized to the secondary replica nodes to ensure redundant storage and high availability of the data. During the data synchronization process, the system judges the consistency result through the response status and processes it according to different synchronization protocol types. For failed nodes, the system adopts a retry mechanism and updates the replica distribution strategy when necessary to cope with replica node failures or network problems. The overall solution ensures the stability, reliability, and data consistency of the distributed storage system through an efficient data writing, replication, and consistency verification mechanism.

[0016] Optionally, after the steps of writing the data shard into the primary replica node and synchronously initiating a replica replication operation to the secondary replica nodes according to the replica distribution strategy, the following steps are further included: Real-time collect the load metrics of the primary replica node and the secondary replica nodes; Calculate the load score based on a pre-constructed load evaluation model; Judge whether the load score of the primary replica node is lower than the score threshold for a preset number of consecutive times; If so, select the slave replica node with the highest load score and meeting the topology constraints from the list of slave replica nodes as the new master replica node; Incrementally synchronize the data shards from the original master replica node to the new master replica node, and update the replica distribution policy after the synchronization is completed.

[0017] By adopting the above technical solutions, the load balancing, fault tolerance and overall performance of the distributed storage system can be effectively improved. By collecting and evaluating the load conditions of each node in real time, the system can automatically migrate the master replica node when the load is too heavy, ensuring high availability of the system and avoiding the impact on services caused by single point of failure. The incremental synchronization and topology update strategies ensure data consistency and the efficiency of the migration process.

[0018] In a second aspect, the present application provides an efficient distributed storage system, adopting the following technical solutions: An efficient distributed storage system, the storage system includes: A receiving module, configured to receive a data storage request from a service system; A parsing module, configured to parse the data storage request and output a data unit including a storage policy label; the storage policy label includes a data shard granularity level and a fault tolerance level; A data splitting module, configured to split the data unit into data shards of corresponding sizes according to the data shard granularity level; A replica distribution policy generation module, configured to determine the number of replicas according to the fault tolerance level, generate a replica distribution policy based on the distributed cluster topology structure, and specify a list of master replica nodes and slave replica nodes; A data storage module, configured to write the data shards into the master replica node according to the replica distribution policy, and synchronously initiate a replica replication operation to the slave replica nodes; A storage success response module, configured to receive a storage confirmation signal from the master replica node and return a storage success response to the service system.

[0019] Optionally, the storage system further includes: A load metric collection module, configured to collect the load metrics of the master replica node and the slave replica nodes in real time; A load score calculation module, configured to calculate a load score based on a pre-constructed load evaluation model; A judgment module, configured to judge whether the load score of the master replica node is lower than a score threshold for a preset number of consecutive times; if so, output a first judgment result; A node screening module, configured to, in response to the first judgment result, select the slave replica node with the highest load score and meeting the topology constraints from the list of slave replica nodes as the new master replica node; The data incremental synchronization module is used to incrementally synchronize data shards from the original master replica node to the new master replica node; The strategy update module is used to update the replica distribution strategy after synchronization is completed.

[0020] In a third aspect, the present application provides a computer device that adopts the following technical solution: A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a computer program capable of being loaded by a processor and executing any one of the methods in the first aspect.

[0022] In summary, the present application includes at least one of the following beneficial technical effects: by combining the parsing of data storage requests, data sharding, determination of the number of replicas based on the fault tolerance level, and generation of a replica distribution strategy for the distributed cluster topology, the efficiency and reliability of distributed storage are optimized. Through a reasonable master and slave replica node selection and synchronization mechanism, high data availability and fast access are ensured, while the system's load balancing capabilities are improved. The present application can automatically adjust storage strategies to suit different business needs, enhance the flexibility and fault tolerance of the storage system, effectively reduce the risk of data loss, and improve the stability and efficiency of storage operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a first flow chart of an efficient distributed storage method according to one of the embodiments of the present application.

[0024] Figure 2 This is a second flow chart of an efficient distributed storage method according to one of the embodiments of the present application.

[0025] Figure 3 This is a third flow chart of an efficient distributed storage method according to one of the embodiments of the present application.

[0026] Figure 4 This is a fourth flow chart of an efficient distributed storage method according to one of the embodiments of the present application.

[0027] Figure 5 This is the fifth flow chart of an efficient distributed storage method in one of the embodiments of the present application.

[0028] Figure 6 This is the sixth flow chart of an efficient distributed storage method according to one of the embodiments of the present application. Detailed implementation manners

[0029] To make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying Figure 1-6 drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] An embodiment of the present application discloses an efficient distributed storage method.

[0031] Referring to Figure 1 , an efficient distributed storage method, the storage method includes: Step S101, receiving a data storage request from a service system; Among them, in a distributed storage system, the system needs to store the data generated by the service system. The data storage request usually contains specific service requirement information, which may include the type of data (such as text, picture, video, etc.) and the criticality level and storage requirements of the data. The receiving operation of the storage system is essentially to access a request interface, which will process the externally input storage instructions and data content. This is the most basic step in the entire storage process, ensuring that the data can enter the storage process and preparing for subsequent storage processing.

[0032] Exemplarily, if it is an online e-commerce platform, the request received in this step may involve storing order data, customer data or transaction logs. The service system may also specify the importance level of certain data, such as "ordinary order data" or "high-priority financial data". Such requests may come from different modules, such as the order management module, the user management module, etc. After the system receives these requests, the subsequent processing flow will depend on the parsing and understanding of the request content to ensure that the data is correctly allocated, stored and managed.

[0033] Step S102, parsing the data storage request and outputting a data unit containing a storage policy label; wherein, the storage policy label includes a data sharding granularity level and a fault tolerance level; Specifically, after receiving the data storage request, the storage system needs to parse the request content. The key to this step lies in how to judge the data storage policy according to the specific content in the request, and define the basic attributes of data storage according to different data types and critical levels, and output a data unit with a storage policy label. This process includes two important operations: First, parse the "data type" and "criticality level" in the data request, and then generate a storage policy label according to the parsing result. The main content of the label includes the sharding granularity and fault tolerance level of the data.

[0034] Among them, the data sharding granularity level refers to the granularity size when data is split into several independent parts. Usually, there are two options: larger granularity and smaller granularity. Larger granularity sharding means that each data unit is split into relatively larger chunks, which is suitable for scenarios that require overall operations; while smaller granularity sharding is suitable for scenarios with frequent access and modification because it can improve the parallel processing ability and flexibility of the storage system. The fault tolerance level refers to the redundancy requirement of data. Usually, the higher the fault tolerance level, the more copies of the data. For example, for important data, the storage system may require at least 3 copies to ensure that the data can still be recovered from other copies when one copy is lost.

[0035] Specifically, when parsing a storage request for order data, the storage system may find that the data type of the request is "order data" and its importance is relatively high. Therefore, the system may generate a label of "high fault tolerance" and "medium granularity". This means that the order data will be split into multiple shards and three copies will be stored to ensure high availability of the data.

[0036] Step S103: Split the data unit into data shards of corresponding sizes according to the data sharding granularity level. Among them, after receiving and parsing the data storage request, the system starts to split the data unit according to the storage policy label. Sharding is a common operation in distributed storage. By splitting a large data unit into several smaller shards, it is convenient for storage, management, and access. The granularity of sharding is determined according to the sharding granularity level defined in the previous step. For larger data units, the storage system will determine the splitting method according to the granularity level. For scenarios that require high concurrent access, a smaller sharding granularity may be selected to ensure that each small shard can be operated in parallel on multiple nodes, improving the throughput and access speed of the storage system. For some data with less access and a single data type, a larger granularity may be used for sharding to reduce the management overhead caused by sharding.

[0037] For example, when the storage system receives an order data request and defines it as "medium granularity" according to business requirements, the system may split each order into multiple data blocks, such as order headers, order items, payment information, etc., and store them as different shards respectively. Each shard can be accessed independently, thus improving the flexibility of storage and access. In addition, the design of sharding can also be optimized based on the usage scenario of the data. For example, shards with frequent access may be assigned to storage nodes with stronger performance.

[0038] It is understandable that this sharding method can not only improve the concurrent access ability, but also optimize the use of disk space, improve data retrieval efficiency and storage resource utilization in large-scale data storage. Through fine-grained sharding, the system can flexibly control data distribution and access, avoiding performance bottlenecks that may be brought by storing a single large data block.

[0039] Step S104: Determine the number of replicas according to the fault tolerance level, generate a replica distribution strategy based on the distributed cluster topology, and specify the list of primary replica nodes and secondary replica nodes. Among them, the storage system determines the number of replicas according to the fault tolerance level of the data, and generates a replica distribution strategy according to the topology of the distributed cluster. The determination of the number of replicas usually depends on the requirements of the fault tolerance level. For example, in the case of a high fault tolerance level, multiple replicas may need to be created. The replica distribution strategy takes into account the node topology of the storage cluster, the geographical locations of nodes, and network latency, ensures uniform distribution of replicas, and maximally reduces the risk of single point of failure.

[0040] Specifically, the generation of the replica distribution strategy is a complex process. The system not only needs to ensure that the replicas can meet the redundancy requirements, but also needs to select appropriate nodes to store the primary replica and secondary replicas according to the cluster topology. The primary replica is usually stored on nodes with high access frequency or low latency, while the secondary replicas can be distributed on other nodes, usually selecting other nodes with reasonable latency from the node where the primary replica is located, to ensure that even if a replica node fails, other replicas can guarantee the integrity and availability of the data.

[0041] For example, if the fault tolerance requirement for the order data of an e-commerce platform is "high availability", the storage system may create 3 replicas for each data shard. One replica will be stored on the primary replica node, while the other two replicas will select appropriate secondary replica nodes according to the cluster topology to ensure redundancy and low latency. Through a reasonable replica distribution strategy, the high availability, data redundancy and fault tolerance of the system are guaranteed. The optimization of replica distribution not only ensures data consistency, but also improves the reliability and fault recovery ability of the system.

[0042] Step S105: According to the replica distribution strategy, write the data shard into the primary replica node, and synchronously initiate a replica replication operation to the secondary replica nodes. Among them, the system writes the data shard into the primary replica node according to the replica distribution strategy, and synchronously sends a data replication request to the secondary replica nodes. First, the primary replica node will receive the data shard and immediately write it into the local storage engine. The operation of writing into the primary replica node is usually synchronous to ensure that the data is up-to-date on the primary replica. Next, the system will send replication requests to all secondary replica nodes in parallel, which can accelerate the synchronization speed of data replicas.

[0043] It is understandable that through replica replication, the system can synchronize and redundantly store data between different nodes, ensuring that even if a replica node fails, other replicas can still guarantee data non-loss. The synchronous replication mechanism ensures data consistency and avoids inconsistent problems during data updates.

[0044] Step S106, receive the storage confirmation signal from the primary replica node, and return a storage success response to the business system.

[0045] Among them, replica replication operations are usually executed concurrently to ensure data consistency among various replica nodes. The secondary replica nodes will store the data immediately after receiving it, and the primary replica node will return a storage confirmation signal to confirm the success of the data storage operation. The system can then return a storage success response to the business system, indicating that the data has been safely stored.

[0046] In the above embodiments, based on the refined storage strategy and dynamic adjustment mechanism, it has significant advantages in improving the availability, fault tolerance, and performance of the distributed storage system. First, by analyzing the storage request content and parsing out the storage strategy tags, the system can automatically adjust the storage strategy according to different data requirements to ensure that the storage method of each data unit can be optimized. Second, through flexible sharding strategies and replica distribution strategies, the system can efficiently manage data in a large-scale distributed environment, ensuring high data availability and fast access capabilities. Finally, through replica replication and load monitoring mechanisms, the storage system can achieve dynamic load balancing and high fault tolerance, ensuring data integrity and continuous service availability even in extreme cases.

[0047] Refer to Figure 2 , as an embodiment of step S104, the steps of determining the number of replicas according to the fault tolerance level, generating a replica distribution strategy based on the distributed cluster topology structure, and specifying the list of primary replica nodes and secondary replica nodes include: Step S201, obtain the available zone identifier of the business request source of the data unit; Specifically, in a distributed storage system, the storage and access efficiency of data is often affected by the location of the request source. To ensure that data can be efficiently accessed within the available zone of the request source, it is first necessary to obtain the available zone identifier of the business request source of the business data unit. This identifier usually comes from the user region of the requested data or the deployment location of the application service. This identifier helps the system determine the geographical location priority during subsequent replica selection and data migration processes, thereby optimizing data availability and access speed.

[0048] Exemplarily, assume that the business data unit of an e-commerce platform is order data from user A, and user A is located in availability zone AZ1. The system will record this availability zone identifier as the basis for generating subsequent replica strategies.

[0049] Step S202: Query the preset replica quantity mapping table according to the fault tolerance level of the data unit, and output the replica quantity value. Among them, to ensure the high availability and fault tolerance of data, the system determines the replica quantity based on the fault tolerance level label of the data unit. The higher the fault tolerance level, the more replicas are required. The fault tolerance level label is usually pre-mapped to a mapping table of replica quantities, and the replica quantity of this data unit is determined according to this table. This process helps the system dynamically generate replica strategies according to business requirements, so as to ensure the availability of services when the system faces node failures.

[0050] Exemplarily, assume that the fault tolerance level of the data unit order data is "high". In the replica quantity mapping table, the corresponding replica quantity for the high fault tolerance level is 3. Then the output replica quantity value N is 3.

[0051] Step S203: Based on the distributed cluster topology structure data and the availability zone identifier of the business request source, filter out the candidate nodes that meet the first network delay threshold and have the highest health score as the primary replica node, and output the primary replica node identifier. Among them, the selection of the primary replica node is the key to ensuring data access efficiency and system load balancing. Based on the cluster topology structure and the availability zone identifier of the business request source, the system first filters out the nodes within the same zone that meet certain network delay conditions (i.e., nodes with network delay less than the first network delay threshold). Then, by comprehensively considering the health scores of the nodes (including factors such as CPU usage rate and storage delay), the node with the highest score is selected as the primary replica node. The calculation of the health score can ensure that the selected node has high performance and stability and can carry the load of the primary replica.

[0052] Exemplarily, assume that the business request comes from availability zone AZ1. The topology shows that there are nodes A and B in availability zone AZ1, and the network delay of node A is 10ms and that of node B is 30ms. The system first filters out the nodes with a delay ≤ 20ms, and then selects node A as the primary replica node through the health score (assuming the health score of node A is 90 and that of node B is 80).

[0053] Step S204: According to the replica quantity value and the primary replica node identifier, filter out the candidate nodes in different availability zones from the primary replica node and meeting the second network delay threshold as the secondary replica nodes, and output the list of secondary replica nodes. Among them, the selection of secondary replica nodes aims to ensure data redundancy and high availability, and they need to be deployed in different availability zones to avoid data loss caused by multiple node failures in the same area simultaneously. According to the output replica quantity value N, the system will select N - 1 eligible secondary replica nodes. The screening conditions include different availability zones and the second network latency threshold, that is, to ensure that the geographical distribution of the replica nodes is different from that of the primary replica node in different availability zones, reduce the impact of regional failures, and at the same time ensure that the network latency of the secondary replica nodes does not exceed a certain threshold, thereby ensuring the efficiency of data access.

[0054] Exemplarily, assume the replica quantity is 3, and the primary replica node A is located in availability zone AZ1. The system will select nodes that meet the following conditions: Node B: Located in availability zone AZ2, with a network latency of 15ms, which meets the requirements. Node C: Located in availability zone AZ3, with a network latency of 25ms, which meets the requirements. Therefore, the list of secondary replica nodes output is: ["AZ2-NodeB","AZ3-NodeC"].

[0055] It can be understood that by intelligently screening the secondary replica nodes, the system can ensure the geographical distribution of the replicas, enhance the fault tolerance of the system, and optimize the data access speed and network efficiency. Deployment in multiple availability zones can avoid single points of failure and improve the high availability of data.

[0056] Step S205, bind the primary replica node identifier and the list of secondary replica nodes to the data shard identifier, generate a replica distribution policy, and store it in the distributed configuration center.

[0057] Among them, after the selection of replica nodes is completed, the system will bind the primary replica node identifier and the list of secondary replica nodes to the data shard identifier to generate a replica distribution policy file. This policy file contains the specific information of the replica nodes and serves as the basis for the subsequent execution of the storage system. Finally, the replica distribution policy file will be stored in the distributed configuration center, such as ZooKeeper, to ensure that all nodes in the system can synchronize the latest replica distribution policy in a timely manner.

[0058] In the above embodiments, based on a flexible replica policy generation method, combined with the fault tolerance requirements of business data units and the network topology structure, the primary replica node and the list of secondary replica nodes are intelligently selected to ensure the high availability, redundancy, and access efficiency of data. By considering network latency, health scores, and the geographical distribution of nodes, the system optimizes the load balancing and fault tolerance of the distributed storage system. The generated replica distribution policy file ensures the consistency and synchronization of the configurations of each node, thereby improving the reliability and performance of the system.

[0059] Refer to Figure 3, as an implementation manner of step S203, the steps of screening candidate nodes that meet the first network latency threshold and have the highest health score as the primary replica node based on the distributed cluster topology structure data and the available zone identifier of the service request source include: Step S301, obtaining the available zone location and historical failure rate of each candidate node from the distributed cluster topology structure data; Among them, in a distributed storage system, the available zone location and historical failure rate of each node are important factors affecting node selection. The available zone location determines whether a node is physically in the same area as other nodes, while the historical failure rate reflects the reliability of the node. In this step, the system extracts this information from the topology structure data of the distributed cluster. The historical failure rate can be obtained through monitoring data, including the number of node outages, network interruption frequencies, etc. Based on this information, the system can evaluate the applicability of each node in the fault tolerance design.

[0060] Step S302, collecting the load metrics of each candidate node in real time; Among them, the load status of a node directly affects its ability to process requests and response speed. Load metrics usually include CPU usage rate, memory occupancy rate, disk read and write speed, etc. In this step, the system collects the load information of each candidate node in real time to evaluate the current status of the node. A node with a high load may not be able to provide services stably, so it is necessary to exclude it or reduce its priority as the primary replica node.

[0061] Step S303, calculating the health score of each candidate node according to the load metrics and historical failure rate; Among them, the health score is an important indicator for evaluating the current health status (such as load, resource usage) of a node in a distributed system. The system combines the load metrics and historical failure rate of the node and calculates the health score through a weighted formula. Load metrics usually account for a larger weight because the real-time performance of the node directly affects its ability to provide services effectively. The historical failure rate reflects the reliability of the node and usually accounts for a smaller weight. Nodes with a high health score have better performance and lower failure rates and are the preferred choices as primary replica nodes.

[0062] In one embodiment of the present application, the calculation formula of the health score is configured as: health score = (100 - real-time CPU usage rate) × 0.7 + (100 - historical failure rate × 100) × 0.3. By comprehensively considering the load and historical failure rate, the system can generate a scientific health score, enabling the selected node to have better processing capabilities and lower failure rates, thereby improving the reliability of the primary replica node.

[0063] Step S304: Based on the available zone identifier of the service request source, filter candidate nodes that are in the same zone as the available zone of the service request source and meet the first network latency threshold, to obtain the first candidate node set; Among them, to ensure efficient data access, the system should select nodes in the same available zone as the service request source as the primary replica nodes. This can significantly reduce network latency and improve data access speed. First, the system filters out nodes in the same zone according to the available zone identifier of the service request source. Then, it further filters out nodes with network latency within the set threshold to ensure that the response time of the request meets the business requirements.

[0064] Exemplarily, assume that the service request comes from the available zone AZ1, nodes A and B are both located in AZ1, while node C is located in AZ2. Assume that the latency from AZ1 to node A is 10ms, node B is 30ms, and node C is 50ms. The system first filters out nodes A and B in the same zone AZ1, and then filters out node A with a latency ≤ 20ms according to the latency threshold. Therefore, node A will enter the first candidate node set.

[0065] It can be understood that by selecting nodes in the same available zone as the service request source and with network latency meeting the requirements, the system effectively reduces network latency and improves the user's data access experience.

[0066] Step S305: Sort the first candidate node set according to the health score, and select the candidate node with the highest health score as the primary replica node.

[0067] Specifically, after filtering out the candidate node set that meets the network latency requirements, the system sorts these nodes according to the health score. The higher the health score of a node, the better its performance and the more stable it can handle service requests. Finally, the node with the highest health score is selected as the primary replica node, ensuring the high availability and high performance of the system.

[0068] In the above embodiment, combining the distributed cluster topology structure data, real-time load monitoring, historical failure rate, and network latency, comprehensively evaluate the health status of each candidate node, and intelligently select the primary replica node. By accurately filtering nodes with low latency in the same zone and sorting according to the health score, the system can ensure the selection of a primary replica node with excellent performance and stability, maximizing the data access efficiency and the fault tolerance ability of the system. Finally, this solution realizes an efficient replica distribution strategy, ensuring the high availability, low latency, and high reliability of the distributed storage system.

[0069] Referring to Figure 4 , as an implementation manner of step S204, the step of filtering candidate nodes that are in different available zones from the primary replica node and meet the second network latency threshold as secondary replica nodes according to the replica quantity value and the primary replica node identifier includes: Step S401: Based on the primary replica node identifier, filter candidate nodes in different availability zones from the primary replica node to obtain a second candidate node set. Specifically, in a distributed storage system, the selection of secondary replica nodes needs to ensure that they are in different availability zones from the primary replica node, which can avoid the risk of all replicas becoming invalid due to a single availability zone failure. Therefore, the system first filters out nodes in different availability zones from the primary replica node according to the primary replica node identifier to construct a second candidate node set. The goal of this step is to distribute replica nodes in multiple availability zones to improve the fault tolerance of the system and the availability of data.

[0070] Exemplarily, assume that the primary replica node is AZ1-NodeA. The system filters out nodes in different availability zones from AZ1 according to the topology data, such as AZ2-NodeB, AZ3-NodeC, etc. The filtered candidate node set is: ["AZ2-NodeB", "AZ3-NodeC"].

[0071] Step S402: Calculate the network latency between each candidate node in the second candidate node set and the primary replica node. Among them, network latency is a key factor affecting data access efficiency. Especially in a distributed system, the smaller the network latency between the replica node and the primary replica node, the higher the efficiency of data synchronization and access. Therefore, the system will calculate the network latency between each candidate node and the primary replica node. This latency calculation helps to select secondary replica nodes with lower network latency, thereby improving data access speed and the overall performance of the system.

[0072] Exemplarily, assume that the network latencies between the primary replica node AZ1-NodeA and the candidate nodes AZ2-NodeB and AZ3-NodeC are 15ms and 30ms respectively. The system will record and calculate these latency values as the basis for subsequent node filtering.

[0073] Step S403: Filter candidate nodes whose network latency meets the second network latency threshold and sort them in ascending order according to the network latency. Among them, to ensure data synchronization and access efficiency, the system will set a second network latency threshold. Only those candidate nodes with network latency less than or equal to this threshold will be selected as secondary replica nodes. After filtering out nodes that meet the latency requirements, the system will sort them in ascending order according to the latency value to ensure that the nodes with the smallest latency are selected as secondary replica nodes first. This sorting process helps to optimize the selection of nodes and minimize the overall latency of the system.

[0074] Exemplarily, assume that the network latencies of candidate nodes AZ2-NodeB and AZ3-NodeC are 15 ms and 30 ms respectively, and the second network latency threshold set by the system is 20 ms. The filtered nodes are AZ2-NodeB (latency 15 ms) and AZ3-NodeC (latency 30 ms). After ascending sorting, AZ2-NodeB will be in the front.

[0075] Step S404: According to the replica quantity value N, select the first N - 1 candidate nodes as slave replica nodes.

[0076] Among them, according to the replica quantity value N, the system will select the first N - 1 nodes with the lowest latency as slave replica nodes to ensure that the quantity of replica nodes meets the business fault tolerance requirements. The reason for selecting N - 1 nodes is that there is already a master replica node, and the remaining quantity of replica nodes should meet the requirements of the replica quantity value. This step guarantees the geographical distribution, latency requirements, and fault tolerance of the replica nodes, ensuring data redundancy and availability.

[0077] Exemplarily, assume that the number of replicas is 3 (i.e., N = 3). After latency filtering and sorting, the latency of node AZ2-NodeB is 15 ms, and the latency of node AZ3-NodeC is 30 ms. Since the number of replicas is 3, the system needs to select 2 slave replica nodes. In this example, the system will select nodes AZ2-NodeB and AZ3-NodeC as slave replica nodes.

[0078] It can be understood that by selecting the optimal slave replica nodes according to the replica quantity value, the system ensures that the quantity and quality of the replicas meet the high availability requirements of the business, and guarantees that the latency of the replica distribution is as low as possible, thereby improving the stability and performance of the system.

[0079] In the above embodiments, nodes in different availability zones from the master replica node are accurately filtered, and based on the requirements of network latency and the number of replicas, the selection process of slave replica nodes is optimized. First, it is ensured that the slave replica nodes are distributed in different availability zones, thereby improving the fault tolerance of the system; second, by calculating the network latency between nodes and sorting, it is ensured that the nodes with the lowest latency are selected as slave replica nodes, thereby improving data access speed and synchronization efficiency.

[0080] Refer to Figure 5 , as an embodiment of step S105, the steps of writing data shards into the master replica node according to the replica distribution policy and synchronously initiating the replica replication operation to the slave replica nodes include: Step S501: Analyze the replica distribution policy, and extract the master replica node identifier, the list of slave replica nodes, and the synchronization protocol type; Among them, in a distributed storage system, the replica distribution policy file contains key information such as data distribution, node selection, and synchronization rules. The main purpose of this step is to parse the replica distribution policy file and extract the identifier of the primary replica node, the list of secondary replica nodes, and the synchronization protocol type. The primary replica node is responsible for receiving write requests and recording data, while the secondary replica nodes are used for data replication and redundancy. The synchronization protocol type defines the data synchronization method between replica nodes, such as strong synchronization, weak synchronization, etc.

[0081] Exemplarily, by parsing this file, the system extracts that the identifier of the primary replica node is AZ1-NodeA, the list of secondary replica nodes is ["AZ2-NodeB","AZ3-NodeC"], and the synchronization protocol type is "strong".

[0082] Step S502, send a data shard write request to the primary replica node and receive the write timestamp and data checksum returned by the primary replica node; Among them, when a business request needs to write data to the distributed storage system, the system first sends the data shards to the primary replica node. The primary replica node writes the data to the local storage engine, generates a data checksum (such as a hash value) to verify data integrity, and records the physical timestamp of the write. Then, the primary replica node returns the write timestamp and data checksum to confirm that the data has been successfully written. These information are crucial for subsequent data replication and consistency verification.

[0083] Step S503, initiate a data replication request according to the list of secondary replica nodes for synchronization; among them, the data replication request includes the data shard content, data checksum, and write timestamp; Among them, to ensure data high availability and redundancy, the primary replica node will replicate the data to the secondary replica nodes. The system will initiate data replication requests to all secondary replica nodes concurrently according to the list of secondary replica nodes in the replica distribution policy. The data replication request includes the data shard content, write timestamp, and data checksum to ensure the consistency and correctness of the data replicas. This step ensures that after the data is written to the primary replica node, the replica nodes can synchronously receive and store the same data.

[0084] Exemplarily, assume that the data checksum returned by the primary replica node is "abc123" and the timestamp is "2025-06-04T10:00:00Z". The system will initiate replication requests to AZ2-NodeB and AZ3-NodeC, and the requests include: data shard content, checksum "abc123", and timestamp "2025-06-04T10:00:00Z".

[0085] Step S504, collect the response status of each slave replica node, and determine the data consistency result according to the synchronization protocol type and the number of successfully responded replicas; Among them, after the data replication request is sent to the slave replica node, the system will wait for the response of the replica node and collect the response status of each slave replica node. If the response is successful, it means that the replica node has successfully received and stored the data; if the response fails, it means that there is a problem with this node during the data synchronization process. According to different synchronization protocol types, the system will determine the data consistency result based on the number of successfully responded nodes. In the strong synchronization mode, the system requires that most replica nodes respond successfully to consider the data consistency verification passed; in the weak synchronization mode, the system may allow a small number of replica nodes to fail.

[0086] Specifically, the steps of determining the data consistency result according to the synchronization protocol type and the number of successfully responded replicas include: if the synchronization protocol type is the strong synchronization mode, count the number of successfully responded nodes; when the number of successfully responded nodes exceeds half of the total number of slave replica nodes, mark the data consistency result as committed, and generate a status mapping table containing the identifiers of the failed nodes.

[0087] Exemplarily, assume that the synchronization protocol type is "strong", and the replica nodes AZ2-NodeB and AZ3-NodeC respond successfully and fail respectively. The system will count the number of successfully responded nodes and find that there is only 1 successfully responded node. Since the number of successful nodes is insufficient, the system determines that the data consistency verification fails.

[0088] Step S505, according to the list of failed nodes in the data consistency result, perform retry replication of replica data or trigger an update of the replica distribution strategy.

[0089] Among them, if some slave replica nodes fail to synchronize data successfully, the system will record these failed nodes in the list of failed nodes. According to the consistency result, the system will decide whether to perform data replication retry or trigger an update of the replica distribution strategy. The data retry uses the exponential backoff algorithm to calculate the retry delay time: delay time = base delay × (2 重试次数 ) to avoid excessive load on the system caused by multiple retries in a short period. If the data still cannot be synchronized successfully after multiple retries, the system will call the replica distribution strategy generation module to update the replica distribution strategy, replace the failed slave replica node, and broadcast the new replica distribution strategy to all nodes.

[0090] Exemplarily, assume that the replica node AZ3-NodeC still cannot receive data successfully after multiple retries. The system will call the replica distribution strategy generation module to select a new slave replica node AZ4-NodeD to replace AZ3-NodeC, update the replica distribution strategy and broadcast it to all nodes.

[0091] In the above embodiments, by parsing the replica distribution policy file, the system obtains the node information of the write request and the type of synchronization protocol. Then, the data is written through the primary replica node and synchronized to the secondary replica nodes to ensure redundant storage and high availability of the data. During the data synchronization process, the system determines the consistency result through the response status and processes it according to different types of synchronization protocols. For failed nodes, the system adopts a retry mechanism and updates the replica distribution policy when necessary to handle replica node failures or network problems. The overall solution ensures the stability, reliability, and data consistency of the distributed storage system through efficient data writing, replication, and consistency verification mechanisms.

[0092] Referring to Figure 6 , as a further embodiment of the efficient distributed storage method, after the steps of writing data shards into the primary replica node according to the replica distribution policy and synchronously initiating the replica replication operation to the secondary replica nodes, it further includes: Step S601, collecting the load metrics of the primary replica node and the secondary replica nodes in real time; Among them, the load metrics include CPU usage rate, network IO throughput, and storage latency. In a distributed storage system, load monitoring is a key component to ensure the efficient operation of the system. By collecting the load metrics of the primary replica node and all secondary replica nodes in real time, the system can comprehensively understand the current operating status of each node, and then make timely responses and optimization decisions.

[0093] Specifically, the CPU usage rate reflects the usage of the node's processing capacity. An excessively high CPU usage rate indicates that the node may have a computing bottleneck, affecting data processing and response speed. The network IO throughput represents the speed of data transmission between the node and other nodes, directly affecting the efficiency of data synchronization and access. A network with high throughput can respond to data requests faster. Storage latency refers to the time required for the node to read or write data. A higher storage latency usually indicates that the node is overloaded or has insufficient storage performance, which may cause the system response time to become longer and affect the data access speed.

[0094] Step S602, calculating a load score based on a pre-constructed load evaluation model; Among them, the calculation of the load score is based on a preset load evaluation model, which comprehensively scores each node using the collected load metrics. The design of the scoring formula takes into account multiple key factors: the lower the CPU usage rate, the healthier the node; the higher the throughput, the faster it can respond to data requests; and the lower the storage latency, the faster the node responds.

[0095] Specifically, the load evaluation model can calculate the load score through a pre-constructed weighted formula. The specific formula is: Score = (0.5×(100 - cpu_usage)) + (0.3×io_throughput / 300) + (0.2×(50 - storage_latency)); In the above formula, Score is the load score, cpu_usage is the CPU usage rate, io_throughput is the network I / O throughput, and storage_latency is the storage latency.

[0096] Step S603: Determine whether the load score of the primary replica node is lower than the score threshold for a preset number of consecutive times; if so, jump to step S604; if not, do nothing. Specifically, in a distributed storage system, the load status of the primary replica node is crucial for the performance of the entire system. If the load score of the primary replica node is lower than the set threshold for multiple consecutive times, it may indicate that the node is overloaded and cannot continue to provide services efficiently. Therefore, the system checks the load score of the primary replica node according to the set number of consecutive times. If the consecutive detection results are all lower than the threshold, the primary replica migration process will be triggered. Setting the threshold to "80 points" means that a node with a higher load will be considered in a high-risk state, and a score lower than 60 points indicates that migration is required as soon as possible.

[0097] Exemplarily, assume that the load scores of the primary replica node A are 50, 55, and 58, and the threshold is set at 60 points. Then the load score of node A is lower than 60 points in 3 consecutive monitoring times, which will trigger the primary replica migration process.

[0098] Step S604: Select the secondary replica node with the highest load score and meeting the topology constraints from the list of secondary replica nodes as the new primary replica node. Among them, once it is determined that the load score of the primary replica node is lower than the threshold and reaches the preset number of times, the system will select a new primary replica node according to the load scores of the current replica group and the topology constraint conditions. The selection criterion gives priority to the secondary replica node with the highest load score to ensure that the new primary replica node has a lower load and can undertake the tasks of the primary replica. In addition, it is also necessary to meet the topology constraint conditions, such as network latency requirements, availability zone constraints, etc., to ensure that the new primary replica node can provide better access performance and system stability.

[0099] Exemplarily, assume that there are two secondary replica nodes B and C in the replica group. The load score of node B is 85, the score of node C is 95, and the latency of node C with the business source availability zone is less than 25 ms. The system then selects node C as the new primary replica node.

[0100] It is understandable that by dynamically selecting the slave replica node with the lowest load and meeting the topology constraints, the system can balance the load and ensure that the performance of the system is not affected after the master replica is migrated. In addition, this method can improve the load balancing ability of the system and avoid a certain node becoming a bottleneck.

[0101] Step S605: Incrementally synchronize the data shards from the original master replica node to the new master replica node, and update the replica distribution policy after the synchronization is completed.

[0102] Among them, the data migration process must be incremental synchronization, which means that only the data that has changed since the last synchronization is synchronized, rather than the entire data set. This can significantly reduce the time and bandwidth consumption of data migration and improve the efficiency of the migration process. After the synchronization is completed, the system will update the replica distribution topology map, clarify the location of the new master replica node, and notify all replica nodes to take effect the new storage policy.

[0103] Exemplarily, assume that when migrating the order data of the original master replica node A, its incremental data only contains the newly added order information. The original master replica node A synchronizes this newly added data to the new master replica node C. After completion, the replica distribution policy is updated. The new replica topology map will include node C as the master replica node and notify other nodes in the system to synchronize the new replica policy.

[0104] It is understandable that the incremental synchronization mechanism ensures the efficiency of the migration process and avoids the resource waste caused by full-scale data migration. By timely updating the replica distribution policy, it ensures the consistency and data integrity of the system and reduces the service interruption or performance degradation caused by the master replica migration.

[0105] In the above embodiments, by real-time collecting and evaluating the load conditions of each node, the system can automatically migrate the master replica node when the load is too heavy, ensure the high availability of the system, and avoid the impact on the service caused by single point of failure. The incremental synchronization and topology update strategies ensure the data consistency and the efficiency of the migration process.

[0106] The embodiment of the present application also discloses an efficient distributed storage system.

[0107] An efficient distributed storage system, the storage system includes: A receiving module, configured to receive a data storage request from a business system; An analysis module, configured to analyze the data storage request and output a data unit including a storage policy label; the storage policy label includes a data sharding granularity level and a fault tolerance level; A data splitting module, configured to split the data unit into data shards of corresponding sizes according to the data sharding granularity level; The replica distribution strategy generation module is used to determine the number of replicas according to the fault tolerance level, generate a replica distribution strategy based on the distributed cluster topology structure, and specify the list of primary replica nodes and secondary replica nodes; The data storage module is used to write data shards into the primary replica node according to the replica distribution strategy, and synchronously initiate a replica replication operation to the secondary replica nodes; The storage success response module is used to receive the storage confirmation signal from the primary replica node and return a storage success response to the service system.

[0108] As a further implementation of the efficient distributed storage system, the storage system further includes: The load metric collection module is used to collect the load metrics of the primary replica node and the secondary replica nodes in real time; The load score calculation module is used to calculate the load score based on a pre-constructed load evaluation model; The judgment module is used to judge whether the load score of the primary replica node is lower than the score threshold for a preset number of consecutive times; if so, output a first judgment result; The node selection module is used to, in response to the first judgment result, select the secondary replica node with the highest load score and satisfying the topology constraint in the list of secondary replica nodes as the new primary replica node; The data incremental synchronization module is used to incrementally synchronize the data shards from the original primary replica node to the new primary replica node; The policy update module is used to update the replica distribution strategy after the synchronization is completed.

[0109] The efficient distributed storage system of the embodiments of the present application can implement any of the above storage methods, and the specific working processes of each module in the storage system can refer to the corresponding processes in the above method embodiments.

[0110] In several embodiments provided by the present application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are only illustrative; for example, the division of a certain module is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0111] The embodiments of the present application also disclose a computer device.

[0112] The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the efficient distributed storage method as described above.

[0113] The embodiments of the present application also disclose a computer-readable storage medium.

[0114] A computer-readable storage medium stores a computer program that can be loaded and executed by a processor to perform any one of the efficient distributed storage methods as described above.

[0115] Among them, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, device, or component; the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the above.

[0116] It should be noted that in the above embodiments, the descriptions of the various embodiments have their own focuses. For parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0117] The above are all preferred embodiments of the present application. The protection scope of the present application is not limited by this. Any feature disclosed in this specification (including the abstract and drawings), unless specifically described, can be replaced by other equivalent or similar-purpose alternative features. That is, unless specifically described, each feature is only an example in a series of equivalent or similar features.

Claims

1. An efficient distributed storage method, characterized in that, The storage method includes: Receiving a data storage request from a business system; Parsing the data storage request and outputting a data unit containing a storage policy tag; the storage policy tag includes a data sharding granularity level and a fault tolerance level; Splitting the data unit into data shards of corresponding sizes according to the data sharding granularity level; Determining the number of replicas according to the fault tolerance level, generating a replica distribution policy based on the distributed cluster topology structure, and specifying a list of primary replica nodes and secondary replica nodes; Writing the data shards to the primary replica node according to the replica distribution policy, and synchronously initiating a replica replication operation to the secondary replica nodes; Receiving a storage confirmation signal from the primary replica node and returning a storage success response to the business system.

2. An efficient distributed storage method according to claim 1, characterized in that, The step of determining the number of replicas according to the fault tolerance level, generating a replica distribution policy based on the distributed cluster topology structure, and specifying a list of primary replica nodes and secondary replica nodes includes: Obtaining the available zone identifier of the business request source of the data unit; Querying a preset replica number mapping table according to the fault tolerance level of the data unit and outputting a replica number value; Based on the distributed cluster topology structure data and the available zone identifier of the business request source, screening candidate nodes that meet the first network latency threshold and have the highest health score as the primary replica node, and outputting the primary replica node identifier; According to the replica number value and the primary replica node identifier, screening candidate nodes in different available zones from the primary replica node and meeting the second network latency threshold as secondary replica nodes, and outputting a list of secondary replica nodes; Binding the primary replica node identifier and the list of secondary replica nodes to the data shard identifier, generating a replica distribution policy and storing it in the distributed configuration center.

3. An efficient distributed storage method according to claim 2, characterized in that, The step of screening candidate nodes that meet the first network latency threshold and have the highest health score as the primary replica node based on the distributed cluster topology structure data and the available zone identifier of the business request source includes: Obtaining the available zone location and historical failure rate of each candidate node from the distributed cluster topology structure data; Real-time collecting the load metrics of each candidate node; Calculating the health score of each candidate node according to the load metrics and the historical failure rate; Based on the available zone identifier of the business request source, screening candidate nodes in the same available zone as the business request source and meeting the first network latency threshold to obtain a first candidate node set; Sorting the first candidate node set according to the health score, and selecting the candidate node with the highest health score as the primary replica node.

4. An efficient distributed storage method according to claim 3, characterized in that, The step of screening candidate nodes in different available zones from the primary replica node and meeting the second network latency threshold as secondary replica nodes according to the replica number value and the primary replica node identifier includes: Based on the primary replica node identifier, screening candidate nodes in different available zones from the primary replica node to obtain a second candidate node set; Calculating the network latency between each candidate node in the second candidate node set and the primary replica node; Screening candidate nodes whose network latency meets the second network latency threshold and sorting them in ascending order according to the network latency; According to the replica number value N, selecting the first N - 1 candidate nodes as secondary replica nodes.

5. An efficient distributed storage method according to claim 1, characterized in that, According to the replica distribution strategy, the steps of writing the data shards to the primary replica node and synchronously initiating a replica replication operation to the secondary replica nodes include: Analyze the replica distribution strategy, and extract the primary replica node identifier, the list of secondary replica nodes, and the synchronization protocol type; Send a data shard writing request to the primary replica node, and receive the writing timestamp and the data verification code returned by the primary replica node; Synchronously initiate a data replication request according to the list of secondary replica nodes; wherein, the data replication request includes the data shard content, the data verification code, and the writing timestamp; Collect the response status of each secondary replica node, and determine the data consistency result according to the synchronization protocol type and the number of successfully responded replicas; According to the list of failed nodes in the data consistency result, perform retry replication of replica data or trigger an update of the replica distribution strategy.

6. An efficient distributed storage method according to any one of claims 1 to 5, characterized in that, After the step of writing the data shards to the primary replica node and synchronously initiating a replica replication operation to the secondary replica nodes according to the replica distribution strategy, the following steps are further included: Real-time collect the load metrics of the primary replica node and the secondary replica nodes; Calculate the load score based on a pre-constructed load evaluation model; Determine whether the load score of the primary replica node is lower than the score threshold for a preset number of consecutive times; If so, select the secondary replica node with the highest load score and meeting the topology constraint in the list of secondary replica nodes as the new primary replica node; Incrementally synchronize the data shards from the original primary replica node to the new primary replica node, and update the replica distribution strategy after the synchronization is completed.

7. An efficient distributed storage system, characterized in that, The storage system includes: A receiving module, configured to receive a data storage request from a service system; An analysis module, configured to analyze the data storage request and output a data unit including a storage policy label; the storage policy label includes a data shard granularity level and a fault tolerance level; A data splitting module, configured to split the data unit into data shards of corresponding sizes according to the data shard granularity level; A replica distribution strategy generation module, configured to determine the number of replicas according to the fault tolerance level, generate a replica distribution strategy based on the distributed cluster topology structure, and specify the primary replica node and the list of secondary replica nodes; A data storage module, configured to write the data shards to the primary replica node according to the replica distribution strategy, and synchronously initiate a replica replication operation to the secondary replica nodes; A storage success response module, configured to receive the storage confirmation signal from the primary replica node and return a storage success response to the service system.

8. An efficient distributed storage system according to claim 7, wherein, The storage system further includes: A load metric collection module, configured to real-time collect the load metrics of the primary replica node and the secondary replica nodes; A load score calculation module, configured to calculate the load score based on a pre-constructed load evaluation model; A judgment module, configured to determine whether the load score of the primary replica node is lower than the score threshold for a preset number of consecutive times; if so, output a first judgment result; A node screening module, configured to, in response to the first judgment result, select the secondary replica node with the highest load score and meeting the topology constraint in the list of secondary replica nodes as the new primary replica node; The data incremental synchronization module is used to incrementally synchronize data shards from the original primary replica node to the new primary replica node; The policy update module is used to update the replica distribution policy after the synchronization is completed.

9. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: A computer program is stored that can be loaded and executed by a processor to implement the method according to any one of claims 1 to 6.

Images