Log submission and data update method and device based on erasure code and medium

Abstract

The application discloses a log submission and data updating method and device based on erasure code and a medium. The method can be applied to a leader node of a server cluster. First, data shards to be written are received. Then, log data containing update information of the data shards is obtained. Next, nodes to which target shards corresponding to each data shard belong are confirmed. The log data is sent to the corresponding nodes. After receiving confirmation information from the nodes, which indicates that the log data is correctly received, the target shards of the nodes are incrementally updated according to the log data. The embodiment of the application adopts an incremental updating strategy, which can effectively alleviate the write amplification problem and improve the efficiency of data updating.

Description

Technical Field

[0001] The embodiments of this application relate to, but are not limited to, the field of computer technology, and in particular to a log submission and data update method, device and medium based on erasure coding. Background Technology

[0002] In modern distributed systems, log commits and data updates play a crucial role in ensuring data consistency and high availability. However, these technologies often lack detailed descriptions when implementing log commits, particularly regarding write amplification, for which mature solutions are still lacking. Write amplification is especially pronounced in distributed storage systems, causing a surge in disk I / O operations and negatively impacting overall system performance. Specifically, current technologies often employ full updates for data writing, requiring the complete data to be read from all nodes and replaced with new data for each update. This not only exacerbates write amplification but also incurs significant network overhead. Furthermore, existing technologies retain the drawback of full replication in log copying, affecting the efficiency of data updates. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] This application provides a log commit and data update method based on erasure coding, which can effectively alleviate the write amplification problem and improve the efficiency of data update.

[0005] In a first aspect, embodiments of this application provide a log commit and data update method based on erasure coding, applied to the leader node of a server cluster, comprising: receiving a data shard to be written; obtaining log data of the data shard, the log data containing update information of the data shard; confirming the node to which the target shard corresponding to the data shard belongs, and sending the log data of the data shard to the node; receiving confirmation information from the node, the confirmation information being used to indicate that the log data has been correctly received by the node; and incrementally updating the target shard of the node according to the log data.

[0006] In conjunction with the first aspect, in one embodiment of this application, the node includes a data node and a verification node, and the target shard includes a target data shard and a target verification shard, wherein the target verification shard is stored in the verification node, and the target data shard is stored in the data node; the step of incrementally updating the target shard of the node according to the log data includes: obtaining the target data shard and the target verification shard corresponding to the data shard; incrementally updating the target verification shard of the verification node according to the target data shard, the target verification shard, and the log data; and incrementally updating the target data shard of the data node according to the log data.

[0007] In conjunction with the first aspect, in one embodiment of this application, the data node includes the leader node and follower nodes; the data node is the leader node, and the method further includes: generating a log sequence and log records based on the log data, wherein the log records are a merged form of the log sequence; storing the log sequence in memory, and writing the log records to disk.

[0008] In conjunction with the first aspect, in one embodiment of this application, the data node is the follower node, and the method further includes: synchronizing the log records to the follower node, so that the follower node performs recovery and merging processing on the received log records to obtain original log data, and stores the original log data in the disk and memory of the follower node; when the difference between the original log data of the follower node and the log sequence reaches a preset threshold, sending a complete copy of the log sequence to the follower node; the complete copy includes all log entries of the log sequence.

[0009] In conjunction with the first aspect, in one embodiment of this application, the log data includes multiple log fragments, and the step of generating a log sequence and log record based on the log data includes: generating a log sequence based on the multiple log data fragments, the log sequence including multiple log entries; and merging the multiple log entries to obtain a log record.

[0010] In conjunction with the first aspect, in one embodiment of this application, the method further includes: when the node is a faulty node, sending the log data to a candidate node for storage; after receiving a data reconstruction instruction, reading the log data from the candidate node, and reconstructing the target fragment of the faulty node based on the log data.

[0011] Secondly, embodiments of this application provide a log commit and data update method based on erasure coding, applied to the leader node of a server cluster. The method includes: receiving full striped data to be written; setting an update threshold based on the number of nodes in the server cluster; obtaining data shards and check shards based on the full striped data; confirming the target data node corresponding to the data shard and the target check node corresponding to the check shard, sending the data shard to the target data node, and sending the check shard to the target check node; determining the number of nodes based on the confirmation information sent by the target data node and the target check node; and confirming the update operation for the target data node and the target check node when the number of nodes is equal to the update threshold.

[0012] In conjunction with the second aspect, in one embodiment of this application, obtaining data fragments and verification fragments based on the full stripe data includes: splitting the full stripe data to obtain multiple data fragments; calculating the multiple data fragments according to a preset verification algorithm to obtain a verification value; and obtaining a verification fragment based on the verification value.

[0013] On the other hand, embodiments of this application provide an electronic device, including: at least one processor; at least one memory for storing at least one program; and when at least one of the programs is executed by at least one of the processors, implementing the log submission and data update method based on erasure coding as described above.

[0014] On the other hand, a computer-readable storage medium stores a processor-executable computer program, which, when executed by a processor, is used to implement the erasure coding-based log commit and data update method as described above.

[0015] This application provides a log commit and data update method based on erasure coding. First, it receives data fragments to be written. Then, it acquires the log data of these data fragments, which details the update information of the data fragments. Next, it confirms the node to which the target fragment belongs for each data fragment and sends the log data to the corresponding node. Afterward, it receives confirmation information from the node to verify that the log data has been correctly received. Finally, it incrementally updates the target fragment of the node based on this log data. This application adopts an incremental update strategy, that is, during the data writing process, only the changed parts are updated, rather than the entire data fragment. This strategy greatly reduces the number of disk I / O operations, effectively alleviating the write amplification problem, thereby improving the overall storage performance and response speed of the system. Furthermore, since incremental updates only transmit the changed log data, compared to full updates which require transmitting the entire data fragment, network communication volume is significantly reduced. This not only reduces network load but also further improves the efficiency of data updates. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the cluster node structure of related technologies;

[0017] Figure 2 This is a schematic diagram illustrating the conversion of consensus algorithms for related technologies;

[0018] Figure 3 This is a schematic diagram of the CRAFT encoding process;

[0019] Figure 4 This is a flowchart of the log submission and data update method based on erasure coding provided in the embodiments of this application;

[0020] Figure 5 This is a schematic diagram illustrating a specific example of data sharding updates provided in this application;

[0021] Figure 6 This is a schematic diagram illustrating a specific example of a log update scenario for a leader node's memory and disk in this application.

[0022] Figure 7 This is a schematic diagram of a log synchronization scenario between a leader node and follower nodes, provided as a specific example in this application.

[0023] Figure 8 This is another specific example of a log synchronization scenario between a leader node and follower nodes provided in this application;

[0024] Figure 9 This is provided by the embodiments of this application. Figure 4 The detailed flowchart of step 450;

[0025] Figure 10 This is a schematic diagram of the fragment update calculation process provided in the embodiments of this application;

[0026] Figure 11 This is a schematic diagram illustrating the data sharding process for writing to a node, provided as a specific example in this application.

[0027] Figure 12 This is another specific example of the data sharding writing process provided in this application;

[0028] Figure 13 This is a flowchart of a log submission and data update method based on erasure coding provided in another embodiment of this application;

[0029] Figure 14 This is a flowchart illustrating the process of writing full-strip data to a node, provided as a specific example in this application.

[0030] Figure 15 This is another specific example of the flowchart for writing full-strip data to a node provided in this application;

[0031] Figure 16 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] It should be noted that although the flowchart shows a logical order, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. The terms "first," "second," etc., used in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the structures, proportions, sizes, etc., depicted in the drawings are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the implementation conditions of this application. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to size, without affecting the effects and purposes achieved by this application, should still fall within the scope of the technical content disclosed in this application. Similarly, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are only for clarity of description and are not used to limit the scope of implementation of this application. Changes or adjustments in their relative relationships, without substantially altering the technical content, should also be considered within the scope of implementation of this application.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0035] In distributed systems, the consensus algorithm Raft ensures data consistency across multiple nodes. Erasure coding (EC) is a coding technique that enhances data fault tolerance in data storage or communication. It divides complete data into several data fragments, encodes them according to certain rules, and then generates several check fragments, allowing data to be recovered through decoding even if some data is corrupted or lost. Logs are the data transmission format in Raft. Logs use index numbers to record the order in which logs are sent from each node and term numbers to indicate which leader node's log it belongs to. Data is stored on disk through log replication and log writing operations.

[0036] It's worth noting that the Raft algorithm is also used for handling log sending and receiving. In distributed systems, data loss can occur at any level, from a silent data error on a single disk location on a server node to a power outage affecting the entire server or rack. How to recover lost data and maintain system availability is a critical issue in distributed systems. In a multi-replica storage model (where each server node stores the same data), the original Raft algorithm can solve this problem. It's a consensus algorithm for maintaining data consistency across multiple nodes in a distributed storage system. Raft stipulates that nodes in the system must satisfy the "majority" principle: 1. System Availability "Majority Principle": Of N = 2F+1 nodes, F nodes can be allowed to fail. Therefore, as long as F+1 nodes are still functioning normally, the entire system can continue to operate normally. 2. Election "Majority Principle": The node that collects more than half of the votes (quorum) becomes the leader. 3. Log Commit "Majority Principle": Once F+1 nodes have collected the logs, the logs can be committed and applied.

[0037] Raft defines three roles for all nodes: Leader, Follower, and Candidate. For example... Figure 1As shown. Each cluster has only one leader node at any given time, and the others default to follower nodes. They send "heartbeats" to other follower nodes every once in a while to inform that they still exist. When a follower node fails to receive a "heartbeat" after a timeout, it will consider the leader node to have crashed, so it switches to the candidate node state and conducts a new round of elections, soliciting votes from other nodes (such as most other server nodes). When a candidate node obtains more than half of the votes, it will become the leader node. It should be noted that a follower node will only vote for a candidate node when the candidate node's log is at least as up-to-date as the voting follower node. Each server can only vote once in each term or each round of elections. This also ensures the rule that at most only one leader node will be produced within a term. It is worth noting that when the leader node discovers that other nodes have a higher term, it will become a follower node. Data from the client can be received by the leader node and attempts to record this data into other nodes through log information to make the log data of other server nodes consistent with its own. When the leader node confirms that the user data in its term has been replicated to the "majority" of the servers, this entry and previous entries can be safely written to the local disk. The leader node will commit and write to the disk, and then notify the follower nodes to also commit and write the data to the disk. It should be noted that the multi-copy storage mode directly writes data to multiple nodes simultaneously. As long as one node stores the correct and complete data, there will be no data loss problem. However, the multi-copy mode often occupies N times the storage space of the original data. To reduce the storage cost, the erasure code (EC) storage mode emerged. In the erasure code mode, there is a new division of N nodes to replace the original 2N + 1: k + m. It divides the complete data into k data segments of the same size, and through certain encoding rules, m (m < k) parity shards are obtained. Then, the data on the k data segments is stored in k data nodes, and the newly generated m parity data is stored in the parity nodes. It can meet the requirement that when no more than m shards (<= m) of data are lost, the lost data can be recovered through the decoding rules. This ensures the same reliability as the multi-copy mode and reduces the storage cost of N times (assuming there are N copies) of the original data in the multi-copy mode to (k + m) / k times. Among them, RS-RAID is a very representative classic erasure code type.

[0038] See Figure 2 , Figure 2This is a schematic diagram illustrating the conversion of consensus algorithms in related technologies. RS-Paxos is a distributed system based on Paxos (the predecessor of Raft) that applies the RS-RAID consensus algorithm. However, compared to the original Paxos consensus algorithm, RS-Paxos has poor availability. With a configuration of N = 2F + 1 servers, it can no longer tolerate F failures like the original Paxos. Although RS-Paxos incorporates EC to save storage and network costs, this combination also brings challenges in terms of liveness. Specifically, RS-Paxos may fail to achieve the expected liveness level in some situations, causing the system to be unable to respond to requests or process events in a timely manner. The Raft protocol itself is developed based on Paxos and is a more easily understood consensus protocol. CRaft is based on the Raft protocol and introduces EC to further save storage and network costs while maintaining the same liveness level as Raft. In the context of distributed systems and consensus protocols, liveness usually refers to the ability of a system or protocol to complete certain operations or tasks within a finite time, such as submitting proposals and reaching consensus. It is an important indicator for measuring the activity and vitality of a system or protocol. Compared to RS-Paxos, CRaft has significant advantages in maintaining activity. For example, CRaft combines the methods of coded fragment replication and complete entry replication, which saves resources while maintaining high system activity. Figure 3 The CRAFT encoding process is illustrated. When the leader node in CRAFT attempts to replicate an entry using the encoded fragment replication method, it first encodes the entry. In the Raft protocol, each log entry should contain the original content received from the client, as well as the entry's term and index information during the protocol's operation. When the CRAFT leader node begins encoding an entry, it can operate according to the selected encoding scheme (e.g., (km)-RS code). During this process, the original content can be divided into multiple content fragments, such as content fragment1, content fragment2, and content fragment3. These content fragments are then encoded to generate corresponding encoded fragments, namely encoded content fragment1, encoded content fragment2, and encoded content fragment3. It is worth noting that, depending on the specific encoding scheme, parity fragments may also need to be generated to enhance data reliability and recovery capabilities. Figure 3In the example, content fragment4 and content fragment5 can act as parity check fragments. However, it's important to note that terms and indexes, as metadata, do not require encoding. Finally, the leader node combines the terms, indexes, and encoded data fragments into multiple data structures. Each data structure fully contains the corresponding term, index, and encoded content segments, providing the foundation for subsequent data replication and storage processes.

[0039] The encoding process includes an indexing phase. In this phase, the system receives or processes multiple index-related elements, which may include terms, content fragments, and possible splitting operations. Terms typically represent keywords or phrases in a document or data, while content fragments are the specific text blocks in which these terms appear. Next, the system enters a combine phase. In this phase, the system combines multiple content fragments together, preparing them for encoding. Encoding operations may include some form of compression, encryption, or formatting of the content fragments to reduce data volume or improve data security during storage or transmission. During encoding, the system can perform individual encoding operations on each content fragment and generate corresponding verification information (such as parity codes). This verification information is typically used to verify data integrity during data decoding or reconstruction. In the diagram, these encoded content fragments (labeled "content fragment(parity)") are connected together, indicating that they are processed continuously throughout the encoding process. Finally, the encoded content fragments, along with the associated terms and verification information, can be reintegrated into a new index. This new index can be used to support efficient searching, querying, or other data operations.

[0040] In distributed systems, log commits and data updates are crucial for ensuring data consistency and high availability. CRAFT, a distributed system solution based on the Raft consensus algorithm for managing log replication, while improving system availability to some extent, still faces several challenges in practical applications. First, the CRAFT solution lacks detailed descriptions of its log commit implementation, particularly regarding the lack of a robust solution for handling write amplification. Write amplification is particularly prominent in distributed storage systems, increasing the number of disk input / output operations and impacting overall system performance. In CRAFT replication, the write amplification problem often results in performance falling short of expectations, limiting its application in high-concurrency, high-data-volume scenarios. Second, CRAFT uses a full update approach for data writing, requiring each update to read data from all nodes and perform a complete replacement. This not only exacerbates write amplification but also introduces significant network overhead. In distributed systems, the cost of network communication between nodes is substantial, and full updates undoubtedly increase network load and reduce system efficiency. Furthermore, CRAFT retains the drawbacks of full replication in its log replication method. For example, when the number of healthy servers is greater than F but less than F+k (where F is the number of fault-tolerant nodes and k is the number of additional replication nodes), the system's leader node still replicates log entries using full entry replication, which is no different from the original Raft protocol. This replication method not only increases storage costs but also increases network transmission complexity, further limiting the system's scalability and performance.

[0041] In summary, the existing solutions have many shortcomings in log commit and data update, especially in handling write amplification, reducing network overhead, and optimizing log replication methods, which urgently need improvement.

[0042] In view of this, embodiments of this application provide a log commit and data update method, electronic device, and computer-readable storage medium based on erasure coding. This method can be applied to the leader node of a server cluster. First, it receives data shards to be written. Then, it obtains the log data of these data shards, which records detailed update information for each data shard. Next, it confirms the node to which the target shard belongs for each data shard and sends the log data to the corresponding node. Afterward, it receives confirmation information from the node to verify that the log data has been correctly received. Finally, it incrementally updates the target shard of the node based on this log data. Embodiments of this application employ an incremental update strategy, that is, during data writing, only the changed parts are updated, rather than the entire data shard. This strategy greatly reduces the number of disk I / O operations, effectively alleviating the write amplification problem, thereby improving the overall storage performance and response speed of the system. Furthermore, since incremental updates only transmit the changed log data, compared to full updates which require transmitting the entire data shard, network communication volume is significantly reduced. This not only reduces network load but also further improves the efficiency of data updates.

[0043] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0044] Reference Figure 4 , Figure 4 This is a flowchart of a log commit and data update method based on erasure coding provided in an embodiment of this application. This method can be applied to the leader node in a server cluster, and the process may include, but is not limited to, steps 410 to 450.

[0045] Step 410: Receive the data fragments to be written;

[0046] Step 420: Obtain the log data of the data shards. The log data contains the update information of the data shards.

[0047] Step 430: Confirm the node to which the target shard belongs, and send the log data of the data shard to the node;

[0048] Step 440: Receive confirmation information from the node, which indicates that the log data has been correctly received by the node;

[0049] Step 450: Based on the log data, incrementally update the target shards of the node.

[0050] In a feasible embodiment, in a distributed storage system, nodes are typically servers within a server cluster. These servers act as storage nodes, responsible for storing and processing data. The traditional Raft algorithm requires server nodes to adhere to the "majority" principle to ensure system reliability and fault tolerance. Specifically, in a system containing N = 2F+1 nodes, the system can tolerate the failure of F nodes as long as the remaining F+1 nodes remain operational, allowing the entire system to continue running. However, the log commit and data update method proposed in this application employs a different strategy. In this method, instead of requiring each server to have a complete state machine, a state machine is formed by at least k servers. The state machine represents all the states a system or object can experience during its lifecycle and the rules governing the transitions between these states. In the Raft consensus algorithm, the leader node in the server cluster can be considered part of a state machine implemented by multiple servers. It is responsible for coordinating and managing other nodes in the cluster (e.g., follower nodes and check nodes) to ensure that the entire cluster can reach consensus on a value or operation. Based on this, to maintain the normal operation of the system, the number of healthy nodes cannot be less than k, therefore the system can only tolerate m replica failures. In this case, the "majority" principle in the traditional Raft algorithm no longer applies. The quorum in the original Raft algorithm is defined as F+1, that is, more than half of the total number of nodes. However, in this embodiment, a new quorum rule needs to be satisfied, namely quorum = max((1+N) / 2, k), where N = k+m represents the total number of nodes in the system. Since in practical applications, k (the minimum number of nodes to form a state machine) is usually greater than m (the number of faulty nodes that can be tolerated), the quorum can be simplified to k. This preserves the high fault tolerance of the distributed system while improving the system's flexibility and scalability by having multiple servers jointly form the state machine.

[0051] In a feasible embodiment, in a distributed storage system, a data shard to be written refers to a large dataset divided into multiple smaller, independent, and manageable data blocks according to a certain strategy or rule. These data shards are the basic building blocks of a distributed storage system, and they can be distributed and stored across multiple nodes (such as servers) in the system. These shards can form a logical stripe, where each shard can store a portion of the data. It is worth noting that the data sharding process typically involves determining the shard size, selecting a sharding key (used to determine how data is distributed among the shards), and designing a sharding strategy (such as range sharding, hash sharding, consistent hash sharding, etc.). Through reasonable data sharding, a distributed storage system can efficiently manage large datasets and provide high-performance data access and processing capabilities. Suppose there is a complete dataset of 16KB in size, which can be divided into multiple 4KB shards. According to this rule, the 16KB dataset can be evenly divided into four shards. Subsequently, these four shards can be flexibly distributed across different nodes in a server cluster to achieve distributed data storage.

[0052] In a feasible embodiment, the log data for the data shard to be written refers to logs that record information related to the data shard writing process, status, and results. Specifically, the log data typically includes the following information: Timestamp: Records the time the log was generated, allowing for subsequent investigation and analysis based on chronological order. Operation Type: Identifies the type of operation recorded in the log, such as data writing, reading, or deletion. Data Shard Information: Includes key information such as shard identifier, size, and storage location to pinpoint the specific data shard. Operation Result: Records whether the operation was successful and the reason for failure, facilitating rapid problem identification. System Status: May include system load and resource usage, aiding in system performance analysis.

[0053] In a feasible embodiment, obtaining log data for the data shards to be written can rely on methods such as the log system of the distributed storage system, monitoring tools, direct access to log files, or log subscription services. By properly configuring and using these methods, log data can be effectively collected, processed, and stored. Specifically: Log system: Distributed storage systems typically integrate log systems, such as Flume and Logstash, for collecting, processing, and storing log data. These log systems can be configured to capture and store log information during the data shard writing process. Monitoring tools: Monitoring tools (such as Prometheus and Grafana) can be deployed to monitor the system's operational status and log data in real time. These monitoring tools can be configured to capture specific log information and provide a visual query and analysis interface. Direct access to log files: If the distributed storage system allows direct access to log files, they can be downloaded and viewed through the file system or network protocols (such as FTP and SFTP). However, this method requires certain system permissions and log parsing capabilities. Log subscription services: Some distributed storage systems provide log subscription services, allowing users to subscribe to specific log topics or keywords and receive relevant log information in real time. This method is typically used for real-time monitoring and alerting.

[0054] In a feasible implementation, in a server cluster, nodes can be divided into data nodes and verification nodes. Data nodes are responsible for storing and processing the actual data. They directly participate in data read and write operations and are the core of data storage and processing. In data sharding scenarios, data shards are typically distributed across different data nodes for storage to achieve distributed storage and parallel processing. Verification nodes are primarily responsible for data verification and integrity checks. They do not directly participate in data storage and processing but verify the correctness of data by calculating checksums (such as hash values). Verification nodes play a crucial role in data replication, migration, and recovery, ensuring data integrity and consistency. It's important to note that not all server clusters explicitly distinguish between the roles of data nodes and verification nodes. In some cluster architectures, nodes may simultaneously perform both data storage and verification functions.

[0055] In a feasible embodiment, the node to which the target shard belongs may include a data node and a verification node. The target shard may include a target data shard and a target verification shard, wherein the target verification shard may be stored on the verification node, and the target data shard may be stored on the data node. Specifically, each data node may store data from one or more shards. When data needs to be read or written, the data request can be routed to the corresponding data node according to the sharding rules. The verification node may store verification information for each data shard, such as a hash value or checksum. When it is necessary to verify the integrity of the data, the verification information of the verification node can be read and compared with the actual data to ensure that the data has not been damaged or tampered with during transmission or storage.

[0056] In a feasible embodiment, when confirming the node to which the target shard belongs, methods such as shard mapping table lookup, hash algorithm location, metadata service query, and inter-node communication protocols can be used. Specifically, for the shard mapping table lookup method, the system typically maintains a detailed shard mapping table that records the mapping relationship between each data shard and its corresponding data node. By querying this mapping table, the node containing the target shard can be quickly located. For the hash algorithm location method, a hash value is obtained by hashing the key information of the data shard (such as the shard ID), which can directly or indirectly indicate the location of the node containing the data shard. For the metadata service query method, the metadata service is a component in the distributed system responsible for managing and maintaining data metadata. By sending a query request to the metadata service, detailed information about the data shard and its associated node can be obtained. Furthermore, within the distributed system, nodes typically interact through specific communication protocols. These protocols may contain relevant information or instructions for locating the node to which a data shard belongs.

[0057] In one feasible embodiment, after confirming the node to which the target shard belongs, the log data of that shard can be sent to these confirmed nodes. Once the nodes receive this log data, they immediately initiate a verification and parsing process. The verification phase covers data integrity verification, correctness verification, and source reliability assessment. This process ensures that the received log data is accurate and from a reliable source. The parsing phase is responsible for converting the log data into a format that the system can recognize and properly storing it in the appropriate storage medium. The successful completion of this step signifies that the log data has been successfully transformed into a usable data resource within the system. After the log data is correctly received and properly stored by the nodes, the nodes will send back confirmation information to the sender (which may be other nodes in the system). This confirmation information records key information such as the log data's reception status and storage location, providing the sender with clear reception confirmation and storage details.

[0058] In a feasible implementation, in a scenario where data sharding updates are optimized, log data can be sent only to m verification nodes and 1 data node to be updated (a total of m+1 nodes) to reduce data transmission and storage overhead. Meanwhile, to maintain the cluster's synchronization, other nodes still communicate via heartbeat messages from the Raft algorithm. Figure 5 As shown, the system includes data nodes D0, D1, D2, and D3, as well as verification nodes P0 and P1. Data node D0 is the leader node in the cluster. Figure 5 In this process, log data d0v1 can be sent to data node D0, check node P0, and check node P1; log data d2v1 can be sent to data node D2, check node P0, and check node P1; log data d3v1 can be sent to data node D3, check node P0, and check node P1; and log data d0v2 can be sent to data node D0, check node P0, and check node P1.

[0059] In a feasible embodiment, in a distributed system, the data nodes in the cluster include a leader node (also called a master node) and follower nodes (also called slave nodes). Specifically, the leader node is the core node in the cluster, primarily responsible for managing and coordinating other nodes in the cluster. Its main responsibilities include at least: Transaction processing: The leader node is typically responsible for handling transactional operation requests in the cluster, such as adding, deleting, and modifying data. These operations need to ensure data consistency and integrity. Decision making: In the cluster, the leader node is typically responsible for decision making, such as selecting a new leader node (when the leader node fails) and assigning tasks to follower nodes. Entry point: The leader node typically acts as the entry point of the cluster, receiving requests from clients and distributing these requests to other nodes in the cluster for processing. Follower nodes are auxiliary nodes in the cluster; they follow the instructions of the leader node and execute corresponding tasks. Their main responsibilities include at least: Non-transactional operations: Follower nodes are typically responsible for handling non-transactional operation requests in the cluster, such as data queries. These operations typically do not require ensuring data consistency and integrity but require fast response. Voting election: When the leader node fails or a new leader node needs to be elected, follower nodes participate in the voting process to determine the new leader node. Data storage and backup: Follower nodes typically store copies of the data to ensure high availability and fault tolerance. In the event of a leader node failure, follower nodes can take over some or all of the leader node's responsibilities to ensure the cluster's continued operation.

[0060] In one feasible implementation, to ensure the continuity of the Raft log while effectively reducing write amplification during log writes to disk and decreasing the amount of Raft log data transmitted over the network, the leader node can retain all log records in its memory to ensure log integrity and consistency. However, for efficiency and storage optimization, the leader node can only store logs for data shards directly related to the current task on its disk. This ensures the immediate availability of the logs while reducing the storage burden on the disk. For follower nodes, whether in memory or on disk, only logs directly related to the data shards they are currently responsible for can be stored. This targeted log storage method not only reduces unnecessary log storage overhead but also improves data access efficiency.

[0061] In one feasible embodiment, when the node to which the target shard belongs is the leader node, after the leader node obtains the log data of the data shard, it can generate a log sequence and log records based on the log data. The log sequence is a merged version designed to optimize storage and improve data access efficiency. Subsequently, the leader node temporarily stores the complete log sequence in its memory for fast access and processing. Simultaneously, for persistent data storage, the leader node also writes the merged log records to its disk, ensuring that the necessary log information can be recovered even in the event of a system restart or failure.

[0062] In one feasible embodiment, when the target shard corresponding to the data shard is located on the leader node, i.e., when the data shard on the leader needs to be updated, reference can be made to... Figure 6 Let's examine the specific logs of the Leader (let's assume it's D0) in memory and on disk. Figure 6In the scenario shown, the leader node adopts different processing strategies as logs are generated. Specifically, when the log with index=1 is created, the Raft protocol triggers a write-to-disk operation to ensure its persistence on disk. However, for the subsequently generated logs with index=2, index=3, and index=4, the Raft protocol does not immediately trigger their write-to-disk operation. This strategy aims to reduce the frequency of disk write operations, thereby improving system performance. It is worth noting that when the log with index=5 is created, the Raft protocol triggers a write-to-disk operation again, but this time it is not only for the log with index=5. To optimize storage and improve efficiency, the system merges the logs with index=2, index=3, and index=4 with the log with index=5, and writes the merged result (5(2,5)) to disk. It should be noted that the "merge" operation here is not simply concatenating the log content, but rather using a more efficient method: only the starting index of each merged log segment is retained, while the specific log data content is omitted (because this data may already be redundant in the merged log segment, or can be reconstructed in other ways).

[0063] In one feasible embodiment, when the node to which the target shard belongs is a follower node, the log records of the leader node can be transmitted to the follower node through the leader node's log recording synchronization mechanism. Subsequently, the follower node can perform recovery and merging processing on these received log records, a process aimed at reconstructing the original log data. After completing this step, the follower node can simultaneously store this reconstructed original log data in its disk storage and memory space.

[0064] In a feasible embodiment, when the target shard corresponding to a data shard is located on a follower node, i.e., when the data shard on the follower needs to be updated, this update process can be implemented through the leader node (e.g., D0). Specifically, the leader synchronizes the logs of the follower (e.g., D1) using its in-memory log. It is important to note that this synchronization operation does not transmit the entire content of the data shard, but only synchronizes the shard's metadata to the target node. Once synchronization is complete, this operation is considered a commit, meaning that the update operation has been confirmed by the leader and is ready to be submitted to the follower; at this point, the operation is considered a commit. Figure 7The process of synchronizing D0 with D1 is described in detail: During the log generation phases of index=1, index=2, and index=3, the Raft protocol does not immediately trigger the synchronization operation. The synchronization process for index=1, index=2, index=3, and index=4 is only triggered when the log for index=4 is generated. To improve synchronization efficiency, the logs of index=1, index=2, and index=3 can be merged into the log of index=4, forming a merged log packet for transmission. When Follower (D1) receives this merged log (marked as 4(1,4), where (1,4) indicates that the log packet contains the log content from index=1 to index=4), it first restores it to its original form before merging and then hands it over to the Raft protocol for further processing. Regarding log writing to disk, the Follower's operation is similar to the Leader's. The Follower merges the data in its memory into a single log entry (4[1,4] in this example) and then writes it to disk. This consolidation strategy helps reduce the number of disk write operations, improve storage efficiency, and ensure data integrity and consistency.

[0065] In one feasible embodiment, when the gap between the current log data (i.e., the stored raw log data) of a follower node and the log sequence of the leader node accumulates to a preset threshold, a complete copy of the leader node's log sequence can be sent to the follower node. This complete copy covers all log entries in the log sequence, from beginning to end, ensuring that the follower node obtains comprehensive log information. By sending the complete copy, the follower node can quickly update its log status, narrowing the log gap with the leader node, thereby ensuring high availability and data accuracy of the entire system.

[0066] It's important to note that the Leader is primarily responsible for coordinating and managing the log replication process within the cluster. However, as the Leader's in-memory log grows over time, the log gap between it and the Followers (i.e., the following nodes) can gradually widen. If the log gap between the Leader and Followers becomes too large, a so-called "gap" phenomenon can occur, where Follower nodes cannot catch up with the Leader's log progress through regular log replication operations. This gap can compromise the consistency and reliability of the cluster. To avoid this, when the log gap reaches a threshold, the Leader can trigger a no-op log. This no-op log itself does not contain any actual data updates or operation instructions; its main function is to serve as a marker or signal to trigger the subsequent snapshot sending process. A snapshot is a common data backup and recovery mechanism in distributed systems. It allows the system to capture the state of the entire cluster at a specific moment and save this state as a snapshot file. This snapshot file can then be used to restore the cluster to its state at that specific point in time. After triggering the no-op logging, the Leader can begin preparing snapshots and send the snapshot data to all lagging Followers. Upon receiving the snapshots, Followers can apply them to their own logs, quickly narrowing the log gap with the Leader. In this way, even if previous gaps existed, consistency and reliability can be restored through the application of snapshots. Figure 8 As shown, when the log discrepancy between the leader node D0 and the follower nodes D1, D2, and D3 exceeds a preset threshold, the leader node D0 can take the following measures to synchronize the log state: First, D0 can trigger a no-op log entry. Then, D0 generates a snapshot of the current system, containing enough information to restore the system to the state at the time the snapshot was triggered. This snapshot data is sent to the lagging follower nodes D1, D2, and D3. Upon receiving the snapshot data, these follower nodes apply it to their own log sequences to quickly update to a state consistent with the leader node D0. Once the follower nodes have successfully applied the snapshot data and updated their logs, they can also choose to synchronize these updated log entries or snapshot information to the check nodes P0 and P1.

[0067] like Figure 9 As shown, the specific process of incrementally updating the target shard of the node based on the log data in step 450 may include at least steps 910 to 930.

[0068] Step 910: Obtain the target data shard and target verification shard corresponding to the data shard;

[0069] Step 920: Based on the target data shards, target verification shards, and log data, incrementally update the target verification shards of the verification node;

[0070] Step 930: Incrementally update the target data shards of the data nodes based on the log data.

[0071] In a feasible embodiment, in step 910, the data shard to be updated (target data shard) and its corresponding verification shard (target verification shard) can be located and read. These shards are typically stored on different nodes of a distributed storage system, with the target data shard stored on a data node and the target verification shard stored on a verification node. For example, to obtain these shards, the metadata or index of the distributed storage system can be queried to determine the specific locations of the target data shard and the target verification shard, and read requests can be sent to the corresponding data node and verification node to obtain the data of these shards.

[0072] In a feasible embodiment, in step 920, a new verification shard value can be calculated using the target data shard, the target verification shard, and the log data. This new verification shard value is then written back to the verification node to replace the original verification shard. For example... Figure 10 As shown, if only the target data shard D0 needs to be updated, then it is not necessary to read all data shards (D0 to D...). k-1 Only data fragment D0 and its associated target verification fragments (let's assume they are P0 to P1) need to be read. r-1 To perform incremental updates on the target verification shard, the data shard D0 and its updated state (D) can be used. ′ 0 (This status can be obtained from log data) is sent to the node where the verification shard resides. The specific calculation formula is P. j ′ =α j,i (D i ′ -D i )+P j , where α j,i (This is a specific function related to data sharding and parity sharding, P) j This is the parity fragment value before the update. For example, for parity fragment P0, the new parity fragment value is calculated using the following formula: P0 ′ =α j,0 (D ′The calculation of (0-D0)+P0, followed by writing the calculated new parity fragment value back to the parity node, completes the update of that parity fragment. Similarly, for other parity fragments, after calculating their corresponding parity fragment values ​​using this formula, the new parity fragment value replaces the original parity fragment value, thus completing the update. During this process, it is not necessary to read or transmit all data fragments to update a specific parity fragment. Only when a data fragment changes does it need to send its old and new state information to the relevant parity node. This method significantly reduces data transmission volume, thereby improving the efficiency of incremental updates.

[0073] In one feasible embodiment, in step 930, the target data shard can be updated directly based on the log data. This process may include: parsing the log data to determine the changes made to the data shard (e.g., insertions, deletions, modifications, etc.); applying these changes to the target data shard (in the local storage or memory of the data node); and further, writing the updated data shard back to persistent storage on the data node.

[0074] In one feasible embodiment, the entire incremental update process from steps 910 to 930 is designed to optimize storage system performance by updating only the changed data shards and checksum shards. This reduces unnecessary I / O operations and network transfers, and lowers latency and resource consumption during the update process.

[0075] In a feasible embodiment, the normal write (or update) process for data shards can be divided into two ordered phases: In the first phase, the Raft log mechanism ensures that the data shard content to be written or updated is safely saved to m+1 replicas. These replicas are located on the original node where the target shard (including the target data shard and the target parity shard) resides, and on m parity nodes, respectively. Once all data has been successfully written and confirmed to be error-free, the system returns a success signal. Next, the second phase, the asynchronous write state machine phase, begins. In this phase, an incremental update of the target parity shard is performed first. This process aims to update the data parity value on the parity nodes by transmitting only the necessary data changes (rather than the entire data shard). The target data shard itself is then updated to ensure data consistency and integrity in case the parity shard update fails and the original data shard cannot be recovered. Figure 11As shown, the system includes a leader node D0, follower nodes D1, D2, and D3, and check nodes P0 and P1. Normal data shard writing involves two phases: Phase 1 (the content within the first dashed box) involves a client writing a data shard (d1v1) to the leader node D0. The leader node D0 then sends this data shard to the node containing its corresponding target shard in Raft log format. In this example, we assume follower node D1 is the data node containing the target data shard corresponding to d1v1, and check nodes P0 and P1 are the check nodes containing the target check shard corresponding to d1v1. After sending the log data of d1v1 to follower nodes D1, check nodes P0, and check nodes P1, receiving confirmation from these nodes indicates successful reception of the log data, confirming successful data transmission. It's worth noting that during this process, D1 can additionally return the old shard data (i.e., the target data shard, assuming it's d1v0) at the corresponding position of d1v1 to the leader node D0. The second phase (the content within the second dashed box) then begins. The leader node D0 first sends the old shard data d1v0 to the verification nodes P0 and P1. Upon receiving the data from the leader node D0, verification nodes P0 and P1 verify and store it. If data verification is successful, the verification nodes return an acknowledgment message to the leader node D0, indicating that the data has been successfully received and stored (including updating the target verification shard for their respective nodes). Next, the leader node D0 can send an apply request to the follower node D1. This request contains the log data of the old shard data d1v0 and related metadata (such as terms of office). The purpose of the apply request is to instruct follower node D1 to apply it to its local state machine to ensure data consistency. Upon receiving the apply request from the leader node D0, follower node D1 verifies the validity of the request (e.g., checking if terms of office match). If verification is successful, follower node D1 applies the data items in the request to its local state machine (including updating the target data shard for its respective node). After applying the data item, follower node D1 can return a success message to leader node D0, indicating that the data item has been successfully applied. Upon receiving the success message from follower node D1, leader node D0 can confirm that the data item has been successfully applied to the state machines of all relevant nodes. At this point, leader node D0 can return a write success message to the client.

[0076] In one feasible embodiment, log data of the data shard can be sent to candidate nodes in the cluster for storage. Upon receiving a data reconstruction instruction, the log data is read from the candidate nodes, and the target shard of the failed node is reconstructed based on the log data. It is worth noting that when a data shard is written to the target shard, the system simultaneously generates corresponding log data. This log data records changes to the data shard, such as insertion, update, or deletion operations. To ensure the security and availability of the log data, a set of reliable candidate nodes can be selected to store this log data. These candidate nodes are typically nodes with good performance and high stability in the cluster. When the node containing the target shard fails, the log data related to that shard can be sent to the selected candidate nodes for storage. When it is detected that the target shard of the failed node needs to be reconstructed, a data reconstruction instruction can be sent to the candidate nodes. Upon receiving the data reconstruction instruction, the candidate nodes can read the log data stored on them. This log data contains all changes to the target shard of the failed node before the failure. Based on the read log data, these change operations can be re-executed on the candidate nodes or the new target shard to recover the data of the target shard of the failed node. Through the above process, when the node containing the target shard corresponding to a data shard fails, the log data stored on the candidate node can be used to reconstruct the target shard of the failed node, thereby ensuring data integrity and service continuity. Figure 12The flowchart shown represents the operation of a faulty node corresponding to a data shard. The specific process includes: In the first stage (the content within the first dashed box), the Client can write a data shard (d1v1) to the leader node D0. Subsequently, the leader node D0 can send this data shard to its corresponding target shard's follower node D1, check node P0, and check node P1 in Raft log format. Once confirmation information is received from these nodes, it indicates that the log data has been successfully received, confirming successful data transmission. During this process, assuming follower node D1 fails, the d1v1 log data needs to be saved first. Then, the second stage (the content within the second dashed box) begins. In this stage, the erasure coding-specific recovery algorithm can be used to recover the old data d1v0 (i.e., the target data shard corresponding to d1v1) on the faulty node (D1) from data from any k nodes (replicas). The old data is then sent to the check nodes (P0 and P1). After successfully receiving and verifying the data, check nodes P0 and P1 can send an confirmation message to the leader node D0, indicating success. After receiving an acknowledgment message from the validator node, the leader node D0 can send an Apply request to the follower node D1 (which has now recovered the data). The Apply request includes the data item to be applied (i.e., the log item of the recovered old data d1v0) and related metadata. After receiving the Apply request and verifying its success, the follower node D1 applies the requested data item to its local state machine. After applying the data item, the follower node D1 returns a success message to the leader node D0. Upon receiving the success message from the follower node, the leader node D0 confirms that the data item has been successfully applied to the state machines of all relevant nodes. At this point, the leader node D0 can return a write success message to the client.

[0077] See Figure 13 , Figure 13 This is a flowchart of a log commit and data update method based on erasure coding, provided in another embodiment of this application. This method can be applied to the leader node in a server cluster, and the process may include, but is not limited to, steps 1310 to 1350.

[0078] Step 1310: Receive the full stripe data to be written;

[0079] Step 1320: Set the update threshold based on the number of nodes in the server cluster;

[0080] Step 1330: Based on the full stripe data, obtain the data fragments and check fragments;

[0081] Step 1340: Confirm the target data node corresponding to the data shard and the target verification node corresponding to the verification shard, send the data shard to the target data node, and send the verification shard to the target verification node;

[0082] Step 1350: Determine the number of nodes based on the confirmation information sent by the target data node and the target verification node. When the number of nodes equals the update threshold, confirm the update operation for the target data node and the target verification node.

[0083] It's important to note that striping in a disk array involves dividing the storage space of multiple disks into identically sized storage units according to certain rules, and then combining these units into a logical storage space. These storage units are called stripe cells or blocks. Full striped data refers to data being written to the disk array according to the striping rules, distributed across multiple disk stripe cells. A full stripe is formed when a logical storage space (i.e., a stripe) is completely filled with data. In distributed storage systems, full striped data can be used as part of a data distribution strategy. By distributing data across multiple nodes (or disks), load balancing and fault tolerance can be achieved.

[0084] In one feasible embodiment, the leader node can receive full-strip data from clients or other sources and perform preliminary data verification and integrity checks.

[0085] In one feasible embodiment, the leader node dynamically calculates and sets an update threshold based on the current number of nodes in the server cluster (including data nodes and check nodes) and a preset fault tolerance strategy. The update threshold is typically set as a proportion of the total number of nodes in the cluster to ensure data recovery even if some nodes fail.

[0086] In a feasible embodiment, in the process of obtaining data shards and verification shards based on full-strip data, the full-strip data can first be split into multiple data shards; then, a preset verification algorithm is used to calculate the verification values ​​of the multiple data shards; finally, verification shards can be obtained based on the verification values. It can be understood that data sharding involves dividing full-strip data according to certain rules or sizes to obtain multiple smaller data blocks. For example, the specific steps are as follows: Determine the rules or sizes for data splitting based on actual needs. For example, it can be split according to a fixed size (e.g., 300 data points per block), or according to the logical structure of the data. Use a programming language or data processing tool to split the full-strip data according to the determined splitting rules. For example, in Python, list slicing or array slicing functionality in the NumPy library can be used to implement data splitting. Verification sharding involves calculating the verification values ​​of the data shards according to a preset verification algorithm, and generating verification shards based on the verification values. For example, the specific steps are as follows: Select a suitable verification algorithm based on actual needs and data characteristics. Common verification algorithms include parity check, BCC (Exclusive Code Check), and CRC (Cyclic Redundancy Check). Then, using the selected verification algorithm, a checksum is calculated for each data fragment to obtain the corresponding check value. Next, the calculated checksum is used as part of the check fragment, or the checksum is combined with other relevant information (such as the data fragment's identifier and order) to generate a complete check fragment.

[0087] In one feasible embodiment, data shards can be sent to target data nodes, while parity shards can be sent to target parity nodes. Upon receiving the corresponding data or parity shards, these nodes can store them locally and simultaneously record shard logs for these shards. Shard logs typically contain shard information (such as shard ID, size, storage location, etc.) and related metadata (such as timestamps, version numbers, etc.) to ensure data traceability and consistency. After completing the storage operation, the target data nodes and target parity nodes can send corresponding confirmation information to the leader node. This confirmation information typically includes detailed information about the successfully stored shards and node identification information, enabling the leader node to accurately track and verify the storage status of each node. After receiving the confirmation information from the target data nodes and target parity nodes, the leader node can count the number of nodes that have confirmed the storage of data and parity shards. This number is compared with a preset update threshold to determine whether the data update conditions are met. The update threshold is typically set based on the system's fault tolerance and consistency requirements; it represents the minimum number of nodes required to successfully store data shards. When the number of confirmed nodes equals the update threshold, the leader node can further confirm that the update operations for the target data node and the target verification node have been completed. This means that the data has been successfully distributed and stored across multiple nodes. It should be noted that, for log synchronization, the leader node can also simultaneously store the logs of each data shard in its memory.

[0088] See Figure 14 , Figure 14This is a schematic diagram illustrating the writing process of full-striped data, provided as a specific example of this application. The specific process can include: First stage (the content within the first dashed box): The Client can send a full-striped data (data) to the leader node D0. After receiving the data and completing data integrity verification, the leader node D0 can split the data into multiple data fragments and calculate a check fragment based on the data fragments. Subsequently, the leader node D0 can synchronize the data fragments and check fragments to all replica nodes, including follower nodes D1, D2, and D3, as well as check nodes P0 and P1. Once all replica nodes return confirmation information, indicating that the data has been successfully received and stored, the leader node D0 can confirm that the data transmission was successful. Then, the second stage (the content within the second dashed box): After confirming successful data transmission, the leader node D0 can send an apply command to all replica nodes, notifying them to apply the data fragments and check fragments to their respective storage media. Upon receiving the apply command, each replica node checks whether the data fragments and check fragments stored locally are complete and applies them to its local storage. Once the data has been successfully applied to the storage media of all replica nodes, they can send an acknowledgment message to the leader node D0. After receiving acknowledgments from all replica nodes, the leader node D0 considers the write I / O operation to be successfully completed. At this point, the leader node D0 can send an acknowledgment message to the client, informing it that the data has been successfully written to the storage system.

[0089] See Figure 15 , Figure 15This is another specific example of the full-strip data writing process provided in this application. The specific process can include: First stage (the content within the first dashed box): The Client can write a full-strip data to the leader node D0. After receiving the data, the leader node D0 can split the data into multiple data fragments and calculate and generate a parity fragment. Then, it can synchronize the data fragments and parity fragments to each replica, including follower nodes D1, D2, and D3 (assuming D1 is a faulty node, but not yet identified at this time), and parity nodes P0 and P1. During the data synchronization process, the leader node D0 identifies a fault in follower node D1. At this point, to ensure data reliability, the leader node D0 can first save a log of the current data for data recovery in a later stage. Then, the process proceeds to the second stage (the content within the second dashed box). In this stage, the leader node D0 can notify the remaining normal replicas (D2, D3, P0, and P1) to apply the received data fragments and parity fragments. The normal replicas confirm receipt and application of the fragments, replying to the leader node D0 with a confirmation message. Subsequently, the leader node D0 can use the stored logs and checksum shards to prepare for the recovery of data from the failed node D1. For example, it can replicate data shards from other healthy replicas or reconstruct the lost data using checksum shards. Once all healthy replicas have confirmed that the data has been correctly applied and that the processing of the failed node is ready or in progress, the leader node D0 can send a confirmation message to the client confirming the successful write operation.

[0090] Reference Figure 16 This application also discloses an electronic device, the electronic device 1600 comprising:

[0091] At least one processor 1610;

[0092] At least one memory 1620 is used to store at least one program;

[0093] When at least one program is executed by at least one processor 1610, the erasure coding-based log commit and data update methods described above are implemented.

[0094] This application also discloses a computer-readable storage medium storing a processor-executable computer program, which, when executed by a processor, is used to implement the erasure coding-based log submission and data update method described above.

[0095] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A log commit and data update method based on erasure coding, characterized in that, The leader node used in a server cluster includes: Receive data fragments to be written; Obtain the log data of the data shard, wherein the log data contains the update information of the data shard; Identify the node to which the target shard corresponding to the data shard belongs, and send the log data of the data shard to the node. The node includes a data node and a verification node. The target shard includes a target data shard stored in the data node and a target verification shard stored in the verification node. Receive confirmation information from the node, the confirmation information being used to indicate that the log data was correctly received by the node; Based on the log data, the target shard of the node is incrementally updated; Log sequences and log records are generated based on the log data, wherein the log records are a merged form of the log sequences, the log sequences are stored in memory, and the log records are written to disk; The data nodes include the leader node and follower nodes. If the node to which the target data shard belongs is the follower node, the log records are synchronized to the follower node so that the follower node can restore and merge the received log records into the original log data, and store the original log data in the disk and memory of the follower node. When the difference between the original log data of the follower node and the log sequence reaches a preset threshold, a complete copy of the log sequence is sent to the follower node. The complete copy includes all log entries of the log sequence. The step of incrementally updating the target shard of the node based on the log data includes: Using the target data shards, the target verification shards, and the log data, through the formula = ( - )+ Calculate the new parity fragment value, where, For specific functions related to data sharding and parity sharding, The updated state of the target data fragment. The target data fragment is in its state before being updated. The new value of the target verification shard is set as the value before the update; the new value of the verification shard is written back to the verification node to replace the original value of the target verification shard; the target data shard of the data node is incrementally updated according to the log data.

2. The log commit and data update method based on erasure coding according to claim 1, characterized in that, The incremental update of the target shard of the node based on the log data includes: Obtain the target data fragment and target verification fragment corresponding to the data fragment; Based on the target data shard, the target verification shard, and the log data, the target verification shard of the verification node is incrementally updated. Incremental updates are performed on the target data shards of the data nodes based on the log data.

3. The log commit and data update method based on erasure coding according to claim 1, characterized in that, The log data includes multiple log fragments, and the step of generating log sequences and log records based on the log data includes: A log sequence is generated based on multiple log data fragments, and the log sequence includes multiple log entries; Multiple log entries are merged to obtain a log record.

4. The log commit and data update method based on erasure coding according to claim 1, characterized in that, The method further includes: When the node is a faulty node, the log data is sent to a candidate node for storage. After receiving the data reconstruction instruction, the log data is read from the candidate node, and the target fragment of the faulty node is reconstructed based on the log data.

5. The log commit and data update method based on erasure coding according to claim 1, characterized in that, The method further includes: Receive full-strip data to be written; Set the update threshold based on the number of nodes in the server cluster; Based on the full stripe data, data fragments and verification fragments are obtained; Confirm the target data node corresponding to the data shard and the target verification node corresponding to the verification shard, send the data shard to the target data node, and send the verification shard to the target verification node; The number of nodes is determined based on the confirmation information sent by the target data node and the target verification node. When the number of nodes is equal to the update threshold, the update operation for the target data node and the target verification node is confirmed.

6. The log commit and data update method based on erasure coding according to claim 5, characterized in that, The step of obtaining data fragments and verification fragments based on the full stripe data includes: The full stripe data is split into multiple data fragments; The verification value is obtained by calculating the verification value of multiple data fragments according to the preset verification algorithm; The verification fragment is obtained based on the verification value.

7. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; The erasure coding-based log commit and data update method as described in any one of claims 1 to 6 is implemented when at least one of the programs is executed by at least one of the processors.

8. A computer-readable storage medium, characterized in that, It stores a processor-executable computer program, which, when executed by the processor, is used to implement the erasure coding-based log commit and data update method as described in any one of claims 1 to 6.

Images