Data backup method, electronic device, storage medium, and program product

Abstract

The application discloses a data backup method, an electronic device, a storage medium and a program product, relates to the technical field of data processing, and comprises the following steps: when an exception is handled on a first node, determining access information of a data block of a first non-main copy, and creating a first shadow copy in a third node based on the access information, so that the purpose of creating the first shadow copy in the third node on demand is achieved, thereby reducing resource consumption for creating the copy; and the first shadow copy only performs incremental backup on changed data in the first non-main copy, thereby reducing resource consumption for data transmission, and in the case that the first node and the second node may be abnormal at the same time, emergency redundancy can be provided through the first shadow copy, and the problem that data is completely unavailable due to simultaneous failure of double copies is avoided. Therefore, the technical problem of high data backup resource consumption in the prior art can be solved, and the technical effect of reducing data backup resource consumption is achieved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to data backup methods, electronic devices, storage media, and program products. Background Technology

[0002] Distributed storage systems ensure high data availability through multi-replica technology. In a two-replica architecture, during rolling upgrades, if upgrading one node renders its data replicas unavailable, and the corresponding replica's node also fails, the associated data will be completely unavailable, leading to service interruption. Related technologies address this by increasing the number of data replicas before upgrading nodes to create data redundancy protection, but this significantly increases storage and network bandwidth costs. Summary of the Invention

[0003] This application provides data backup methods, electronic devices, storage media, and program products to at least solve the technical problem of high data backup resource consumption in related technologies.

[0004] This application provides a data backup method, comprising: during the process of initiating anomaly handling of a first node, determining access information of data blocks in a first non-primary replica, wherein the first non-primary replica is set in a second node, and the data blocks in the first non-primary replica are backups of the data blocks in the first primary replica set in the first node; creating a first shadow replica of the first non-primary replica in a third node based on the access information, wherein the data blocks in the first shadow replica are backups of the data blocks in the first non-primary replica; and using the first shadow replica to back up the data that has changed in the first non-primary replica; wherein the first node, the second node, and the third node are all nodes set in a distributed storage system.

[0005] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described data backup methods when executing the computer program.

[0006] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data backup methods.

[0007] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data backup methods.

[0008] This application addresses the issue of high data backup resource consumption in related technologies by determining access information for the first non-primary replica's data blocks during anomaly handling at the first node. Based on this access information, a first shadow replica is created in the third node, enabling on-demand creation of the first shadow replica on the third node and reducing resource consumption for replica creation. Furthermore, the first shadow replica only incrementally backs up data that has changed in the first non-primary replica, reducing data transmission resource consumption. It also provides emergency redundancy in cases where both the first and second nodes may fail simultaneously, preventing complete data unavailability due to simultaneous failure of both replicas. Therefore, this application solves the technical problem of high data backup resource consumption in related technologies, achieving the technical effect of reducing data backup resource consumption. Attached Figure Description

[0009] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a hardware structure block diagram of the data backup method according to an embodiment of this application;

[0011] Figure 2 This is a flowchart of a data backup method according to an embodiment of this application;

[0012] Figure 3 This is a flowchart illustrating the creation of a first shadow copy according to an embodiment of this application;

[0013] Figure 4 This is a flowchart illustrating the backup of modified data in the first non-master copy according to an embodiment of this application;

[0014] Figure 5 This is a flowchart illustrating the synchronization of incremental data according to an embodiment of this application;

[0015] Figure 6 This is a flowchart illustrating the adjustment of the duration of the first shadow copy according to an embodiment of this application;

[0016] Figure 7 This is a flowchart illustrating the execution data recovery process according to an embodiment of this application;

[0017] Figure 8 This is an overall flowchart of runtime maintenance for incremental synchronization and state adaptation according to an embodiment of this application;

[0018] Figure 9 This is a flowchart illustrating the workflow of each module in a distributed storage system according to an embodiment of this application.

[0019] Figure 10 This is a structural block diagram of a data backup device according to an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] The specific application environment architecture or specific hardware architecture on which the data backup method depends is described here.

[0024] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of the data backup method according to an embodiment of this application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor (MCU) or a field-programmable gate array (FPGA)) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0025] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data backup method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0026] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0027] This embodiment provides a data backup method. Figure 2 This is a flowchart of a data backup method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0028] Step S202: During the process of starting the exception handling of the first node, the access information of the data blocks in the first non-primary replica is determined, wherein the first non-primary replica is set in the second node, and the data blocks in the first non-primary replica are backups of the data blocks in the first primary replica set in the first node.

[0029] In at least one embodiment, the aforementioned first node may be, but is not limited to, a physical or logical storage node in a distributed storage system that carries the primary copy of a data block. The first node may enter a state of unavailability or limited service due to reasons such as hot upgrades, system maintenance, hardware failures, or software anomalies. For example, suppose a distributed storage cluster includes nodes A, B, C, and D. Data block File1 stores its first primary copy (i.e., primary data) on A. When the system plans to upgrade the software version of A, A is identified as the first node, and its primary copy will be temporarily disabled during the upgrade process, entering an exception handling procedure.

[0030] In at least one embodiment, the aforementioned first non-primary replica may be, but is not limited to, a replica in a distributed storage system that provides redundant backup for the first primary replica. The content of its data blocks is consistent with the first primary replica, but it is stored on a different node than the first node; this can be referred to as a second replica or a slave replica. The first non-primary replica is located on the second node, and its data originates from the first primary replica of the first node. For example, the first primary replica of data block File1 is located at node A, and its first non-primary replica is located at node B. B, as the second node, carries a backup replica of File1. When node A enters an upgrade state, the replica on node B becomes the only currently available data replica.

[0031] In at least one embodiment, the access information of the aforementioned data blocks may refer to, but is not limited to, the frequency, time distribution, and request source of each data block read or written by the distributed storage system within a preset time window for each data block in the first non-primary replica. The access information is used to identify which data blocks are hot data (e.g., real-time user session data accessed more than 500 times in the past hour) and which data blocks are cold data (e.g., historical operation logs accessed less than 10 times in the past 24 hours). For example, in the first non-primary replica of File1 hosted on B, it is detected that its 64KB data block B1 was accessed 820 times in the past hour (from the user login service), while data block B2 was only accessed 3 times (from historical archive logs). In this case, the system identifies B1 as hot data and B2 as cold data, and decides based on this whether to create a shadow replica for B1.

[0032] In at least one embodiment, the anomaly handling process refers to a series of preset response procedures initiated by the system after detecting that the first node is in an abnormal service state, including but not limited to: node isolation, service migration, replica status marking, resource locking, and upgrade initiation. This process does not refer to fault repair, but rather to the proactive management of the first node's impending unavailability. For example, when the first node A is scheduled to enter the upgrade preparation state, the cluster control center broadcasts an event that A is about to go offline to all nodes, triggering the method of this embodiment. At this time, the system initiates step S202, starting to analyze the data access information of the first non-primary replica on the second node B to determine whether to create a shadow replica.

[0033] Step S204: Based on the access information, create a first shadow copy of the first non-primary copy in the third node, wherein the data blocks in the first shadow copy are backups of the data blocks in the first non-primary copy;

[0034] In at least one embodiment, the aforementioned third node can be, but is not limited to, a third independent storage node in the distributed storage system that is neither the first node (primary replica node) nor the second node (first non-primary replica node). Its role is solely to temporarily host the first shadow replica, not to participate in the master-slave synchronization process of the original data, and only to provide emergency redundancy during anomaly handling. The third node has the following characteristics: it is in a healthy, online, and resource-sufficient state; it does not carry any original replica of the data block (i.e., it is neither the first primary replica nor the first non-primary replica); it is not included in the current upgrade plan and has no risk of impending failure. For example, the first node is A, the second node is B, and the third node can be C or D. Assuming the system detects that A will be upgraded, and B carries a replica of File1, to prevent a sudden failure of B during the upgrade, D is selected as the third node. D originally did not store any replica of File1; it is selected only for this step to host the shadow replica.

[0035] In at least one embodiment, the aforementioned first shadow copy may be, but is not limited to, a read-only / writable backup copy temporarily created for the first non-primary copy, used solely for emergency protection. Its data content is consistent with the first non-primary copy at the time of creation, but it is stored on an independent third node. Its lifecycle is bound to the exception handling process; it is not a permanent copy. That is, the first shadow copy is neither a replacement nor an enhanced copy, but rather an insurance copy for the risk window period, used to cope with extreme combinations of primary copy unavailability and non-primary copy failure. For example, after the first non-primary copy of File1 on B is identified as high-risk, the system creates a first shadow copy on D that is completely identical to it. This first shadow copy is initially a snapshot of the first non-primary copy and does not participate in daily read / write operations. It is only taken over by the system to respond to incremental data generated by client requests when A is offline or B experiences a sudden failure. The first shadow copy does not occupy the resources of B or A and exists independently.

[0036] It should be noted that this embodiment does not create shadow copies for all data blocks of the first non-primary replica. Instead, it selectively creates shadow copies for specific data blocks based on the access information obtained in step S202. The purpose is to avoid resource waste and protect only the data most likely to be affected by a failure. If a data block is shown as "hot" data in the access information, a shadow copy is created for it; if a data block is accessed very infrequently, no shadow copy is created to save resources on node D. For example, suppose File1 stored on node B contains 100 64KB data blocks. After analyzing its access information, the system finds that data blocks B1–B10 were accessed a total of 1200 times in the past hour (user order service), triggering the creation of shadow copies; blocks B11–B100 were accessed only 8 times in the past 24 hours (archived audit logs), so no shadow copies are created. Therefore, the system only creates shadow copies for B1–B10 on N3, while B11–B100 remain unchanged. This significantly reduces the storage overhead of the third node while ensuring the availability of data for critical business operations.

[0037] Optionally, the data blocks in the first shadow copy are backups of the data blocks in the first non-primary copy, referring to a copy of the data content, not a synchronous channel or real-time replication. When the first shadow copy is created, its content is completely identical to the first non-primary copy, but subsequent real-time synchronization is not guaranteed. This embodiment can obtain a static data image of the first non-primary copy at a certain point in time through a snapshot; or copy the specified data blocks from the second node to the third node through a full copy. It should be noted that this process does not block the main service and can be completed asynchronously in the background. For example, when the system decides to create shadow copies for B1–B10, the control node sends a snapshot request to B. B exports the data content of B1–B10 at timestamp T0 as a temporary image and transmits it to D through the internal network. After receiving it, D writes it to local storage, generating the first shadow copy, whose content is completely identical to B at time T0.

[0038] Step S206: Back up the changed data in the first non-primary copy using the first shadow copy;

[0039] The first node, the second node, and the third node are all nodes set up in the distributed storage system.

[0040] In at least one embodiment, this step refers to, under the premise that the first shadow copy has been created, during the abnormal handling of the first node, the system continuously monitors the write changes of data blocks in the first non-primary copy (located in the second node), and incrementally copies these added, modified or deleted changes to the first shadow copy (located in the third node) to ensure that the data content of the shadow copy is always up-to-date with the first non-primary copy.

[0041] It should be noted that the data backup in this embodiment is not a one-time full copy, but an incremental synchronous backup; the synchronization object is the change to the first non-primary replica, not the first primary replica; the synchronization target is the first shadow replica, that is, a secondary backup performed on the second node through the third node's changes; this process occurs within the window period when the first node is unavailable but the second node is still serving, with the aim of providing protection against potential failures of the second node. The first shadow replica in this step is dynamic and has real-time fault tolerance capabilities, and is not only effective at the time of creation.

[0042] In at least one embodiment, the data changes in the first non-primary replica occur after the first node enters an abnormal state. These changes are initiated by a client or application through write operations on the original data object (i.e., data initially provided by the first primary replica). These write operations are ultimately routed by the system to the first non-primary replica (the second node) for updates, resulting in changes to the data block content in that replica. For example, a user submits an order, updates the order status field, and writes block B3 to the File1 replica on node B; the log service appends logs, adding content to block B7 of the File1 replica on node B; or the historical cache is deleted, clearing the content of block B5 on node B. Such changes do not involve the first node (because the first node is in an abnormal state and the service is isolated). All write requests are automatically redirected by the system to the first non-primary replica, which then becomes the only writable replica.

[0043] To avoid the high network and I / O load caused by full retransmission, this embodiment only captures and transmits changed data block units (such as 64KB blocks), rather than the entire file or the entire copy. Each changed block carries its location identifier (offset address), change type (added / modified / deleted), and data checksum, which is sent by the control module to the third node to complete the update.

[0044] For example, a distributed storage system includes a first node A (including the first primary replica), a second node B (including the first non-primary replica), and a third node D (the third node). Data block File1 consists of 100 64KB blocks, with its primary replica on A and its non-primary replica on B. A enters an abnormal processing state due to an upgrade, and the system executes steps S202 and S204; it detects frequent access to blocks B1–B10 on B, and creates a first shadow replica on D, with its initial content being a snapshot of B at time T0; after proceeding to step S206:

[0045] At time T1: The client writes new data, updating block B3 on B (original content: 0x00…00 → new content: 0x12…34); B's storage module detects the change in B3 and generates a change log; the log is compressed and sent to D; D reads the log and updates the content at the corresponding B3 position in the first shadow copy to 0x12…34;

[0046] At time T2: Block B7 on B is appended with a new 2KB log, and the system also generates a log and synchronizes it to the first shadow copy in D.

[0047] At time T3: Block B5 on B is deleted. The system logs the deletion of B5, and the corresponding block in the first shadow copy is set to invalid or cleared.

[0048] Throughout the upgrade process, regardless of the number of writes to B, the first shadow copy on D maintains real-time consistency with the first non-primary copy on B through incremental synchronization. If B experiences a sudden disk failure at time T4, the system can immediately take over the shadow copy to respond to read and write requests, ensuring zero-impact service interruption.

[0049] Through the above steps, by determining the access information of the data blocks of the first non-primary replica during anomaly handling of the first node, and creating a first shadow replica in the third node based on this access information, the goal of creating the first shadow replica on demand in the third node can be achieved, thereby reducing the resource consumption of replica creation. Furthermore, the first shadow replica only performs incremental backups of data that has changed in the first non-primary replica, reducing the resource consumption of data transmission. It can also provide emergency redundancy in the event that the first and second nodes may fail simultaneously, avoiding the problem of complete data unavailability due to the simultaneous failure of both replicas. Therefore, it can solve the technical problem of high data backup resource consumption in related technologies, achieving the technical effect of reducing data backup resource consumption.

[0050] As an optional implementation, before determining the access information of the data blocks in the first non-master replica during the process of initiating anomaly handling for the first node, the method further includes: calculating the risk parameters set for each node in the distributed storage system, wherein the risk parameters are used to represent the urgency of handling the anomaly of the node; generating an anomaly handling sequence for the nodes set in the distributed storage system based on the risk parameters of each node; and initiating anomaly handling for the first node according to the anomaly handling sequence.

[0051] In at least one embodiment, the risk parameter is a numerical indicator used to quantify the impact of a node performing abnormal processing (e.g., upgrade, offline, maintenance) on the overall system availability. A higher parameter indicates that the node is less suitable for priority processing; a lower parameter indicates that the node is more suitable for priority processing. This is to avoid high-risk nodes being processed first due to random or sequential node processing, triggering a cascading failure, thereby achieving risk-driven orderly operation and maintenance.

[0052] In at least one embodiment, calculating the risk parameters of each node set in the distributed storage system includes: determining the node information of each node, wherein the node information includes at least one of the following: the node's failure rate in a first historical time period, the node's real-time load rate, the node's storage parameters, and the concentration of replicas set in the node; and calculating the risk parameters of each node based on the node information of each node and a preset coefficient.

[0053] For example, the value of the risk parameter can be calculated using at least one of the following node information (not all of them are required): the proportion of the number of second replicas carried by the node (calculated as the ratio of the number of data blocks that the node acts as the first non-primary replica to the total number of replicas in the cluster, i.e., the number of data blocks acting as non-primary replicas ÷ the total number of replicas in the cluster); the node's historical failure frequency (the number of abnormal restarts in the past 30 days, or a normalized calculation based on the number of abnormal restarts in the past 30 days); the node's current load rate (the load rate of the real-time central processing unit (CPU), memory, and disk I / O utilization); and the criticality tags of the data carried by the node (predefined by a data tagging system, for example, core transaction data and user session data are marked as high priority).

[0054] In at least one embodiment, the preset coefficient refers to a numerical constant manually set in the system deployment or operation and maintenance strategy to weight the contribution of different node information items to the final risk parameter. Its function is to make the risk score more closely match actual business needs. The preset coefficient is a fixed value (not obtained through dynamic training); the sum of all coefficients does not necessarily have to be 1, but should satisfy logical rationality (e.g., if the failure rate has a large impact, its coefficient should be higher than the load rate); the coefficients can be modified in the system configuration file without recompiling the program. Assuming that replica concentration has the greatest impact on risk, followed by the failure rate, and the load rate and storage parameters have a smaller impact, the following preset coefficients can be set: the preset coefficient for failure rate is 0.3, the preset coefficient for real-time load rate is 0.2, the preset coefficient for storage parameters is 0.3, and the preset coefficient for replica concentration is 0.2. The parameter value of the risk parameter can be calculated using the weighted formula: R = 0.3 × failure rate + 0.2 × CPU load + 0.3 × criticality weight + 0.2 × concentration.

[0055] For example, suppose the metrics are: Replica Concentration: 0.4 (carrying 40% of the second replica), Historical Failure Rate: 0.15 (3 anomalies in the past 30 days), CPU Load: 0.75 (75%), and Data Criticality Weight: 0.8 (high proportion of core business data). Then the risk parameters can be calculated using a weighted sum: R = 0.3 × Replica Concentration + 0.2 × Historical Failure Rate + 0.2 × CPU Load + 0.3 × Data Criticality Weight, R = 0.3 × 0.4 + 0.2 × 0.15 + 0.2 × 0.75 + 0.3 × 0.8 = 0.54. This value is the parameter value of the risk parameters for that node, used for ranking.

[0056] In at least one embodiment, the anomaly handling sequence is a list of node processing orders generated by sorting the risk parameters of each node from low to high. The system will strictly follow this sequence to initiate anomaly handling (such as upgrades, isolation, and maintenance) for nodes sequentially. Low-risk nodes are processed first: low-risk parameters have a small impact on the system, so they are processed first to shorten the overall window period; high-risk nodes are processed later: high-risk parameters are prone to triggering cascading failures, so they are postponed until the system load is low and other nodes have stabilized. For example, the first node A has a low failure rate (0.05), light load (20%), and dispersed replica distribution (concentration 0.1), resulting in a low-risk score of 0.15, and is prioritized for upgrade; while the second node B has a recent failure record (0.2), high load (80%), and a second replica that centrally carries 40% of the data (concentration 0.4), resulting in a high-risk score of 0.62, and is placed at the end of the sequence, thereby maximizing system stability in the early stages of the upgrade.

[0057] In at least one embodiment, initiating exception handling for the first node according to the exception handling sequence means that before determining the access information of the first non-primary replica, the system must first confirm that the first node has been scheduled in the exception handling sequence and trigger its exception handling process in sequence. That is, it is not an arbitrary triggering of node processing, but a controlled, orderly scheduling behavior based on global risk assessment.

[0058] For example, a distributed storage cluster includes nodes A through F, carrying 100,000 data blocks. The following steps are used to initiate exception handling:

[0059] S1, Risk parameter calculation:

[0060] A: Replication Concentration = 0.08, Failure Rate = 0.02, Load = 0.15, Criticality = 0.5 → R = 0.19;

[0061] B: Replication Concentration = 0.45, Failure Rate = 0.25, Load = 0.85, Criticality = 0.9 → R = 0.71 ← Highest Risk;

[0062] C: Replication concentration = 0.1, failure rate = 0.05, load = 0.2, criticality = 0.6 → R = 0.24;

[0063] D: Replication Concentration = 0.3, Failure Rate = 0.01, Load = 0.4, Criticality = 0.7 → R = 0.41;

[0064] E: Replication Concentration = 0.05, Failure Rate = 0.03, Load = 0.1, Criticality = 0.3 → R = 0.16;

[0065] F: Replication Concentration = 0.35, Failure Rate = 0.15, Load = 0.6, Criticality = 0.8 → R = 0.53;

[0066] Generate the exception handling sequence: EACDFB.

[0067] S2, initiate anomaly handling for node E. The system confirms E as the first node in the sequence → marks it as the first node; only then does it proceed to S202: query which nodes (e.g., C, D) contain E's second replicas, and analyze the data access information of these replicas (on C, D); proceed to S204: create shadow replicas for high-access data blocks on C, D at F (third-party node); proceed to S206: synchronize changes occurring on C, D to the shadow replicas; complete the upgrade of E, and destroy the shadow replicas;

[0068] S3, continue processing the next node; repeat the above process until all nodes have completed exception handling.

[0069] This embodiment calculates the risk parameters of each node in the distributed storage system and generates an anomaly handling sequence for each node based on these parameters. Anomaly handling for the first node is then initiated according to this sequence. This allows for setting the processing order of nodes, resulting in a relatively dispersed distribution of replicas and achieving the effect of reducing the number of shadow replicas created and lowering costs.

[0070] As an optional implementation, before determining the access information of the data blocks in the first non-primary replica during the process of initiating anomaly handling of the first node, the method further includes: obtaining a replica distribution map of the first node, wherein the replica distribution map includes the distribution information of the non-primary replicas of each data block stored in the first node in the distributed storage system; aggregating data blocks of non-primary replicas located on the same node using the replica distribution map to obtain a risk group set; selecting the node corresponding to the target risk group from the distributed storage system to obtain a third node, wherein the target risk group is the risk group in the risk group set where the first non-primary replica is located.

[0071] In at least one embodiment, the replica distribution map is a structured metadata set in a distributed storage system used to describe the physical distribution location of each data block replica, recording the node information of the primary replica and non-primary replicas of each data block. In this embodiment, before initiating node upgrades, a full data scan is performed on the node to be upgraded (first node A). By querying the distributed metadata service, the complete replica distribution map of each data block is obtained. The specific steps include: Step 1, triggering a full scan process; Step 2, locating all data blocks of the node to be upgraded, A; Step 3, constructing the replica distribution map; the replica distribution map is a structured data set. The construction method involves using the metadata list returned in Step 2, with the data block ID as the key, to construct an in-memory mapping table; Step 4, ensuring the integrity and consistency of the map; Step 5, outputting the complete replica distribution map, including the unique identifier of all data blocks; and the node information of the primary replica (A) and non-primary replicas (C, D, E) of each data block.

[0072] In at least one embodiment, data block aggregation where non-primary replicas are located on the same node refers to grouping all data blocks whose primary replicas are on the first node and whose non-primary replicas are located on the same node (e.g., C) into a logical group, called a risk group, based on the replica distribution map. For example, if the second replicas (non-primary replicas) of data blocks File1, File2, and File3 on node A are all located on node B, then they are grouped into the same risk group G_AB. The purpose of creating risk groups is to identify those two-point failure sensitive groups where the relevant data will be completely lost if the first node and a specific node (e.g., the second node) become unavailable simultaneously. That is, if the primary replica is on the first node and the non-primary replica is on the second node, and the data becomes completely unavailable if the first node is upgraded or the second node fails, it is a high-risk group. The aggregation method in this embodiment is not based on data size, access frequency, or load, but only on the replica distribution topology, which can accurately identify the uniquely dependent vulnerable path.

[0073] In at least one embodiment, the risk group set is a collection of multiple risk groups generated by the above aggregation steps. Each risk group consists of three parts: a primary replica node (i.e., the first node), non-primary replica nodes (i.e., the second node), and a list of data blocks it carries. Each risk group represents a potential risk unit of dual-node failure. This embodiment aggregates data blocks into risk groups according to the replica distribution topology, thereby transforming the originally discrete data block management into protection objects based on groups, greatly improving the efficiency and accuracy of execution.

[0074] In at least one embodiment, the target risk group is the risk group in the risk group set whose non-primary replica (second replica) node is the same as the risk group of the first non-primary replica node.

[0075] The third node is a healthy third-party node that is neither the first node nor the second node, selected to protect the target risk group, and is used to host the shadow copy.

[0076] In at least one embodiment, selecting a node corresponding to a target risk group from a distributed storage system to obtain a third node includes: if the frequency of access to data blocks in the target risk group is greater than a preset frequency within a preset time period, obtaining resource indicator information of candidate nodes in the distributed storage system, wherein the candidate nodes are nodes in the distributed storage system other than the first node and the second node; calculating resource parameters of the candidate nodes using the resource indicator information of the candidate nodes, wherein the resource parameters are used to represent the resource usage of the candidate nodes; and determining the third node based on the resource parameters of the candidate nodes.

[0077] Optionally, the preset time period is a pre-configured time window (e.g., the past 5 minutes, 10 minutes, or 1 hour) used to statistically analyze the access behavior of data blocks in the target risk group. The preset frequency is a pre-set access frequency threshold used to determine whether the target risk group belongs to a high-frequency access group. If the access frequency of data blocks in the target risk group is less than or equal to the preset frequency, it indicates that it is low-frequency data with low performance requirements, and the fine-grained resource assessment can be skipped, and any healthy node can be directly selected as the third node. If the access frequency is greater than the preset frequency, it indicates that it is critical business data that is sensitive to response latency, and the node with the best resources must be selected as the shadow replica carrying node to ensure service quality during switching. For example, assuming the preset time period is 5 minutes and the preset frequency is ≥10 accesses per minute (i.e., ≥50 accesses in 5 minutes), if the target risk group G1 (including data blocks F1, F2, and F5) has a total of 68 read / write operations in 5 minutes, the condition is met, and the resource parameter calculation process is triggered.

[0078] Optionally, candidate nodes refer to the set of all healthy nodes in the distributed storage system that are neither the first node (to be upgraded) nor the second node (i.e., the node where the non-primary replica of the target risk group resides). The selection method is as follows: exclude the first node: it is under upgrade and is unavailable; exclude the second node: that is, the node in the target risk group that carries the non-primary replica, which itself has the risk of failure and cannot be used as a safe backup target.

[0079] Optionally, resource metrics refer to quantifiable system resource status data of candidate nodes at the current moment, used to assess their ability to support shadow replicas. These include the percentage of disk space remaining (the percentage of available space on the candidate node that can be used to store shadow replicas), network bandwidth utilization (the percentage of current node egress / ingress network traffic relative to total bandwidth), CPU load (average CPU utilization over the past minute), memory utilization (the percentage of total memory used), and I / O latency (average disk I / O latency). This embodiment can be implemented with at least two of these metrics (e.g., disk space and network load).

[0080] Optionally, the resource parameter is a comprehensive score calculated by weighting multiple resource indicators of the candidate node, used to quantify the overall resource availability and carrying capacity of the node. The higher the score, the more suitable the node is as a host node for the shadow replica. For example, a multi-factor weighted scoring model can be used to calculate the suitability score (i.e., the parameter value of the resource parameter) for each candidate node: S = (disk remaining space percentage × 0.4) + (1 - network bandwidth utilization × 0.3) + (1 - CPU load × 0.3). The healthy node with the highest score (e.g., node D) is selected as the target for the shadow replica.

[0081] In one specific embodiment, assume the distributed storage system contains nodes A through E, and a hot upgrade is planned for node A. The specific steps include:

[0082] S1. Obtain the replica distribution map. The system queries the metadata service for all data blocks of the primary replicas located in A, as shown in Table 1:

[0083] Table 1:

[0084]

[0085] S2, aggregated risk group, grouped by non-primary replica nodes:

[0086] G1:(A,C)-[F1,F2,F5];

[0087] G2:(A,E)-[F3];

[0088] G3:(A,D)-[F4];

[0089] Risk group set = {G1, G2, G3}.

[0090] S3. Determine the target risk group. The system needs to create protection for the risk group where the first non-primary replica resides (i.e., the node where the non-primary replica resides will face the combined risk of upgrade and failure). Here, the first non-primary replica refers to the replica on C (because C is the second replica node of A). Therefore, the target risk group = G1(A, C).

[0091] S4, select the third node, choosing a node that is neither A nor C. Assume the current node's health status is as follows:

[0092] B: High load, not selected;

[0093] D: Free space, ample space → Select as the third node;

[0094] E: Carrying G2 is risky, so it is not selected;

[0095] F: Healthy, but bandwidth is limited;

[0096] The final choice is: the third node = D.

[0097] Create shadow copies of data blocks F1, F2, and F5 in G1 on D (proceed to subsequent step S204).

[0098] This embodiment obtains the replica distribution map of the first node, aggregates the non-primary replica distribution of its data blocks, and identifies the target risk group where the first non-primary replica is located. This allows for the rapid and accurate selection of the third node, thereby avoiding the resource waste caused by traditional global redundancy.

[0099] As an optional implementation, during the process of initiating the handling of anomalies in the first node, the access information of the data blocks in the first non-primary replica is determined, including: during the process of initiating the upgrade of the first node, obtaining information on the read and write operations performed on the data blocks in the first non-primary replica within a second historical time period to obtain the access information.

[0100] In at least one embodiment, the second historical time period is a fixed-length historical time window set for assessing the access frequency of data blocks, used to collect read and write access behavior of the target data block within that time period. For example, the second historical time period can be 5 minutes (suitable for high-frequency online services (such as database transactions, real-time logs), 1 hour (suitable for medium-frequency services (such as file services, cached data),) or 24 hours (suitable for low-frequency batch processing or archived data). It should be noted that the second historical time period is a time period used to assess the access frequency of data blocks (e.g., the past 5 minutes), that is, a time period for data block-level statistics. This is different from the first historical time period mentioned above, that is, the first historical time period is a time period used to assess the probability of node failure and load level (e.g., the past 30 days), that is, a time period for node-level statistics.

[0101] In at least one embodiment, the access information is a structured data set aggregated from read and write operation information, used to characterize the activity level of the target data block within a historical time period. Specifically, the read and write operation information refers to statistical information on all client read and write requests made to the data block stored on the node where the first non-primary replica resides within a second historical time period. For example, the content of the access information may be as shown in Table 2.

[0102] Table 2:

[0103]

[0104] In this embodiment, during the process of upgrading the first node, access information is obtained by acquiring information on the read and write operations performed on data blocks in the first non-master replica during the second historical time period, which can accurately identify data blocks with high activity.

[0105] As an optional implementation, creating a first shadow copy of the first non-primary copy in a third node based on access information includes: determining the access frequency of data blocks in the first non-primary copy from the access information, and if the access frequency is greater than or equal to a preset access frequency within a second historical time period, creating a first shadow copy in a first storage space of the third node; determining the access frequency of data blocks in the first non-primary copy from the access information, and if the access frequency is less than the preset access frequency within a second historical time period, creating a first shadow copy in a second storage space of the third node; wherein the response speed of the first storage space to data operations is greater than the response speed of the second storage space to data operations.

[0106] In at least one embodiment, the data blocks in the first non-primary replica are a set of data blocks in a distributed storage system where the primary replica is located on the first node (the node to be upgraded), and the non-primary replica (i.e., the second replica) is located on the first non-primary replica node (such as node C).

[0107] In at least one embodiment, the access frequency is the total number of read and write operations performed on a data block within a second historical time period (e.g., the past 5 minutes), expressed as operations per time window. For example, access frequency = number of read requests + number of write requests. A preset access frequency is a pre-configured numerical threshold used to distinguish between hot and cold data. For example, for real-time transaction services (high-frequency access, latency-sensitive), the preset access frequency can be set to ≥500 times / 5 minutes; for file services or logs (medium-frequency access), the preset access frequency can be set to 100–500 times / 5 minutes; for archiving or backup services (extremely low frequency, latency-tolerant), the preset access frequency can be set to <10 times / 5 minutes.

[0108] In at least one embodiment, the first storage space may be a faster-responding storage medium deployed on a third node for storing hot data shadow copies. For example, the first storage space may be a high-performance solid-state drive (Non-Volatile Memory Express Solid State Drive, or NVMe SSD for short). The NVMe SSD has a read latency of <100μs, a write latency of <200μs, and an input / output operations per second (IOPS) greater than 100.

[0109] In at least one embodiment, the second storage space can be a low-response but high-capacity, low-cost storage medium deployed on the third node, used to store shadow copies of cold data. For example, the second storage space can be a Serial ATA Hard Disk Drive (SATA HDD). SATA HDDs have read latency between 5 and 15 ms, write latency between 10 and 20 ms, and IOPS between 100 and 200. It should be noted that the third node must be configured with both types of storage media (e.g., SSD + HDD), and logical isolation and independent namespace management must be implemented by the storage management layer.

[0110] For example, in one specific embodiment, the specific process for creating the first shadow copy is as follows: Figure 3 As shown, it includes the following steps:

[0111] S301, the system receives an upgrade instruction from the operation and maintenance platform or the automatic scheduling system and identifies the node to be upgraded as the first node A (i.e., node A to be upgraded).

[0112] S302, the system initiates a full scan request to obtain the replica distribution information of all data blocks on node A. Specifically, it traverses all data blocks stored on node A (in 64KB block-level units), obtains the primary replica location (A) and the secondary replica location (non-primary replica) of each data block, and generates a replica distribution map.

[0113] S303 identifies the location of the second replica of a data block (i.e., the location of the first non-primary replica). Specifically, from the replica distribution map output in S302, the field information of each data block is extracted as the node where the first non-primary replica is located. Data blocks without a second replica or with abnormal replica status (such as offline) are excluded, and only valid replicas are retained, outputting the set of valid second replica locations.

[0114] S304, for each data block, determines whether its second replica location is the same as the second replica location of other data blocks. Specifically: if a data block has field X, and at least one other data block in the cluster has field X, then node X is a centralized non-primary replica node, meeting the risk group formation conditions. If the second replica of a data block is distributed across multiple different nodes (e.g., F6's replicas are located on C and D), it does not participate in grouping. If the second replica of a data block is node A itself (abnormal), it is marked as an error and an alarm is triggered. The final output is a set of centralized non-primary replica nodes.

[0115] S305, when the second replica is located on the same node, aggregate the data blocks on the same node as the second replica into the same risk group. Specifically: group the data blocks according to the second replica node to generate risk groups G_AB, each group corresponding to a unique identifier G_AX, where A is the node to be upgraded and X is the node where the second replica is located. Finally, output a set of risk groups, with each group containing a list of data blocks.

[0116] S306, calculate the optimal placement node for each risk group. Specifically: For each risk group (e.g., G_AB), select an optimal node from healthy nodes other than A and B (e.g., C, D, E) as the third node for the shadow replica. A suitability score S = (disk space remaining percentage × 0.4) + (1 - network bandwidth utilization × 0.3) + (1 - CPU load × 0.3) can be used to calculate the suitability score for each candidate node using a scoring model. Here, disk space remaining percentage is obtained through node local storage monitoring; network bandwidth utilization is calculated based on traffic sampling every 5 seconds from the node's network interface card; and CPU load is the average load obtained from the system over 1 minute, normalized to the 0–1 range. Candidate nodes can be filtered as follows: exclude nodes with a Down, Maintenance, or those already involved in creating other shadow replicas; exclude nodes with <10% disk space remaining or >90% CPU load; select the node with the highest S value as the placement target; if multiple nodes have the same S value, randomly select one (or sort by node number in ascending order). For example, the output results are: Risk group G_AB - placement node = D; Risk group G_AC - placement node = E; Risk group G_AD - placement node = C. The output results can be recorded in the shadow replica scheduling table.

[0117] S307, determine if the access frequency is greater than 500 times / hour. Specifically: for each data block within each risk group, query its access frequency (read + write requests) in the past hour and compare it with the preset threshold of 500 times / hour. If the access frequency > 500, mark it as hot data and proceed to S308; if the access frequency ≤ 500, mark it as cold data and proceed to S310.

[0118] S308: When the access frequency is greater than 500 times / hour, the shadow copy is created in the SSD of the node. Specifically: if a data block is determined to be hot data (access frequency > 500 times / hour), the system creates its shadow copy in the high-performance storage pool of the third node (such as D) selected in S306.

[0119] S309, Shadow Copy is ready.

[0120] S310, when the access frequency is less than or equal to 500 times / hour, the shadow copy is created in the HDD of the node. For example, if a data block is determined to be cold data (access frequency ≤ 500 times / hour), the system creates its shadow copy in the low-cost storage pool of the third node (such as D) selected in S306.

[0121] S311: If the second replica is not located on the same node, it is classified into another group. Specifically: if the second replica of a data block is distributed on two or more different nodes (e.g., the replicas of F6 are located on C and E respectively), or there is no second replica (abnormal state), then the data block will not participate in the risk group aggregation and will be classified into another group.

[0122] S312: Handle according to the standard strategy, specifically: For all processed risk groups (G_AB, G_AC, G_AD) and other groups, the system performs the following operations: close the metadata scanning session, release memory and connections; mark node A as shadow replica ready, allowing it to enter the upgrade process; activate the incremental log synchronization channel between the shadow replica and the primary replica; add all shadow replicas to the monitoring list during the upgrade, preparing for subsequent adaptive maintenance and recycling.

[0123] This embodiment achieves hierarchical dynamic creation of the first shadow copy on the third node by judging the data access frequency within the second historical time period. For data blocks with high access frequency, their shadow copies are deployed in the first storage space with fast response speed, which can ensure low-latency access to hot data during fault switching and maintain business continuity. For data blocks with low access frequency, their shadow copies are stored in the second storage space with slower response speed but lower cost, which can reduce the resource overhead of temporary redundant storage.

[0124] As an optional implementation, backing up changed data in the first non-primary replica using the first shadow replica includes: determining the first incremental data generated by the target application's access operation to the data block in the first primary replica when the target application initiates an access operation to the data block in the first primary replica; performing data change detection on the backed-up data block in the first non-primary replica when the second node synchronizes the first incremental data to the first non-primary replica; generating a first incremental log of changes to the data block in the first non-primary replica based on the detection result, wherein the first incremental log includes at least one of the following: location information of the first incremental data, sequence number of the data block where the first incremental data is located, and data verification information; and sending the first incremental log to the third node to instruct the third node to synchronize the first incremental data to the first shadow replica based on the first incremental log.

[0125] In at least one embodiment, the first incremental log only records the location, sequence number, and checksum of the changed block, with a very small data volume (e.g., <100 bytes / change), and is transmitted in only a few KB after compression.

[0126] In at least one embodiment, the verification information is used to verify whether the incremental log has been tampered with or corrupted during transmission; and to verify whether the content of the shadow copy after replay is completely consistent with that of the non-primary copy.

[0127] For example, in one specific embodiment, assume the distributed storage system includes node A (for deploying primary replicas), node B (for deploying non-primary replicas), and node D (for deploying shadow replicas). Data block F1 (64KB) is located in A (primary), B (non-primary), and D (shadow). The operation performed is that the client writes new data to node A, modifying 16 bytes of data at offsets 128–192 in F1. The specific implementation process is as follows... Figure 4 As shown, it includes the following steps:

[0128] S401, Client writes to primary replica: The client sends a write request to node A to modify 16 bytes of data in the F1 data block at offsets 128–192.

[0129] S402, primary replica writes and generates the first incremental data: Node A completes the local write, records the change to the metadata service, and sends the first incremental data to node B (for synchronizing non-primary replicas).

[0130] S403, Non-primary replica receives and applies changes (Node B): After receiving incremental data, Node B writes it to its local F1 replica. Node B's change detection module (based on log or block-level difference scan) compares the SHA-256 checksums of the F1 blocks before and after the write. If they are inconsistent, incremental log generation is triggered.

[0131] S404, Generate the first incremental log: Node B generates a structured first incremental log. This includes: location information, sequence number, and verification information.

[0132] S405, Send incremental logs to the third node (D): Node B compresses the above logs and sends them to node D through a dedicated low-latency synchronization channel (non-business network).

[0133] S406, Third Node Replay Log, Update Shadow Copy: After receiving the log, the shadow copy replay engine of node D: locates the shadow copy file, reads the original data, writes 16 bytes of new data at offset 128, recalculates SHA-256, updates the metadata, and records the timestamp of this synchronization.

[0134] This embodiment achieves low-overhead, high-precision, and strongly consistent synchronization of the first shadow replica by proactively detecting data changes and generating a first incremental log containing location, sequence number, and verification information during the synchronization of incremental data to the first non-primary replica. The third node then accurately replays the changes based on this log. This reduces network bandwidth, storage overhead, and synchronization latency, ensuring that the shadow replica remains logically consistent with the production replica during node upgrades. Without increasing the number of replicas, it effectively avoids the risk of data unavailability due to the simultaneous failure of both replicas.

[0135] As an optional implementation, after backing up the changed data in the first non-primary replica using the first shadow replica, the method further includes: obtaining the duration of the exception handling of the first node and the health parameters of the second node, wherein the health parameters are used to represent the probability that the second node is in a normal operating state; when the duration of the exception handling is greater than a preset duration and the health parameters are less than a preset health parameter, when the target application initiates an access operation on a data block in the first primary replica, obtaining the first confirmation information of the second node and the second confirmation information of the third node, wherein the first confirmation information is used to indicate that the second node allows the target application to access the data block in the first primary replica, and the second confirmation information is used to indicate that the third node allows the target application to access the data block in the first primary replica; based on the first confirmation information and the second confirmation information, synchronizing the incremental data generated by the target application's access operation on the data block in the first primary replica to the first non-primary replica and the shadow replica.

[0136] In at least one embodiment, the exception handling duration refers to the time elapsed from when the first node enters the upgrade state to the current moment. It is used to determine whether the upgrade has timed out and is a key threshold for triggering the strong synchronization mode. The strong synchronization mode means that a write operation must receive confirmation from both the second and third nodes simultaneously before returning success to the client, ensuring that data is persisted in both replicas and achieving strong consistency.

[0137] In at least one embodiment, the health parameter represents the probability that the second node is in a normal operating state. It is calculated based on a weighted average of multi-dimensional indicators (disk error rate, network latency, CPU load) and has a value range of [0, 1]. The closer the value is to 1, the more stable the node is.

[0138] In at least one embodiment, the first confirmation information is a response signal returned by the second node, indicating that it can currently receive and persist write requests, thus serving as a valid copy.

[0139] In at least one embodiment, the second confirmation information is a response signal returned by the third node, indicating that its shadow copy is ready and can receive synchronization data.

[0140] In the normal phase of this embodiment, the system adopts asynchronous synchronization: only the second node needs to confirm before returning to the client, and the shadow copy catches up asynchronously; when the upgrade times out or the health of the second node deteriorates, the system automatically upgrades to strong synchronization mode, at which point double confirmation is required, sacrificing some performance for data security.

[0141] For example, in a specific embodiment, assume node A (used to deploy the primary replica) has entered an upgrade phase, and the upgrade has lasted for 35 minutes (the threshold >30 minutes is a preset safety threshold, not a fixed value, and can be set according to specific circumstances). Node B (used to deploy the non-primary replica) has a health parameter of 0.65 (the threshold <0.7 is a warning threshold, not a fixed value, and can be set according to specific circumstances). Node D (used to deploy the shadow replica) is ready. The client initiates a write request to node A to modify data block F1. The specific implementation process (state-driven) is as follows: Figure 5 As shown, it includes the following steps:

[0142] S501, Monitoring Indicator Collection (Periodic): The system collects data every 5 seconds, and has been doing so for 35 minutes. The health parameter value of node B, H = (1 - disk error rate) × 0.4 + (1 - network latency coefficient) × 0.3 + (1 - CPU load) × 0.3 = 0.65.

[0143] S502, strong synchronization condition determination: if the upgrade duration T>30 minutes and the health H<0.7, strong synchronization will be initiated. If the condition is met, the system will switch to strong synchronization mode.

[0144] S503, client write operations trigger a double-confirmation mechanism: When the client writes to F1, the system executes: write to the first primary replica (A): A temporarily stores the write request (due to upgrades, it is not persisted, only forwarded); it forwards the write request to the second node (B) and the third node (D). Waiting for double confirmation: B returns the first confirmation message: ACK_B, and D returns the second confirmation message: ACK_D. Only when both return is a write success message sent to the client. If either fails to return within 500ms, an alarm is triggered, a retry is initiated, or the write operation is marked as failed.

[0145] S504, Synchronization Process: Incremental Log Generation: As before, it is generated by B and sent to D; D-end Replay: After receiving the log, D executes the same incremental replay + verification process as before; Log Persistence: After confirming the write, D writes the log to disk to ensure recovery from a crash.

[0146] S505, State Recovery: After node A completes its upgrade and restarts, the system detects that the data update has ended, automatically shuts down the strong synchronization mode, and resumes asynchronous synchronization, starting consistency verification and shadow copy reclamation.

[0147] This embodiment determines the upgrade duration and the health status of the second node, and only enables strong synchronization when the risk increases, thus avoiding long-term performance degradation; it eliminates the need to pre-set three replicas, saving storage resources; and it ensures strong data consistency through a double confirmation mechanism, which can eliminate the risk of data loss caused by the superposition of upgrades and failures.

[0148] As an optional implementation, after obtaining the duration of the exception handling of the first node and the health parameters of the second node, the above method further includes: if the duration of the exception handling exceeds a preset duration, adjusting the duration of the first shadow copy in the third node based on the difference between the duration of the exception handling and the preset duration and an extension coefficient, wherein the extension coefficient is calculated based on the difference.

[0149] In at least one embodiment, the exception handling duration refers to the total time elapsed from the moment the first node enters the upgrade process until the current system time. It is used to measure whether the upgrade process has timed out.

[0150] In at least one embodiment, the preset duration is a security upgrade time threshold pre-configured by the system (e.g., 30 minutes). If the time exceeds this threshold, the upgrade is considered to have an abnormal risk and may cause service interruption.

[0151] In at least one embodiment, the extension factor is a dynamically calculated multiplier factor used to amplify the retention period of the shadow copy based on the difference between the actual timeout time and a preset threshold. Its value increases non-linearly with the degree of timeout, reflecting a cumulative risk effect. Extension factor = 1 + (anomaly handling duration - preset duration) / baseline unit. For example, if the preset duration = 30 minutes, the baseline unit = 10 minutes, and the timeout is 35 minutes, the extension factor = 1 + (35 - 30) / 10 = 1.5; if the timeout is 50 minutes, the extension factor = 1 + (50 - 30) / 10 = 3.0.

[0152] In at least one embodiment, the duration of the first shadow copy refers to the total duration for which the first shadow copy (located on the third node) is retained in the system. Initially, it is the upgrade duration plus a buffer period. After the extension mechanism is triggered, this duration is dynamically extended until the upgrade is completed or the system actively reclaims it. Duration = Initial Duration × Extension Coefficient. Initial Duration = Preset Duration + Safety Buffer (e.g., 30 + 5 = 35 minutes); if the timeout is 35 minutes, the new duration = 35 × 1.5 = 52.5 minutes; if the timeout is 50 minutes, the new duration = 35 × 3.0 = 105 minutes.

[0153] In at least one embodiment, the dynamic extension mechanism employs an adaptive strategy: when an upgrade times out, the system automatically extends the lifespan of the shadow copy based on the timeout difference multiplied by an extension coefficient to address potential long-term failures or upgrade failures.

[0154] In this embodiment, the shadow copy is a temporary protective copy. However, if the upgrade takes an unusually long time, it indicates that the system is in a high-risk state. If the shadow copy is deleted ahead of schedule as originally planned, the protection will fail.

[0155] For example, in one specific embodiment, assuming the preset duration is 30 minutes (system default), the initial duration is 35 minutes (30 + 5 minute buffer), and the base unit is 10 minutes (used to calculate the extension factor). The extension factor formula is: Extension Factor = 1 + (T - 30) / 10, and the current status is: Node A upgrade has lasted 55 minutes. The specific implementation process is as follows... Figure 6 As shown, it includes the following steps:

[0156] S601, Periodic monitoring of upgrade duration: The system reads the upgrade start timestamp every 10 seconds and calculates the current duration.

[0157] S602, Determine whether the extension mechanism is triggered: Based on the exception handling duration of 105 minutes, update the shadow copy lifecycle.

[0158] S603, Update Shadow Copy Metadata: The system sends an update command to the shadow copy management service.

[0159] S604, System Behavior Change: Original Plan: Shadow replicas would be automatically destroyed after 35 minutes; Current Plan: The recycling process will be triggered after 105 minutes. During this period, the shadow replicas continue to receive incremental logs from node B, without affecting client performance or adding extra storage (only extending the retention time).

[0160] S605, Post-Upgrade Processing: Node A restarts successfully and reports upgrade completion; the system check duration is 55 minutes, less than 105 minutes, so further waiting is required. However, regardless of whether the timeout occurs, as soon as A recovers, a consistency check is immediately initiated. After the check passes, the recycling process is immediately triggered (even if 105 minutes have not yet elapsed).

[0161] This embodiment avoids the risk of data loss due to premature destruction of the shadow copy by dynamically calculating the extension coefficient and adaptively extending the lifespan of the first shadow copy.

[0162] As an optional implementation, after backing up the changed data in the first non-primary replica using the first shadow replica, the method further includes: if, during the anomaly handling of the first node, it is determined that the second node has an anomaly and the first node has recovered, obtaining an incremental operation log, wherein the incremental operation log is used to represent the log of incremental data generated during the anomaly handling of the first node; and synchronizing third incremental data to the first node based on the incremental operation log, wherein the third incremental data is the data generated when the target application accesses data blocks in the first primary replica during the anomaly of the second node and synchronized to the first shadow replica.

[0163] As an optional implementation, the incremental operation log is a set of write change logs recorded by the first shadow replica during the first node's anomaly handling, containing all write change logs that were not synchronized to the second node. Each log entry includes: data block location, sequence number, checksum, timestamp, and source operation node (client).

[0164] As an optional implementation, the third incremental data refers to the data block content of the second node that was not successfully written to the first shadow replica after the client initiated a write operation on the first primary replica during the period when the second node was abnormal.

[0165] As an optional implementation method, abnormal recovery refers to the first node being upgraded / restarted and then re-uploaded, and the system determining that it has returned to a normal service state (e.g., heartbeat recovery, service port listening, and successful metadata registration).

[0166] As an optional implementation method, data recovery refers to the process after a fault is recovered, in which the system extracts the missing data from the shadow copy based on the incremental operation log and writes it accurately to the primary copy of the first node to achieve data consistency restoration.

[0167] For example, in one specific embodiment, T0: Node A begins upgrading (the primary replica is not writable);

[0168] T1: The client writes to F1 (offset 128), and the system synchronizes the data to B (non-primary replica).

[0169] T2: Node B experiences a sudden disk failure (SMART error), service is interrupted, and B is unavailable;

[0170] T3: The client continues to write to F2 and F3. The system determines that B is abnormal and automatically synchronizes the changes to D (shadow copy).

[0171] T4: Node A upgrade complete, restart successful, service restored;

[0172] T5: The system detected that B is still unavailable and started the shadow copy recovery process.

[0173] The specific implementation process is as follows: Figure 7 As shown, it includes the following steps:

[0174] S701, Detect Node Status: The system periodically monitors the node status and initiates data recovery.

[0175] S702, Obtain Incremental Operation Log: The system reads the log file from the shadow copy storage directory of the third node (D). The log contains all missing data: location, content, order, and verification.

[0176] S703, Verify Log Integrity: Compare each log entry with the corresponding block in the shadow copy. If the verification fails, an alarm is triggered, the backup process is paused, and manual intervention is required.

[0177] S704, Perform data replay (write to the first node): The system replays the logs sequentially, writing the third incremental data to the primary replica of node A. It should be noted that the sequence number is not modified during writing, maintaining consistency with the original writing order. If the target block has already been partially written (e.g., old data was written during an upgrade), it is overwritten.

[0178] S705, Completion of patching and state update: After the write is completed, the system performs a consistency check: the SHA-256 hashes of F1~F3 in compute node A are compared with the shadow replicas. After the check passes, patching is marked as complete, the event log is recorded, and the shadow replica immediate reclamation process (relevant resource release) is triggered.

[0179] This embodiment continuously records incremental data written to the shadow copy by the client during the second node's failure and actively performs data replenishment after the first node recovers, thus enabling complete data recovery even if both nodes fail simultaneously. The sequence number sorting, checksum verification, and other methods ensure the accuracy of the replenished data order and content.

[0180] The overall process of runtime maintenance for incremental synchronization and state adaptation in the above embodiments is as follows: Figure 8 As shown, the specific steps include:

[0181] S801, Asynchronous Synchronous Mode. When node A enters the upgrade state, the system initiates refined operation and maintenance management based on quantitative indicators. This stage builds upon the previous steps: the upgrade sequence has been assigned a risk score R, calculated as follows: R = 0.3 × Failure Rate + 0.2 × CPU Load + 0.3 × Criticality Weight + 0.2 × Concentration, and shadow replicas have been placed on the optimal node D according to topology grouping and hierarchical strategies. The system first initiates an incremental synchronization mechanism, the core of which is differential data capture and transmission. When an application initiates a write operation on data in risk group G_AB, the system immediately initiates data block-level change detection after completing the write to the primary replica on node B. By comparing the 64KB data blocks before and after the write (this is the standard block size of the cluster), the system accurately identifies the changed content and generates incremental logs containing location information, sequence numbers, and data checksums. These logs are compressed using the LZ4 algorithm and transmitted to node D through a dedicated data synchronization channel. On node D, the log replay engine strictly parses and applies these incremental logs in sequence number order. The system maintains a synchronization state matrix, recording the last synchronization timestamp, checksum, and synchronization delay for each data block. This matrix provides data support for subsequent adaptive adjustments.

[0182] S802, the system's operational status monitoring module continuously tracks two key metrics: upgrade duration T and the overall health H of the second replica node B. The health score is calculated using a multi-factor weighted model:

[0183] H = (1 - Disk Error Rate) × 0.4 + (1 - Network Latency Coefficient) × 0.3 + (1 - CPU Load) × 0.3, where the disk error rate is obtained from SMART data, the network latency coefficient is obtained through real-time ping testing, and the CPU load is directly collected from system metrics. The monitoring system samples these metrics every 5 seconds and visualizes them in real time on the control panel. The protection policy upgrade is immediately triggered when the system detects that any of the following conditions are met.

[0184] S803, upgrade duration T > 30 minutes (preset safety threshold, not fixed, can be set according to specific circumstances), health H < 0.7 (warning threshold, not fixed, can be set according to specific circumstances). The system automatically switches the synchronous mode from asynchronous to strong synchronous, at which point each write operation needs to obtain write confirmation from both node B and node D simultaneously. Although this increases write latency by approximately 30%, it ensures strong data consistency.

[0185] S804, the system dynamically extends the lifespan of the shadow copy according to the degree to which T exceeds the limit, by an extension coefficient = 1 + (T-30) / 10.

[0186] S805, Node A restarted successfully.

[0187] S806, Node A upgrade complete.

[0188] This embodiment establishes a two-dimensional dynamic perception mechanism based on the upgrade duration and the health of non-primary replicas during node hot upgrades. Based on this, it achieves adaptive switching from asynchronous to strong synchronization and elastically extends the lifespan of shadow replicas, significantly improving data reliability under scenarios with overlapping anomalies. This mechanism ensures strong data consistency during the upgrade process while introducing synchronization overhead only when risks increase, avoiding the waste of high-redundancy resources throughout the entire process in traditional solutions. Furthermore, by dynamically extending the lifespan of shadow replicas, it effectively addresses upgrade delays or occasional node failures, fundamentally resolving the risk of data unavailability caused by uncertain upgrade durations and unstable replica nodes.

[0189] As an optional implementation, after synchronizing the third incremental data to the first node based on the incremental operation log, the above method further includes: when it is determined that the data in the first non-primary replica and the data in the first shadow replica have been synchronized to the first primary replica, sending a clear command to the third node to instruct the third node to delete the first shadow replica.

[0190] As an optional implementation, the clearing command is an atomic command issued by the system control module with a unique shadow copy identifier, used to trigger the third node to delete a specific shadow copy file and metadata.

[0191] For example, in one specific embodiment, assuming node A is restarted, the specific steps include:

[0192] S1, Node A restarts and is ready for service: After Node A completes the upgrade and restart, the system executes a strict data verification process: First, it calculates the SHA-256 checksum of the corresponding data on Node A and Node B block by block, and repairs the data blocks that fail the verification based on Node B.

[0193] S2 retrieves incremental operation logs from the upgrade period and backfills the latest data changes that have not yet been synchronized to Node A from the shadow copy. The control module reads the log file, extracts all operation records generated during the upgrade of Node A, and verifies the sequence number continuity and data integrity of each log entry. The logs are confirmed to be complete and reliable.

[0194] S3, replenish incremental data from the shadow copy to node A. Based on the offset and length in the log, accurately read the target data block from the shadow copy of node D. All client write changes (File8, File9) during the upgrade in node A have been fully written. Node A data status: completely equal to the latest business status.

[0195] S4, perform eventual consistency verification of three replicas: Input: Node A (latest primary replica); Node B (non-primary replica); Node D (shadow replica); Output: The three replicas are completely consistent.

[0196] S5 sends a precise cleanup command, triggering shadow copy reclamation. The system sends a precise cleanup command to node D, deleting all shadow copy files created for this round of upgrades, updating the resource allocation table, marking the temporarily occupied SSD and HDD space as available, completing the entire resource reclamation process, and preparing for subsequent node upgrades.

[0197] This embodiment achieves deterministic and secure deletion of shadow copies by sending a clear command to the third node only after data synchronization based on incremental operation logs is completed and it is confirmed that the data in both the first non-primary copy and the first shadow copy has been completely and consistently synchronized to the first primary copy. This avoids the risk of data loss due to premature or accidental deletion of shadow copies and ensures business continuity.

[0198] Embodiments of this application also provide a distributed storage system, which includes: a pre-upgrade preprocessing module, a node pre-upgrade preparation module, a runtime maintenance module during upgrade, and a post-upgrade cleanup module. The workflow of each module is as follows: Figure 9 As shown, it specifically includes:

[0199] Before performing a rolling upgrade of the distributed storage system, the system first constructs a multi-dimensional node upgrade risk assessment model in the pre-upgrade preprocessing module to quantify the upgrade risk of each node and generate the optimal upgrade sequence to reduce the probability of data unavailability due to node failure during the upgrade process.

[0200] The risk assessment model needs to comprehensively consider the following four key indicators:

[0201] Historical failure rate: Based on the abnormal events (including service downtime, heartbeat loss, disk read / write errors, etc.) recorded by the node in the past 30 days, the failure rate is normalized and calculated as follows: Failure rate = Number of abnormal events in the past 30 days / 30.

[0202] Real-time CPU load rate: The current CPU utilization rate of the data collection node is normalized to the range of 0 to 1, reflecting its instantaneous response pressure.

[0203] The critical weight of stored data: It is predefined by the data tagging system in the cluster metadata service and assigned different weight values ​​(0.1~1.0) according to the business level (e.g., core transactions, log backups, etc.) to characterize the degree of impact of the data carried by the node on business continuity.

[0204] Replica distribution concentration: Defined as the ratio of the number of data blocks carried by this node as a second replica to the total number of data blocks in the cluster. It is used to measure the potential impact range of this node as a single point of failure: Concentration = Number of second replica data blocks carried by this node / Total number of data blocks in the cluster.

[0205] The above four indicators are linearly weighted and fused to form a node upgrade risk score R, which is calculated as follows: R = 0.3 × failure rate + 0.2 × CPU load rate + 0.3 × criticality weight + 0.2 × set distribution concentration.

[0206] The system generates an upgrade sequence based on the R value of each node in ascending order, and prioritizes the node with the lowest risk score to perform the upgrade operation.

[0207] For example, node A has the following metrics: failure rate 0.05, CPU load rate 0.2, criticality weight 0.1, and concentration 0.1. Its risk score R = 0.3 × 0.05 + 0.2 × 0.2 + 0.3 × 0.1 + 0.2 × 0.1 = 0.15. Node B has the following metrics: failure rate 0.2, CPU load rate 0.8, criticality weight 0.9, and concentration 0.4. Its risk score R = 0.3 × 0.2 + 0.2 × 0.8 + 0.3 × 0.9 + 0.2 × 0.4 = 0.62. Based on this, the system prioritizes node A at the front of the upgrade sequence and node B at the back, thereby reducing the probability of the entire system being exposed to a high-risk state in the early stages of the upgrade and improving the overall operational stability and data availability of the cluster during the rolling upgrade process.

[0208] Before initiating the hot upgrade process for any node (denoted as the first node A), the system performs refined data topology awareness and dynamic shadow copy placement operations through the node upgrade preparation module to accurately identify and protect against potential dual-copy failure risks.

[0209] Before a node upgrade, the preparation module creates a shadow copy using the following steps:

[0210] Step 1: Risk Group Partitioning Based on Replica Distribution Topology (i.e., Data Topology Analysis and Grouping): First, a full scan is performed on all data blocks on the first node A. By accessing the distributed metadata service, complete replica distribution information for each data block is obtained, including the primary replica location and the secondary replica location. Subsequently, the system uses a risk group partitioning algorithm to aggregate all data blocks whose secondary replicas are located on the same second node B into a risk group, and generates a unique identifier G_AB for this group, where AB indicates that the replicas of the first node A depend on the second node B as their secondary replicas.

[0211] To ensure the definition of risk groups is clear and unique, the system requires that all data blocks in the same risk group G_AB must have their second copies located on node B, and must not contain any data blocks whose second copies are located on other nodes. For example, if the second copies of data blocks File1, File2, and File3 on node A are all located on node B, then these three are assigned to risk group G_AB; if the second copy of a data block is located on node C, then it is not included in this group.

[0212] This partitioning mechanism enables accurate identification of single-point coupling failure risks; that is, if node A and node B become unavailable simultaneously, all data in G_AB will be completely lost.

[0213] Step Two: Shadow Replica Placement Node Selection Based on Multi-Factor Resource Assessment: After identifying all risk groups, the system initiates the optimal placement node selection process to choose a suitable third node in the cluster for each risk group as the temporary storage target for shadow replicas. The system collects the following three resource indicators from all healthy nodes in the cluster in real time:

[0214] The percentage of remaining disk space (denoted as S_disk, normalized to the [0, 1] interval);

[0215] Network bandwidth utilization (denoted as U_net, normalized to the [0, 1] interval);

[0216] CPU load rate (denoted as L_cpu, normalized to the [0, 1] interval);

[0217] Based on the above indicators, the suitability score S for each candidate node is calculated using the formula: S = (S_disk × 0.4) + ((1 - U_net) × 0.3) + ((1 - L_cpu) × 0.3). The healthy node with the highest score (denoted as the third node D) is selected as the target for placing the shadow replica of this risk group, ensuring that the selected node has sufficient storage capacity and low network and computing load, thereby minimizing the impact on the normal service of the cluster.

[0218] Step 3: Integrating Shadow Replica Layered Storage with Hot and Cold Data Awareness: After determining the placement node D, the system calls the data access pattern analysis engine to analyze the historical access behavior of data blocks within each risk group to distinguish between hot and cold data: Hot data: Data blocks accessed ≥500 times in the past hour are identified as high-priority service data (such as real-time transaction sessions, online status cache, etc.); Cold data: Data blocks accessed ≤10 times in the past 24 hours are identified as low-frequency access data (such as archived logs, historical backups, etc.).

[0219] Based on the above classification, the system implements a differentiated shadow copy storage strategy: for hot data, its shadow copy is written to the high-performance storage pool of the third node D to ensure low-latency read / write response during sudden failure switching; for cold data, its shadow copy is written to the large-capacity storage pool of the third node D to reduce the storage costs caused by temporary redundancy. This tiered storage mechanism achieves automatic mapping and spatial isolation through a dynamic resource allocation engine, ensuring that the storage location of the shadow copy strictly matches its data service value.

[0220] Before node upgrades, the preparatory module implements a risk-driven, resource-aware, and policy-precise dynamic shadow replica protection mechanism through the above steps, without increasing the number of global replicas. By identifying vulnerable paths through topology awareness, selecting optimal placement through resource scoring, and optimizing costs through hot / cold data awareness, it significantly reduces storage and network overhead caused by temporary redundancy while ensuring high data availability during upgrades.

[0221] Once node A enters the upgrade state, the system initiates the runtime maintenance module during the upgrade. This module employs an adaptive operation and maintenance management process based on quantitative monitoring indicators. This process builds upon the risk group division (G_AB) and shadow copy placement (on node D) established in the previous steps, aiming to dynamically ensure data consistency and availability during the upgrade process. Specific steps include:

[0222] Step 1: Incremental Synchronization Mechanism Startup and Execution. First, the incremental synchronization mechanism based on data blocks is started to minimize network and I / O overhead. Specifically, this includes: when a client initiates a write operation on any data block belonging to risk group G_AB, after completing the write operation on the second node B (primary replica), the local difference detection module is immediately triggered; this module compares the binary content of the data blocks before and after the write operation at a cluster-preset 64KB data block granularity to identify the changed parts; a structured incremental operation log is generated, including the following fields: data block identifier; write offset; changed byte length; strictly incrementing sequence number; SHA-256 checksum of the changed data block; after the above log is compressed using a compression algorithm, it is compared with the system-preset... A dedicated synchronization channel for isolated business traffic asynchronously transmits data to the third node D. At the third node D, the log replay engine parses and replays incremental logs sequentially according to their sequence numbers, ensuring that data changes are written to the shadow replica in a globally consistent time sequence. The system maintains a synchronization state matrix to record the following status information for each data block in the shadow replica: the timestamp of the last successful synchronization; the checksum of the corresponding data block; and the synchronization delay (the time difference between writing to the primary replica and the confirmation of completion by the shadow replica). This matrix serves as the core basis for subsequent dynamic policy adjustments, providing the system with real-time, fine-grained synchronization quality feedback.

[0223] Step Two: Dual-Dimensional Status Monitoring and Triggering Conditions. The system runs the status monitoring module in parallel, collecting the following two key operational indicators every 5 seconds to assess the risk evolution trend during the upgrade process: Upgrade duration T: timed from the start of the first node A entering the upgrade state, in minutes; Comprehensive health H (i.e., monitoring level) of the second replica node B: calculated using a multi-factor weighted model, the formula is as follows: H = (1 - disk error rate) × 0.4 + (1 - network latency coefficient) × 0.3 + (1 - CPU load) × 0.3, where the disk error rate is derived from the accumulated read / write error count in the SMART self-test data of node B, after normalization; the network latency coefficient is obtained by periodically pinging the end-to-end average latency, comparing it with the baseline value, and then normalizing it; the CPU load is the average CPU utilization of node B in the last 10 seconds, normalized to the [0, 1] interval. The above indicators are visualized in real time in the system control panel and serve as input for policy triggering.

[0224] Step 3: Dynamic Upgrade of Adaptive Protection Strategy. The protection strategy upgrade mechanism is immediately triggered when any of the following conditions are met to enhance data redundancy protection: Condition 1: Upgrade duration T > 30 minutes (this threshold is a configurable parameter, with an initial default value of 30 minutes); Condition 2: The overall health H of the second replica node B < 0.7 (this threshold is a configurable warning value, with an initial default value of 0.7). Upon meeting either condition, the following two adaptive adjustments are executed:

[0225] Synchronization Mode Switching: The shadow replica's synchronization mode is switched from asynchronous write to strong synchronous mode. In this mode, all write operations targeting risk group G_AB must receive write confirmations from both the second node B and the third node D before a success response can be returned to the client. This switch will increase write latency by approximately 30%, but it ensures that in the event of a sudden failure of node B, the shadow replica can still serve as a strongly consistent data replica, guaranteeing no data loss.

[0226] Shadow copy lifespan dynamically extended: The system dynamically extends the lifespan of the shadow copy according to the degree to which the upgrade duration T exceeds the limit, using the following formula: Extension coefficient K = 1 + (T - 30) / 10; The default lifespan of the shadow copy is 5 minutes after the upgrade is completed, and its final lifespan is updated as: Final lifespan = 5 minutes × K. For example, when T = 40 minutes, K = 1 + (40 - 30) / 10 = 2, and the shadow copy lifespan is extended to 10 minutes; when T = 60 minutes, K = 3, and the lifespan is extended to 15 minutes.

[0227] The above steps can effectively address unexpected delays caused by abnormally long upgrades, system resource contention, or operational interventions, and avoid the risk of data unavailability due to premature removal of shadow copies.

[0228] After the upgrade and cleanup module completes the upgrade and successfully restarts on the first node A, it executes a deterministic data consistency check and shadow copy resource reclamation process to ensure node data integrity, service status consistency, and accurate release of temporary redundant resources. This process includes the following three ordered execution steps:

[0229] Step 1: Checksum-based primary replica consistency repair: First, perform a block-by-block checksum comparison operation on all data blocks belonging to the processed risk group G_AB on both node A and node B. Specifically, this includes: reading the current content of each data block on nodes A and B; calculating its SHA-256 checksum (based on the cluster's unified block size of 64KB); comparing the checksum values ​​to identify the inconsistent data block set (denoted as the inconsistent block set Δ); for any data block in set Δ, the system uses node B (the original primary replica) as a trusted benchmark and performs a data repair operation: the corresponding data block on node B is overwritten to the corresponding location on node A via an internal high-speed synchronization channel, ensuring its content remains consistent with the latest state before the upgrade. This step ensures that node A is restored to a basic state logically consistent with other replicas in the cluster, eliminating file system anomalies, metadata misalignment, or partial write issues that may occur during the upgrade process.

[0230] Step Two: Incremental Data Completion Based on Incremental Logs: After the basic consistency repair is completed, an incremental data completion mechanism is initiated to capture the latest changes that were not synchronized to node A due to incomplete asynchronous synchronization during the upgrade. The system reads the set of incremental operation logs generated and persistently stored during the upgrade (sorted by sequence number); iterates through this log set, filtering out all log entries whose target node is node A and whose corresponding data blocks were not overwritten in Step One; for each eligible log entry, the system reads the latest version of the data block from the third node D (shadow replica) (this version has been kept synchronized with the primary replica through the aforementioned incremental synchronization mechanism); this data version is written to the corresponding position on node A, completing the incremental data completion; this step ensures that node A not only recovers to its pre-upgrade state but also synchronizes all write changes that occurred during the upgrade process, achieving global up-to-date consistency of the data state.

[0231] Step 3: Deterministic cleanup and resource reclamation of shadow copies: After completing the above consistency verification and data replenishment, the system confirms that the first node A has a complete, accurate, and up-to-date data copy, which meets the service recovery conditions.

[0232] At this point, the system performs precise resource reclamation: a cleanup command driven by a unique identifier is sent to the third node D, explicitly including the risk group identifier G_AB associated with this upgrade; based on this identifier, the third node D locates and deletes all shadow copy files belonging to G_AB, whose paths follow a unified naming convention. After the deletion operation is completed, the system updates the resource allocation table, marking the SSD and HDD storage space occupied by the aforementioned shadow copies as available, and releasing the related I / O handles and cache entries; simultaneously, it clears the synchronization state matrix records, incremental log files, and temporary metadata related to G_AB, completing a closed-loop cleanup of the entire lifecycle. This step ensures that the lifecycle of the shadow copies is strictly bound to the upgrade task, realizing a resource management mechanism of creating, surviving, and destroying on demand.

[0233] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0234] Embodiments of this application also provide a data backup device. Figure 10 This is a structural block diagram of a data backup device according to an embodiment of this application, such as... Figure 10 As shown, the device includes:

[0235] The first determining module 1002 is used to determine the access information of the data blocks in the first non-primary replica during the process of starting the exception handling of the first node, wherein the first non-primary replica is set in the second node, and the data blocks in the first non-primary replica are backups of the data blocks in the first primary replica set in the first node.

[0236] The first creation module 1004 is used to create a first shadow copy of the first non-primary copy in the third node based on access information, wherein the data blocks in the first shadow copy are backups of the data blocks in the first non-primary copy;

[0237] The first backup module 1006 is used to back up the data that has changed in the first non-primary copy using the first shadow copy; wherein the first node, the second node and the third node are all nodes set up in the distributed storage system.

[0238] By using the above device, when handling anomalies at the first node, the access information of the data blocks of the first non-primary replica is determined. Based on this access information, a first shadow replica is created in the third node. This achieves the goal of creating the first shadow replica on demand in the third node, thereby reducing the resource consumption of replica creation. Furthermore, the first shadow replica only performs incremental backups of data that has changed in the first non-primary replica, reducing the resource consumption of data transmission. It can also provide emergency redundancy in the event that the first and second nodes may fail simultaneously, avoiding the problem of complete data unavailability due to the simultaneous failure of both replicas. Therefore, it can solve the technical problem of high data backup resource consumption in related technologies and achieve the technical effect of reducing data backup resource consumption.

[0239] Optionally, before determining the access information of the data blocks in the first non-master replica during the process of initiating anomaly handling for the first node, the above apparatus further includes: a first calculation module, used to calculate the risk parameters of each node set in the distributed storage system, wherein the risk parameters are used to represent the urgency of handling the anomaly of the node; a first generation module, used to generate anomaly handling sequence for the nodes set in the distributed storage system based on the risk parameters of each node; and a first initiation module, used to initiate anomaly handling for the first node according to the anomaly handling sequence.

[0240] Optionally, the first calculation module includes: a first determining unit, used to determine the node information of each node, wherein the node information includes at least one of the following: the failure rate of the node in a first historical time period, the real-time load rate of the node, the storage parameters of the node, and the concentration of replicas set in the node; and a first calculation unit, used to calculate the risk parameters of each node based on the node information of each node and preset coefficients.

[0241] Optionally, before determining the access information of the data blocks in the first non-primary replica during the process of initiating the exception handling of the first node, the above device further includes: a first acquisition module, used to acquire the replica distribution map of the first node, wherein the replica distribution map includes the distribution information of the non-primary replicas of each data block stored in the first node in the distributed storage system; a first aggregation module, used to aggregate data blocks of non-primary replicas located on the same node using the replica distribution map to obtain a risk group set; and a first selection module, used to select the node corresponding to the target risk group from the distributed storage system to obtain a third node, wherein the target risk group is the risk group in the risk group set where the first non-primary replica is located.

[0242] Optionally, the first selection module includes: a second acquisition unit, configured to acquire resource indicator information of candidate nodes in a distributed storage system when the frequency of access to data blocks in the target risk group is greater than a preset frequency within a preset time period, wherein the candidate nodes are nodes in the distributed storage system other than the first node and the second node; a second calculation unit, configured to calculate resource parameters of the candidate nodes using the resource indicator information of the candidate nodes, wherein the resource parameters are used to represent the resource usage of the candidate nodes; and a first determination unit, configured to determine a third node based on the resource parameters of the candidate nodes.

[0243] Optionally, the first determining module includes: a third obtaining unit, used to obtain information on read and write operations performed on data blocks in the first non-master replica during the second historical time period during the process of starting the upgrade of the first node, so as to obtain access information.

[0244] Optionally, the first creation module includes: a first creation unit, configured to create a first shadow copy in the first storage space of the third node if the access frequency of data blocks in the first non-primary copy is determined from the access information and is greater than or equal to a preset access frequency within a second historical time period; and a second creation unit, configured to create a first shadow copy in the second storage space of the third node if the access frequency of data blocks in the first non-primary copy is determined from the access information and is less than the preset access frequency within a second historical time period; wherein the response speed of the first storage space to data operations is greater than the response speed of the second storage space to data operations.

[0245] Optionally, the first backup module includes: a second determining unit, configured to determine the first incremental data generated by the target application's access operation to the data block in the first primary replica when the target application initiates an access operation to the data block in the first primary replica; a first detecting unit, configured to perform data change detection on the data block backed up in the first non-primary replica when the second node synchronizes the first incremental data to the first non-primary replica; a first generating unit, configured to generate a first incremental log of changes to the data block in the first non-primary replica based on the detection result, wherein the first incremental log includes at least one of the following: location information of the first incremental data, sequence number of the data block where the first incremental data is located, and data verification information; and a first sending unit, configured to send the first incremental log to the third node to instruct the third node to synchronize the first incremental data to the first shadow replica based on the first incremental log.

[0246] Optionally, after backing up the changed data in the first non-primary replica using the first shadow replica, the apparatus further includes: a second acquisition module, used to acquire the duration of the exception handling of the first node and the health parameters of the second node, wherein the health parameters represent the probability that the second node is in a normal operating state; a third acquisition module, used to acquire first confirmation information of the second node and second confirmation information of the third node when the target application initiates an access operation on a data block in the first primary replica, provided that the duration of the exception handling is greater than a preset duration and the health parameters are less than a preset health parameter, wherein the first confirmation information indicates that the second node allows the target application to access the data block in the first primary replica, and the second confirmation information indicates that the third node allows the target application to access the data block in the first primary replica; and a first synchronization module, used to synchronize the incremental data generated by the target application's access operation on the data block in the first primary replica to the first non-primary replica and the shadow replica based on the first confirmation information and the second confirmation information.

[0247] Optionally, after obtaining the duration of the anomaly handling of the first node and the health parameters of the second node, the above device further includes: a first adjustment module, used to adjust the duration of the first shadow copy in the third node based on the difference between the duration of the anomaly handling and the preset duration and an extension coefficient when the duration of the anomaly handling exceeds a preset duration, wherein the extension coefficient is calculated based on the difference.

[0248] Optionally, after backing up the changed data in the first non-primary replica using the first shadow replica, the above apparatus further includes: a third acquisition module, used to acquire incremental operation logs when it is determined that the second node has an anomaly during the anomaly handling of the first node and the first node has recovered to normal, wherein the incremental operation logs are used to represent the logs of incremental data generated during the anomaly handling of the first node; and a second synchronization module, used to synchronize third incremental data to the first node based on the incremental operation logs, wherein the third incremental data is the data generated when the target application accesses data blocks in the first primary replica during the anomaly of the second node and synchronized to the first shadow replica.

[0249] Optionally, after synchronizing the third incremental data to the first node based on the incremental operation log, the above device further includes: a first sending module, used to send a clearing instruction to the third node when it is determined that the data in the first non-primary replica and the data in the first shadow replica have been synchronized to the first primary replica, so as to instruct the third node to delete the first shadow replica.

[0250] For a description of the features in the embodiment corresponding to the data backup device, please refer to the relevant description in the embodiment corresponding to the data backup method, which will not be repeated here.

[0251] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data backup method embodiments.

[0252] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data backup method embodiments when it is run.

[0253] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0254] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data backup method embodiments.

[0255] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data backup method embodiments.

[0256] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, CPUs, FPGAs, Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System-on-Chips (SoCs), Complex Programmable Logic Devices (CPLDs), Microcontroller Units (MCUs), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The aforementioned computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0257] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0258] The data backup method provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data backup method, characterized in that, include: During the process of initiating the exception handling of the first node, the access information of the data blocks in the first non-primary replica is determined. The first non-primary replica is set in the second node, and the data blocks in the first non-primary replica are backups of the data blocks in the first primary replica set in the first node. Based on the access information, a first shadow copy of the first non-primary copy is created in the third node, wherein the data blocks in the first shadow copy are backups of the data blocks in the first non-primary copy; Use the first shadow copy to back up the data that has changed in the first non-primary copy; The first node, the second node, and the third node are all nodes set up in the distributed storage system; Before determining the access information of data blocks in the first non-primary replica during the process of initiating exception handling for the first node, the method further includes: Obtain the replica distribution map of the first node, wherein the replica distribution map includes the distribution information of non-primary replicas of each data block stored in the first node in the distributed storage system; The data blocks of non-primary replicas located on the same node are aggregated using the replica distribution map to obtain a risk group set. The aggregation of data blocks of non-primary replicas located on the same node means that, according to the replica distribution map, all data blocks of primary replicas on the first node and non-primary replicas located on the same node are grouped into a risk group. Each risk group consists of a primary replica node, a non-primary replica node, and a list of data blocks carried. Each risk group represents a potential risk unit of dual-node failure. The third node is obtained by selecting the node corresponding to the target risk group from the distributed storage system, wherein the target risk group is the risk group in the risk group set where the first non-primary replica is located.

2. The data backup method according to claim 1, characterized in that, Before determining the access information of data blocks in the first non-primary replica during the process of initiating exception handling for the first node, the method further includes: Calculate the risk parameters for each node in the distributed storage system, wherein the risk parameters represent the urgency of handling anomalies of the node; An anomaly handling sequence for the nodes configured in the distributed storage system is generated based on the risk parameters of each node. Anomaly handling for the first node is initiated according to the anomaly handling sequence.

3. The data backup method according to claim 2, characterized in that, Calculate the risk parameters for each node in the distributed storage system, including: Determine the node information for each node, wherein the node information includes at least one of the following: the failure rate of the node in a first historical time period, the real-time load rate of the node, the storage parameters of the node, and the concentration of replicas set in the node; The risk parameters for each node are calculated based on the node information and preset coefficients.

4. The data backup method according to claim 1, characterized in that, The third node is obtained by selecting the node corresponding to the target risk group from the distributed storage system, including: If the frequency of access to data blocks in the target risk group is greater than the preset frequency within a preset time period, the resource indicator information of candidate nodes in the distributed storage system is obtained, wherein the candidate nodes are nodes in the distributed storage system other than the first node and the second node. The resource parameters of the candidate nodes are calculated using the resource index information of the candidate nodes, wherein the resource parameters are used to represent the resource usage of the candidate nodes; The third node is determined based on the resource parameters of the candidate nodes.

5. The data backup method according to claim 1, characterized in that, During the process of initiating exception handling for the first node, the access information of the data blocks in the first non-primary replica is determined, including: During the process of initiating the upgrade of the first node, information on read and write operations performed on data blocks in the first non-primary replica during the second historical time period is obtained to obtain the access information.

6. The data backup method according to claim 1, characterized in that, Based on the access information, a first shadow copy of the first non-primary copy is created in the third node, including: If the access frequency of the data block in the first non-master replica is determined from the access information and is greater than or equal to the preset access frequency within the second historical time period, the first shadow replica is created in the first storage space of the third node. If the access frequency of the data block in the first non-master replica is determined from the access information and is less than the preset access frequency within the second historical time period, the first shadow replica is created in the second storage space of the third node. The first storage space has a faster response speed to data operations than the second storage space.

7. The data backup method according to claim 1, characterized in that, Backing up modified data in the first non-primary replica using the first shadow copy includes: In the event that the target application initiates an access operation on a data block in the first primary replica, the first incremental data generated by the target application initiating the access operation on the data block in the first primary replica is determined; When the second node synchronizes the first incremental data to the first non-primary replica, data change detection is performed on the data blocks backed up in the first non-primary replica; Based on the detection results, a first incremental log of changes to data blocks in the first non-master replica is generated, wherein the first incremental log includes at least one of the following: location information of the first incremental data, sequence number of the data block where the first incremental data is located, and data verification information; The first incremental log is sent to the third node to instruct the third node to synchronize the first incremental data to the first shadow replica based on the first incremental log.

8. The data backup method according to claim 1, characterized in that, After backing up the changed data in the first non-primary replica using the first shadow copy, the method further includes: Obtain the duration of the exception handling of the first node and the health parameters of the second node, wherein the health parameters are used to represent the probability that the second node is in a normal operating state; When the duration of the exception handling exceeds a preset duration and the health parameter is less than a preset health parameter, when the target application initiates an access operation on the data block in the first primary replica, the first confirmation information of the second node and the second confirmation information of the third node are obtained. The first confirmation information is used to indicate that the second node allows the target application to access the data block in the first primary replica, and the second confirmation information is used to indicate that the third node allows the target application to access the data block in the first primary replica. Based on the first confirmation information and the second confirmation information, the incremental data generated by the target application's access operation on the data block in the first primary replica is synchronized to the first non-primary replica and the shadow replica.

9. The data backup method according to claim 8, characterized in that, After obtaining the duration of the exception handling for the first node and the health parameters for the second node, the method further includes: If the duration of the exception handling exceeds the preset duration, the duration of the first shadow copy in the third node is adjusted based on the difference between the duration of the exception handling and the preset duration and an extension coefficient, wherein the extension coefficient is calculated based on the difference.

10. The data backup method according to claim 1, characterized in that, After backing up the changed data in the first non-primary replica using the first shadow copy, the method further includes: If, during the anomaly handling of the first node, it is determined that the second node has an anomaly and the first node has returned to normal, an incremental operation log is obtained, wherein the incremental operation log is used to represent the log of incremental data generated during the anomaly handling of the first node. The third incremental data is synchronized to the first node based on the incremental operation log. The third incremental data is generated when the target application accesses the data block in the first primary replica during the period when the second node is abnormal, and is synchronized to the first shadow replica.

11. The data backup method according to claim 10, characterized in that, After synchronizing the third incremental data to the first node based on the incremental operation log, the method further includes: If it is determined that the data in the first non-primary replica and the data in the first shadow replica are both synchronized to the first primary replica, a clear command is sent to the third node to instruct the third node to delete the first shadow replica.

12. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data backup method as described in any one of claims 1 to 11 when executing the computer program.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data backup method as described in any one of claims 1 to 11.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data backup method as described in any one of claims 1 to 11.

Images