Snapshot generation method and device, electronic equipment and storage medium
By combining write requests and dataset version serial numbers in a distributed system, ensuring the consistency of snapshot data and dynamically monitoring version changes, the data consistency and blocking problems in snapshot operations are solved, and efficient and reliable snapshot generation is achieved.
Patent Information
- Application Number
- CN202410992015.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-22
AI Technical Summary
In distributed systems, due to differences in network delay, machine capabilities and load pressure during snapshot operation, data storage nodes cannot form a unified consensus, resulting in the inability to guarantee data chronological order and spatial integrity, which affects database applications in the cloud computing field Reliability and consistency.
By combining the write request sent by the service application with the dataset version serial number to generate a second write request, ensuring the consistency of the write data with the current dataset version version, and dynamically monitoring the dataset version serial number during the snapshot creation process Change, stop unnecessary snapshot creation in time, and avoid snapshots based on outdated data.
It effectively avoids client I/O blocking, ensures the accuracy and consistency of snapshot data, prevents resource waste, and ensures the timeliness and consistency of data, especially in high concurrent write scenarios.
Smart Images

Figure CN119003544A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of cloud computing, and in particular to a method and device for generating a snapshot, an electronic device, and a storage medium. Background Art
[0002] The root cause of the snapshot consistency problem is that the data storage nodes in the distributed system cannot form a unified consensus when performing snapshot operations. Due to differences in factors such as network latency, machine capabilities, and load pressure, the time points and execution speeds at which each node receives the execution request may be inconsistent. As a result, the temporal order and spatial integrity of the data cannot be guaranteed in the snapshot operation. In the field of cloud computing, especially in database applications, this inconsistency may lead to data corruption, application crashes, or unrecoverable problems. Therefore, solving the snapshot consistency problem is crucial to ensuring the reliability and consistency of distributed system data.
[0003] In the existing technology, when solving the distributed system snapshot consistency problem, the application performance is seriously affected due to blocking the client I / O when the load is high or the network conditions are poor; and the lack of client participation in the consistency guarantee process may lead to insufficient integrity of the snapshot data space, that is, the snapshot may not contain all the latest written data, especially when the data is written in blocks and non-sequentially. Therefore, how to complete the creation of snapshot data without blocking the client I / O and ensure the accuracy and consistency of the created snapshot data has become an urgent problem to be solved. Summary of the invention
[0004] The present disclosure provides a method and device for generating snapshots, an electronic device and a storage medium, which are mainly intended to solve the problem that when creating snapshot data, the snapshot data is not accurate and there is a high frequency of client I / O blocking.
[0005] According to a first aspect of the present disclosure, a method for generating a snapshot is provided, the method being applied to a client, the method comprising:
[0006] Combining the first write request sent by the service application with the data set version serial number to generate a second write request, wherein the data set version serial number is a time version serial number of writing the data set in the distributed system;
[0007] sending the second write request to a corresponding storage unit, so as to store the write data set corresponding to the first write request in the storage unit;
[0008] In response to the snapshot request, using the data set version serial number, creating a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and after the snapshot creation is completed, sending snapshot reply information to the data storage management component;
[0009] A serial number confirmation request is sent to the data storage management component to determine whether the data set version serial number has changed; after the data set version serial number has changed, a stop instruction is sent to the target storage unit to stop creating a snapshot of the target storage unit.
[0010] In some embodiments, the data storage management component includes: a snapshot coordination component, wherein the snapshot coordination component is used to assist the data storage management component in performing snapshot processing on the client;
[0011] After creating a snapshot of the target storage unit where the target data corresponding to the snapshot request is located in response to the snapshot request, the method further includes:
[0012] Monitoring the processing time of creating a snapshot of the target storage unit;
[0013] When it is determined that the processing time exceeds a timeout threshold, snapshot failure information is sent to the snapshot coordination component, and snapshot creation for the target storage unit is stopped.
[0014] In some embodiments, in response to the snapshot request, using the data set version serial number, creating a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and sending snapshot reply information to the data storage management component after the snapshot is created, includes:
[0015] In response to the snapshot request sent by the data storage management component, generating a data sequence range of the target data based on the current data set version sequence number, wherein the data sequence range is composed of a starting data set version sequence number and an ending data set version sequence number of the data in the storage unit;
[0016] According to the current snapshot creation mode, a snapshot is created for the data blocks in the target storage unit, wherein the created snapshot includes the data sequence range and the reference relationship; and after the snapshot is created, snapshot reply information is sent to the data storage management component.
[0017] In some embodiments, creating a snapshot of the data block in the target storage unit according to the current snapshot creation mode includes:
[0018] In the case where the client includes a data block index service, if the current snapshot creation mode is a write-time redirection mode, based on the data block index service, a first data block is created for a write data set received when creating the snapshot and redirection processing is performed on the first data block;
[0019] If the current snapshot creation mode is the copy-on-write mode, then based on the data block index service, determining a target data block corresponding to the write data set received when creating the snapshot, and creating a second data block, so as to copy the data in the target data block to the second data block;
[0020] In a case where the client does not include a data block index service, the second write request is forwarded to the data block index service; the data block index service compares the data set version serial number in the second write request with the data set version serial number of the data block in the target storage unit to determine whether to create the first data block or the second data block according to the current snapshot creation mode.
[0021] In some embodiments, the method further comprises:
[0022] Before creating a snapshot, if a snapshot is deleted, the data corresponding to the data sequence range of the deleted snapshot and the data corresponding to the current data set version serial number are used as the target data for snapshot creation, and a snapshot is created for the storage unit where the target data is located.
[0023] In some embodiments, the method further comprises:
[0024] In response to the cloning request, determining a target clone storage unit that needs to be cloned, and determining a target snapshot corresponding to the target clone storage unit;
[0025] Obtaining a termination data set version serial number in the data sequence range of the target snapshot, incrementing the termination data set version serial number, and mounting a clone storage unit; and cloning the target source data to the clone storage unit;
[0026] After the cloning process is completed, a clone reply message is sent to the data storage management component.
[0027] In some embodiments, the method further comprises:
[0028] In response to the rollback request, determine the rollback snapshot corresponding to the rollback process, and determine the range of dataset version serial numbers that need to be covered based on the current dataset version serial number and the rollback snapshot;
[0029] Determine the target coverage data corresponding to the data set version serial number range, and delete the target coverage data;
[0030] After the rollback process is completed, a rollback reply message is sent to the data storage management component.
[0031] According to a second aspect of the present disclosure, a method for generating a snapshot is provided. The method is applied to a storage unit, the storage unit comprising: a data block, and the method comprises:
[0032] Receive a second write request sent by the client, read the first write request in the second write request, and store the write data set corresponding to the first write request and the data set version serial number in the second write request together, wherein the data set version serial number is a time version serial number of writing the data set in the distributed system;
[0033] In response to a snapshot request, determining a target data block where target data corresponding to the snapshot request is located, and creating a snapshot of the target data block based on a current snapshot creation mode;
[0034] A stop instruction sent by the client is received, and snapshot creation of the target data is stopped.
[0035] In some embodiments, in response to a snapshot request, determining a target data block where target data corresponding to the snapshot request is located, and creating a snapshot of the target data block based on a current snapshot creation mode includes:
[0036] If the current snapshot creation mode is the write-time redirection mode, the write data set received when the snapshot is created is stored in the first data block created by the data block index service;
[0037] If the current snapshot creation mode is the copy-on-write mode, the target data block corresponding to the write data set received when creating the snapshot is determined, the data in the target data block is copied to the second data block, and the write data set is stored in the target data block.
[0038] According to a third aspect of the present disclosure, a method for generating a snapshot is provided, the method being applied to a data storage management component, the method comprising:
[0039] Send snapshot requests to all clients;
[0040] Receive the snapshot reply message sent by the client, and after receiving all the snapshot reply messages from the client, increment the dataset version serial number, wherein the dataset version serial number is the time version serial number of writing the dataset in the distributed system;
[0041] Receive the serial number confirmation request sent by the client, and reply to the client with the current data set version serial number.
[0042] In some embodiments, the data storage management component includes: a snapshot coordination component, wherein the snapshot coordination component is used to assist the data storage management component in performing snapshot processing on the client;
[0043] The receiving the snapshot reply message sent by the client, and after receiving all the snapshot reply messages from the client, incrementing the data set version serial number, includes:
[0044] After receiving the snapshot reply messages from all the clients, the snapshot coordination component records the snapshot information and determines that the snapshot creation is completed;
[0045] After it is determined that the snapshot is created, the data set version serial number is incremented.
[0046] In some embodiments, the method further comprises:
[0047] Receive the clone reply message and the rollback reply message sent by the client, and after receiving the clone reply message and the rollback reply message, increment the data set version serial number.
[0048] According to a fourth aspect of the present disclosure, a device for generating a snapshot is provided, the device being applied to a client, the device comprising:
[0049] A generating unit, configured to combine a first write request sent by a service application with a data set version serial number to generate a second write request, wherein the data set version serial number is a time version serial number of writing a data set in a distributed system;
[0050] a first sending unit, configured to send the second write request to a corresponding storage unit, so as to store a write data set corresponding to the first write request in the storage unit;
[0051] A first creation unit is used to respond to the snapshot request, use the data set version serial number to create a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and send snapshot reply information to the data storage management component after the snapshot is created;
[0052] The first determination unit is used to send a serial number confirmation request to the data storage management component to determine whether the data set version serial number has changed; after the data set version serial number has changed, a stop instruction is sent to the target storage unit to stop creating a snapshot of the target storage unit.
[0053] In some embodiments, the data storage management component includes: a snapshot coordination component, wherein the snapshot coordination component is used to assist the data storage management component in performing snapshot processing on the client;
[0054] The device also includes:
[0055] a monitoring unit, configured to monitor a processing time for creating a snapshot of a target storage unit where target data corresponding to the snapshot request is located, after the creating unit creates a snapshot of the target storage unit in response to the snapshot request;
[0056] The second sending unit is used to send snapshot failure information to the snapshot coordination component and stop creating a snapshot for the target storage unit when it is determined that the processing time exceeds a timeout threshold.
[0057] In some embodiments, the first creating unit includes:
[0058] A generation module, configured to generate a data sequence range of the target data based on the current data set version sequence number in response to the snapshot request sent by the data storage management component, wherein the data sequence range is composed of a starting data set version sequence number and an ending data set version sequence number of the data in the storage unit;
[0059] The creation module is used to create a snapshot of the data blocks in the target storage unit according to the current snapshot creation mode, wherein the created snapshot includes the data sequence range and the reference relationship; and after the snapshot is created, the snapshot reply information is sent to the data storage management component.
[0060] In some embodiments, the creation module is further configured to:
[0061] In the case where the client includes a data block index service, if the current snapshot creation mode is a write-time redirection mode, based on the data block index service, a first data block is created for a write data set received when creating the snapshot and redirection processing is performed on the first data block;
[0062] If the current snapshot creation mode is the copy-on-write mode, then based on the data block index service, determining a target data block corresponding to the write data set received when creating the snapshot, and creating a second data block, so as to copy the data in the target data block to the second data block;
[0063] In a case where the client does not include a data block index service, the second write request is forwarded to the data block index service; the data block index service compares the data set version serial number in the second write request with the data set version serial number of the data block in the target storage unit to determine whether to create the first data block or the second data block according to the current snapshot creation mode.
[0064] In some embodiments, the apparatus further comprises:
[0065] The second creation unit is used to, before creating a snapshot, if a snapshot is deleted, use the data corresponding to the data sequence range of the deleted snapshot and the data corresponding to the version serial number of the current data set as target data for snapshot creation, and create a snapshot of the storage unit where the target data is located.
[0066] In some embodiments, the apparatus further comprises:
[0067] A second determining unit is used to determine, in response to the cloning request, a target clone storage unit that needs to be cloned, and determine a target snapshot corresponding to the target clone storage unit;
[0068] The cloning unit is used to obtain the termination data set version serial number in the data sequence range of the target snapshot, and after incrementing the termination data set version serial number, mount the clone storage unit; and clone the target source data to the clone storage unit;
[0069] The third sending unit is used to send a clone reply message to the data storage management component after completing the cloning process.
[0070] In some embodiments, the apparatus further comprises:
[0071] A fourth determining unit, configured to determine, in response to the rollback request, a rollback snapshot corresponding to the rollback process, and determine a range of dataset version serial numbers that need to be covered based on the current dataset version serial number and the rollback snapshot;
[0072] A deleting unit, used to determine the target coverage data corresponding to the data set version serial number range, and delete the target coverage data;
[0073] The fourth sending unit is used to send a rollback reply message to the data storage management component after completing the rollback process.
[0074] According to a fifth aspect of the present disclosure, a device for generating a snapshot is provided, the device being applied to a storage unit, the storage unit comprising: a data block, the device comprising:
[0075] A storage unit, used to receive a second write request sent by a client, read the first write request in the second write request, and store the write data set corresponding to the first write request and the data set version serial number in the second write request together, wherein the data set version serial number is a time version serial number of writing the data set in the distributed system;
[0076] a creating unit, configured to, in response to a snapshot request, determine a target data block where target data corresponding to the snapshot request is located, and create a snapshot of the target data block based on a current snapshot creation mode;
[0077] The receiving unit is used to receive a stop instruction sent by the client and stop creating a snapshot of the target data.
[0078] In some embodiments, the creating unit includes:
[0079] A first creation module, configured to store a write data set received when creating a snapshot into a first data block created by the data block index service when the current snapshot creation mode is a write-time redirection mode;
[0080] The second creation module is used to determine the target data block corresponding to the write data set received when creating the snapshot when the current snapshot creation mode is the write-time copy mode, copy the data in the target data block to the second data block, and store the write data set in the target data block.
[0081] According to a sixth aspect of the present disclosure, a device for generating a snapshot is provided, the device being applied to a data storage management component, the device comprising:
[0082] A sending unit, used for sending the snapshot request to all clients;
[0083] A first receiving unit is used to receive the snapshot reply message sent by the client, and after receiving all the snapshot reply messages from the client, increment the data set version serial number, wherein the data set version serial number is a time version serial number of writing the data set in the distributed system;
[0084] The reply unit is used to receive the serial number confirmation request sent by the client and reply the serial number of the current data set version to the client.
[0085] In some embodiments, the data storage management component includes: a snapshot coordination component, wherein the snapshot coordination component is used to assist the data storage management component in performing snapshot processing on the client;
[0086] The first receiving unit comprises:
[0087] A determination module, configured to record snapshot information and determine that the snapshot creation is completed after the snapshot coordination component receives snapshot reply messages from all the clients;
[0088] The processing module is used to increment the version serial number of the data set after determining that the snapshot is created.
[0089] In some embodiments, the apparatus further comprises:
[0090] The second receiving unit is used to receive the clone reply message and the rollback reply message sent by the client, and after receiving the clone reply message and the rollback reply message, increment the data set version serial number.
[0091] According to a seventh aspect of the present disclosure, there is provided an electronic device, including:
[0092] at least one processor; and
[0093] a memory communicatively connected to the at least one processor; wherein,
[0094] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.
[0095] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.
[0096] According to a ninth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method described in the first aspect is implemented.
[0097] The present disclosure provides a method and device for generating a snapshot, an electronic device, and a storage medium, which combines a first write request sent by a service application with a data set version serial number to generate a second write request, wherein the data set version serial number is a time version serial number of a data set written in a distributed system; sends the second write request to a corresponding storage unit so as to store the written data set corresponding to the first write request in the storage unit; in response to the snapshot request, uses the data set version serial number to create a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and after the snapshot creation is performed, sends a snapshot reply message to a data storage management component; sends a sequence number confirmation request to the data storage management component to determine whether the data set version serial number has changed; after the data set version serial number has changed, sends a stop instruction to the target storage unit to stop creating a snapshot of the target storage unit. Compared with the related art, the embodiment of the present disclosure ensures the consistency of the written data with the current data set version by combining the write request of the service application with the data set version serial number to generate the second write request. The dataset version serial number serves as a timestamp, reflecting the latest status of the dataset, which helps avoid data conflicts and inconsistencies, especially in high-concurrency write scenarios. By sending serial number confirmation requests to the data storage management component, the changes in the dataset version serial number are dynamically monitored, and timely responses can be made after the version is updated, such as stopping unnecessary snapshot creation, avoiding snapshots based on outdated data, and ensuring the timeliness and consistency of the data. After detecting a change in the dataset version serial number, a stop command is immediately sent to the target storage unit, effectively preventing resource waste and avoiding client I / O blocking.
[0098] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0099] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0100] Figure 1 A flowchart of a method for generating a snapshot provided in an embodiment of the present disclosure;
[0101] Figure 2 Create a process diagram for one snapshot;
[0102] Figure 3 A schematic diagram of a process where a snapshot fails to be created;
[0103] Figure 4 It is a snapshot chain relationship view;
[0104] Figure 5 A schematic diagram of a ROW mode snapshot principle provided by an embodiment of the present disclosure;
[0105] Figure 6 An architectural diagram of a data block indexing service in a distributed system;
[0106] Figure 7 A schematic diagram of a COW mode snapshot principle provided in an embodiment of the present disclosure;
[0107] Figure 8 Architectural diagram of another distributed system serving data blocks
[0108] Fig. 9 A relationship chain diagram for snapshot deletion;
[0109] Fig.10 A relationship chain diagram deleted for another snapshot;
[0110] Fig.11 A schematic diagram of a cloning process provided in an embodiment of the present disclosure;
[0111] Fig.12 A flowchart of another snapshot generation method provided by an embodiment of the present disclosure;
[0112] Fig.13 A flowchart of another snapshot generation method provided by an embodiment of the present disclosure;
[0113] Fig.14 A system architecture diagram of a distributed system provided in an embodiment of the present disclosure;
[0114] Fig.15 A schematic diagram of the structure of a device for generating a snapshot provided in an embodiment of the present disclosure;
[0115] Fig.16 A schematic diagram of the structure of another device for generating snapshots provided in an embodiment of the present disclosure;
[0116] Fig.17 A schematic diagram of the structure of another device for generating snapshots provided in an embodiment of the present disclosure;
[0117] Fig.18 A schematic diagram of the structure of another device for generating snapshots provided in an embodiment of the present disclosure;
[0118] Fig.19 A schematic block diagram of an exemplary electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION
[0119] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0120] The following describes a method and apparatus for generating a snapshot, an electronic device, and a storage medium according to an embodiment of the present disclosure with reference to the accompanying drawings.
[0121] Figure 1 A flowchart of a snapshot generation method provided in an embodiment of the present disclosure.
[0122] like Figure 1 As shown, the method comprises the following steps:
[0123] Step 101: Combine a first write request sent by a service application with a data set version serial number to generate a second write request, wherein the data set version serial number is a time version serial number for writing a data set in a distributed system.
[0124] In the embodiments of the present disclosure, when the service application is ready to submit its first write request, it is actually trying to write new data to a specific volume or group. This request usually contains the specific content of the data to be written, the target location (i.e., volume and group information), and possible metadata information. A snapshot is essentially a data set based on a time version, which links each written data (write data set) to a time version (not an absolute time, but used to indicate the order). This time version is called a data set version sequence number (Data Group Sequence, DGS).
[0125] In some embodiments of the present disclosure, the first write request sent by the service application is merged with the DGS to generate a second write request, and the following steps may be adopted but are not limited to: the client queries the data storage management component for the current DGS of the target volume or group. It can ensure that the client obtains the latest DGS value. After obtaining the DGS, the service application merges this DGS value with all other necessary information in the original write request (such as data content, target location, etc.). This means that a new field is added to the original request structure, which is specifically used to carry the DGS value. In this way, the original simple write request is upgraded to a "second write request" containing time version information. The process of merging the service application's write request with the dataset version sequence number DGS is not just a simple data append, but a complex mechanism to ensure data consistency and historical traceability, which can improve the reliability and efficiency of distributed system data management.
[0126] Step 102: Send the second write request to a corresponding storage unit, so as to store the write data set corresponding to the first write request in the storage unit.
[0127] In the embodiment of the present disclosure, after generating the second write request including DGS, the client (distributed system) will encapsulate it, and in addition to basic information such as data content and target location, the current DGS value will be added. After the encapsulation is completed, the system will perform preliminary verification on the request, including checking the integrity of the request format, the validity of the DGS value (for example, whether it is the latest serial number), and the available status of the target storage unit.
[0128] In order to process write requests efficiently, the system will intelligently select the most appropriate target storage unit based on the current load balancing strategy and the distribution of storage resources. This may involve evaluating factors such as the health, capacity, response time, and geographic location of each storage node to ensure that data can be written quickly and reliably. After the storage unit is selected, the second write request is sent to the target storage node through an encrypted network channel. In this step, the system will use security protocols such as SSL / TLS to protect the security of data during transmission and prevent data from being intercepted or tampered with.
[0129] After receiving the request, the storage unit will first perform decryption and deep verification to confirm the source, authority, and legitimacy of the request. Then, based on the DGS value in the request, the storage unit will compare it with the currently stored dataset version to ensure that the data is written in the correct version sequence. After verification, the storage unit writes the dataset to the disk at the location specified in the request and records the DGS value associated with it locally.
[0130] Step 103, in response to the snapshot request, using the data set version serial number, a snapshot is created for the target storage unit where the target data corresponding to the snapshot request is located, and after the snapshot is created, snapshot reply information is sent to the data storage management component.
[0131] In an embodiment of the present disclosure, a snapshot request may be generated in a distributed system according to a preset strategy or generated according to a user instruction, which is not limited in the embodiment of the present disclosure. In response to a snapshot request, the client (distributed system) will initiate a set of collaborative work processes, aiming to efficiently and accurately capture the instantaneous state of the target data in order to ensure the integrity and consistency of the snapshot. The client (distributed system) identifies the target data (set) specified in the snapshot request and the target storage unit in which it is located. This process involves complex metadata parsing in order to ensure that the physical location of the data can be accurately located even in a distributed storage environment. Subsequently, the system enters a highly coordinated snapshot creation phase for these target storage units. It should be noted that the snapshot request may specify data in all storage units in the client or partial data, which is not limited in the present disclosure.
[0132] In the current snapshot creation process, creating a snapshot will temporarily suspend any write operations to the target data to avoid inconsistent data states during snapshot creation. In good performance conditions, applications or users are given a snapshot experience without perceived blocking. But this also means that in the case of poor performance or high load pressure, snapshot behavior will greatly affect the read and write capabilities of applications and users. And because of its characteristic of blocking the client, in the consistency group scenario of multiple hosts and multiple clients, the barrel effect pointed to by problems such as network latency differences and slow hosts will amplify this performance impact to the extreme. Therefore, in the disclosed embodiment, based on the DGS stored together with the target data during data storage, it can reflect the order of write data in the time series, so when creating a snapshot, it can be created directly according to the DGS of the target data. In this way, the target data (set) that needs to be snapshotted and its target storage unit can be processed without blocking the client I / O (read and write).
[0133] After receiving the snapshot request, the client switches to snapshot mode. The client's write logic will be specially adjusted. Before each write operation, it will actively communicate with the management module to query and verify whether the Current_DGS has changed. This is done to ensure that all subsequent write operations are based on the latest version of the data set to avoid introducing inconsistencies during the snapshot process. Before each write, the client calls the management module interface to obtain a data packet containing the current volume information, especially paying attention to the value of Current_DGS. This ensures that even if other write operations occur during the snapshot preparation phase, the latest data status can be accurately captured to maintain data consistency and integrity.
[0134] When the client enters snapshot mode, the coordination module confirms that all target data is ready, and then instructs the client to perform a snapshot operation on the data in the target storage unit. Snapshot creation is a process of instantly freezing the data state. It does not interfere with the normal reading and writing of data, but copies an instant image of the data, providing a reliable baseline for subsequent backup, recovery or analysis. When the client enters snapshot mode, the client immediately sends a snapshot reply message to the data storage management component. This information may include but is not limited to the status of the snapshot operation (success / failure), as well as detailed metadata of the snapshot, such as DGS value, snapshot timestamp, snapshot reference relationship (Ref), etc. This information is crucial for subsequent management and use, especially since DGS directly marks the version of the data set represented by the snapshot, and Ref reflects the data dependencies and link relationships before the snapshot, providing precise guidance for data recovery.
[0135] Step 104, sending a serial number confirmation request to the data storage management component to determine whether the dataset version serial number has changed; after the dataset version serial number has changed, sending a stop instruction to the target storage unit to stop creating a snapshot of the target storage unit.
[0136] In the embodiments of the present disclosure, in order to ensure the consistency of the dataset version sequence number (DGS) and the accuracy of the snapshot operation, the client (distributed system) starts a background monitoring task at a key node in the snapshot creation process, such as in the preparation stage or the initial stage of snapshot creation. The main responsibility of this task is to initiate a sequence number confirmation request to the data storage management component regularly or when a specific trigger condition is met. The request is intended to verify whether the DGS remains unchanged or has been updated since the snapshot creation instruction was issued.
[0137] After receiving the serial number confirmation request, the data storage management component will quickly check and compare the currently recorded DGS value with the DGS value mentioned in the request. If the DGS value is found to have not changed, it means that the dataset version based on which the current snapshot is created is still valid, and the monitoring task will continue to periodically check the DGS status until the snapshot is created. Figure 2 As shown in FIG. 1 , once a change in the DGS value is detected, the client (distributed system) system will immediately trigger a response process. In other words, in the face of the DGS change, an explicit stop instruction is sent to the target storage unit responsible for the creation of the current snapshot through the internal communication mechanism.
[0138] After receiving the stop command, the target storage unit will immediately terminate the snapshot operation and perform necessary cleanup work, such as releasing temporarily allocated resources and rolling back any unfinished write operation changes, to ensure that the storage unit returns to a stable and consistent state.
[0139] The present disclosure provides a method for generating a snapshot, which combines a first write request sent by a service application with a data set version serial number to generate a second write request, wherein the data set version serial number is a time version serial number of a data set written in a distributed system; sends the second write request to a corresponding storage unit so as to store the written data set corresponding to the first write request in the storage unit; in response to the snapshot request, uses the data set version serial number to create a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and after the snapshot creation is performed, sends a snapshot reply message to a data storage management component; sends a sequence number confirmation request to the data storage management component to determine whether the data set version serial number has changed; after the data set version serial number has changed, sends a stop instruction to the target storage unit to stop creating a snapshot of the target storage unit. Compared with the related art, the embodiment of the present disclosure ensures the consistency of the written data with the current data set version by combining the write request of the service application with the data set version serial number to generate the second write request. The dataset version serial number serves as a timestamp, reflecting the latest status of the dataset, which helps avoid data conflicts and inconsistencies, especially in high-concurrency write scenarios. By sending serial number confirmation requests to the data storage management component, the changes in the dataset version serial number are dynamically monitored, and timely responses can be made after the version is updated, such as stopping unnecessary snapshot creation, avoiding snapshots based on outdated data, and ensuring the timeliness and consistency of the data. After detecting a change in the dataset version serial number, a stop command is immediately sent to the target storage unit, effectively preventing resource waste and avoiding client I / O blocking.
[0140] Further, as a refinement of the above embodiment, the data storage management component includes: a snapshot coordination component, wherein the snapshot coordination component is used to assist the data storage management component in performing snapshot processing on the client.
[0141] Specifically, in the disclosed embodiment, in order to better coordinate the snapshot creation process, a snapshot coordination component is added in the disclosed embodiment, which provides a unified snapshot data version consensus for a volume (Volume, Vol) or a consistency group. This component has a strong perception of the snapshot version of the application data space and is usually not an independent service. Therefore, it can be integrated into different components or services in different scenarios. In the disclosed embodiment, the snapshot coordination component is integrated in the data storage management component. In the public cloud scenario, it can be integrated into a service that has volume or consistency group information, such as Openstack's Cinder service or the next layer, the volume or consistency group information management service of the distributed system itself. It should be noted that in the consistency snapshot or consistency group snapshot scenario, this service is usually also the highest integration node for initiating snapshot requests, so its integration of the snapshot coordination component is reasonable. Since the snapshot coordination component is integrated into the data storage management component (volume, group information management module), DGS related information will also be stored in the volume, group information and snapshot information.
[0142] Further, as an implementable manner of the embodiment of the present disclosure, after creating a snapshot of the target storage unit where the target data corresponding to the snapshot request is located in response to the snapshot request, the method further includes:
[0143] Monitoring the processing time of creating a snapshot of the target storage unit; when determining that the processing time exceeds a timeout threshold, sending snapshot failure information to the snapshot coordination component and stopping creating a snapshot of the target storage unit.
[0144] Specifically, in the disclosed embodiment, during the snapshot creation process, the client needs to enter snapshot mode, so that each IO adds a request process that affects the performance for a period of time. The disconnection and downtime of one client cannot cause the performance of other clients to be impaired for a long time until the client is restored. Therefore, the snapshot cannot be timed out indefinitely, but the snapshot needs to be handled when it times out. That is, in the above steps, if Figure 3 As shown, when the snapshot creation request is sent from the snapshot coordination component of the data storage management component (volume and group management service) to the client, a timeout period (i.e., timeout threshold) is set. If no response is received, the snapshot fails. At this time, since the snapshot is not successfully created, the DGS is not progressive. Therefore, when the client snapshot mode is used to obtain the volume and group information Current DGS, the snapshot coordination component returns a snapshot failure message, so that normal clients can actively exit the snapshot mode.
[0145] Further, as a refinement of the above embodiment, in response to the snapshot request, using the data set version serial number, creating a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and after the snapshot is created, sending snapshot reply information to the data storage management component, including:
[0146] In response to the snapshot request sent by the data storage management component, generating a data sequence range of the target data based on the current data set version sequence number, wherein the data sequence range is composed of a starting data set version sequence number and an ending data set version sequence number of the data in the storage unit;
[0147] According to the current snapshot creation mode, a snapshot is created for the data blocks in the target storage unit, wherein the created snapshot includes the data sequence range and the reference relationship; and after the snapshot is created, snapshot reply information is sent to the data storage management component.
[0148] Specifically in the disclosed embodiment, when the data storage management component issues a snapshot request, the client verifies the received snapshot request to ensure the legitimacy and integrity of the snapshot request, including checking the request source, the identifier of the target data set, and the expected snapshot type.
[0149] Based on the currently valid dataset version sequence number (DGS), a data sequence range of the target data is generated. This range defines the dataset version interval that the snapshot will capture, from the starting DGS of the data in the storage unit to the current latest DGS. This process involves complex metadata queries and version control logic to ensure that the selected data range is accurate.
[0150] The snapshot creation mode can be divided into the Copy-on-Write (COW) mode and the Write-Through (ROW) mode according to the way data changes are captured. Based on the selected snapshot mode, the client starts to perform the snapshot creation operation of the data block on the target storage unit. This process involves reading the data blocks within the specified data sequence range, recording their current status, and generating corresponding copies or difference records in the snapshot storage area. At the same time, the reference relationship between the snapshot and the original data is maintained to ensure the traceability and consistency of the snapshot data.
[0151] When creating snapshot data, the system records in detail the mapping relationship between each snapshot data block and the original data block. This reference relationship is crucial for subsequent data recovery, version comparison, and space recovery. It enables snapshots to provide instant data views without affecting the original data.
[0152] For example, Figure 4It is a snapshot chain relationship view. The volume information adds the following entry to indicate the current dataset version serial number: Current_DGS:5. Taking snapshot 3 as an example, the snapshot information adds the following entry to indicate the snapshot incremental data range and the snapshot chain reference relationship: DGS: [3,3], Ref: the identity of snapshot 1. In addition, the original data relationship between the volume and the snapshot will not change. The metadata changes of the entire process are shown in Table 1:
[0153] Table 1
[0154]
[0155]
[0156] In fact, the DGS range in the snapshot information only indicates the current incremental snapshot data range. For example, when reading the full snapshot data of snapshot 3, you will find that the dataset version serial number set corresponding to the full snapshot data range of snapshot 3 is [0,3] after traversing the Ref relationship. Similarly, you can find that the full data set of snapshot 4 is [0,3,4].
[0157] The snapshot data range relies on the progressive version number, which makes the DGS range of the new snapshot in the snapshot chain always larger than the DGS range of the old snapshot. This is conducive to the fact that there will be no DGS conflict when distinguishing data ranges. This can be called the monotonic increase of DGS. Note that the data sequence range is defined as [start DGS, end DGS] instead of simply defined as a version number. Because under the design of this proposal, a single data set version number DGS cannot always accurately represent the snapshot incremental data range.
[0158] Further, as a refinement of the above embodiment, the step of creating a snapshot of the data block in the target storage unit according to the current snapshot creation mode includes:
[0159] In the case where the client includes a data block index service, if the current snapshot creation mode is a write-time redirection mode, based on the data block index service, a first data block is created for a write data set received when creating the snapshot and redirection processing is performed on the first data block;
[0160] If the current snapshot creation mode is the copy-on-write mode, then based on the data block index service, determining a target data block corresponding to the write data set received when creating the snapshot, and creating a second data block, so as to copy the data in the target data block to the second data block;
[0161] In a case where the client does not include a data block index service, the second write request is forwarded to the data block index service; the data block index service compares the data set version serial number in the second write request with the data set version serial number of the data block in the target storage unit to determine whether to create the first data block or the second data block according to the current snapshot creation mode.
[0162] Specifically in the embodiments of the present disclosure, Figure 5 As shown, according to the ROW mode snapshot principle, when creating a snapshot, the original data blocks 1-4 can be set to read-only and cannot be modified for data protection of snapshot 1. When the storage unit (volume, group) has new written data, a new data block is created and redirected based on the data block range where the write is located. That is, when there is a new write to (logical) data block 4, an actual storage data block 5 will be created on the disk of the storage node, and data block 4 will be redirected to the actual storage data block 5. At the same time, data block 5 also uses data block 4 as the base. In this embodiment, the distributed system relies on the index architecture of the distributed storage system itself for volume and group data shards, which is roughly divided into a two-layer architecture and a three-layer architecture to realize the perception of the creation of new data blocks for newly written data and the creation of new data blocks in the storage nodes. It should be noted that in Figure 5 In the context of, a storage node is a reference to an actual physical storage medium, which indicates the exact hardware location where the data resides. In contrast, the storage unit discussed in the present disclosure is an abstract level in a distributed system environment. It is a virtualized storage entity used to represent the logical location of data storage. In short, both serve the purpose of indicating data storage points, the former is based on physical existence, and the latter focuses on logical data organization and access. Unless otherwise specified in the drawings of the present disclosure, storage nodes can be understood as equivalent to storage units, and will not be described one by one in this disclosure.
[0163] Figure 6 This is an architecture diagram of a data block indexing service in a distributed system, that is, the client directly sends IO to the storage node, and the storage node has a local disk. At this time, the data block indexing capability is implemented in the client or metadata service (depending on whether there is a centralized metadata service).
[0164] In the ROW mode snapshot, the client itself or the metadata service saves part of the snapshot information and is responsible for the creation and redirection of data block 5. After the above upper-layer snapshot process is completed, the client itself knows that the current new DGS is 1 and needs a new data block 5 to store the new write data. Then it requests the metadata service to request the storage node to create a new data block for the data corresponding to the new DGS to be written according to the distributed algorithm.
[0165] In COW mode snapshot, Figure 7 As shown in the figure, when data block 4 is newly written after the snapshot is created, in order to protect the data of data block 4 snapshot 1, the data block 4 DGS 0 is copied to the new data block 5 DGS 0. Then the new write of data block 4 by the volume and group will directly overwrite the data with DGS 1 at the original location of data block 4.
[0166] Figure 8 This is another architecture diagram of a data block index service in a distributed system. Usually, a three-tier architecture refers to the client forwarding IO through the data block index service. In the ROW mode or COW mode snapshot, the index service is originally responsible for requesting the creation of new data blocks and redirection. In this embodiment, when the data block index service is responsible for forwarding and finds that the write Current_DGS issued by the client for a volume or group does not match the original data block DGS, it knows that a new data block 5 should be created to store the new write request.
[0167] Furthermore, in the case that a previous snapshot is deleted before creating a snapshot, the embodiment of the present disclosure also provides the following method for creating a snapshot:
[0168] Before creating a snapshot, if a snapshot is deleted, the data corresponding to the data sequence range of the deleted snapshot and the data corresponding to the current data set version serial number are used as the target data for snapshot creation, and a snapshot is created for the storage unit where the target data is located.
[0169] Specifically, in the embodiment of the present disclosure, the snapshot information data sequence range is a combination of a starting data set version sequence number and an ending data set version sequence number [starting DGS, ending DGS], and the main factor is used to adapt the snapshot deletion capability. For example, Fig. 9 In this case, after snapshot 3 is deleted, a new snapshot 4 is created. It should be noted that after snapshot 3 is deleted, the data with DGS 3 in the protection data range of snapshot 3 and the data with DGS 4 newly written after snapshot 3 is created will not be deleted. At this time, snapshot 4 is created, and the protection data range of snapshot 4's incremental snapshot should naturally be [3,4]. The metadata of snapshot 4 is {DGS: [3,4], Ref: snapshot 1 identity}, and its full data range is [0,3,4].
[0170] In addition, some distributed storage systems support the function of deleting the intermediate snapshots in the snapshot chain and changing the incremental data range of other snapshots on the chain. Fig.10As shown in the figure, snapshots 3 and 4 are created sequentially. After deleting the data of snapshot 3 DGS[3,3], the metadata of snapshot 4 changes to {DGS:[3,4],Ref: snapshot 1 identity}. This means that the incremental snapshot data of snapshot 4 changes to [3,4], and the full snapshot data remains unchanged at [0,3,4]. This is consistent with the semantics of deleting the intermediate snapshots in the snapshot chain in most distributed storage systems.
[0171] In addition, in order to save the underlying disk space, each distributed storage system will merge the incremental data of snapshot 3 and snapshot 4 after deleting the intermediate snapshot 3. In the embodiment of the present disclosure, when merging the incremental data of snapshot 3 and snapshot 4, the data of the older DGS is always overwritten with the data of the newer DGS. For example, the data merger is to overwrite the data of DGS3 with DGS4 to ensure the time consistency.
[0172] Furthermore, when performing rollback processing based on snapshot data, the method further includes:
[0173] In response to the rollback request, determine the rollback snapshot corresponding to the rollback process, and determine the range of dataset version serial numbers that need to be covered based on the current dataset version serial number and the rollback snapshot;
[0174] Determine the target coverage data corresponding to the data set version serial number range, and delete the target coverage data;
[0175] After the rollback process is completed, a rollback reply message is sent to the data storage management component.
[0176] Specifically in the embodiments of the present disclosure, due to its recovery characteristics for data space, rollback itself requires the client to uninstall and remount it under almost all distributed storage designs. Similar to cloning, Current_DGS is naturally known after remounting, which solves the DGS consistency problem. The reason why the rollback volume needs progressive DGS is to distinguish the data before and after the rollback. This is to distinguish the data range at the bottom layer. Among them, DGS2 data is the newly written data after creating snapshot 2, and the data range of the source volume at time 3 is [0, 1, 2]. DGS3 data is the newly written data after rolling back to snapshot 1, and the data range of the source volume at time 4 is [0, 3]. After the rollback, the DGS2 data is equivalent to the data abandoned by the application or user. It is not protected by any snapshot and is regarded as invalid data at the bottom layer. It can be deleted and the disk space can be reclaimed. Table 2 shows the data change process in the rollback process:
[0177] Table 2
[0178]
[0179] The present disclosure also provides a method for data cloning. Based on the data storage method in the above embodiment, when performing data cloning, the method further includes:
[0180] In response to the cloning request, determining a target clone storage unit that needs to be cloned, and determining a target snapshot corresponding to the target clone storage unit;
[0181] Obtaining a termination data set version serial number in the data sequence range of the target snapshot, incrementing the termination data set version serial number, and mounting a clone storage unit; and cloning the target source data to the clone storage unit;
[0182] After the cloning process is completed, a clone reply message is sent to the data storage management component.
[0183] Specifically, in the disclosed embodiment, cloning is actually an act of creating a new volume based on the source volume snapshot. Usually, the initial Current_DGS when a new volume is created is the DGS (termination DGS) of the snapshot of the source volume incremented by 1, and the Current_DGS of the source volume does not actually change. Due to the monotonically increasing nature of DGS, after the clone volume increments by 1 on the termination DGS of the snapshot based on it, it can be ensured that the DGS of the newly written data after the clone volume is cloned will never conflict with the DGS of the source volume data it references. Fig.11 As shown, when creating a clone volume, the expression form of the clone volume reference relationship in the data storage management component (volume management service) does not need to be modified, and it can be based on the old design of each distributed system; in addition, the entry Current_DGS is added as described above. The source volume and source volume snapshot 3 based on the old reference information can be queried, and the record of Current_DGS can be added to find that the data range of the clone volume at this time is [0, 3, 4]. Since the creation of a new clone volume must require the client to mount the new clone volume, the client naturally knows when mounting that the subsequent write request needs to carry CurrentDGS 4 to send to the storage node, so there will be no inconsistency between the data storage management component and the client DGS version. At the same time, the data with a clone volume DGS of 4 should obviously be distinguished from the data with a source volume DGS of 4 (the latter is represented by the incremental data range of the source volume snapshot 4), which depends on the ability of each original distributed system to use the volume label to distributed index mapping.
[0184] Fig.12 A flowchart of another snapshot generation method provided by an embodiment of the present disclosure.
[0185] like Fig.12 As shown, the method comprises the following steps:
[0186] Step 201, receive a second write request sent by the client, read the first write request in the second write request, and store the write data set corresponding to the first write request and the data set version serial number in the second write request together, wherein the data set version serial number is the time version serial number of writing the data set in the distributed system.
[0187] In an embodiment of the present disclosure, the storage unit listens to and receives a second write request from the client through its network interface module. This request is encrypted and decrypted by the network transport layer to ensure the security of the data. Subsequently, the request processing module inside the storage unit begins to parse the request, carefully separating the first write request data and the data set version sequence number (DGS) contained therein. Before formally processing the write operation, the storage unit will verify the data set in the first write request, including but not limited to data integrity verification, data format compliance check, and permission verification, to ensure that only legal and correctly formatted data can be further processed. At the same time, the system will also verify the validity of the DGS, confirming that it matches or updates the version information currently maintained by the storage unit to prevent data confusion caused by inconsistent versions. After confirmation, the storage unit processes the data set in the first write request and the corresponding DGS as a whole. This means that on the physical or logical storage medium, the data is not only securely stored, but the DGS associated with it is also clearly recorded.
[0188] During the storage process, the storage unit will also use the DGS mechanism to implement strict data consistency control. If the system detects that other write operations have caused the DGS to be updated during the processing of the current request, the storage unit may need to suspend the current operation, re-verify and possibly adjust the DGS value to ensure that the stored data is always consistent with the dataset version currently recognized by the system. After completing the storage of data and DGS, the storage unit will generate a corresponding operation log, recording the metadata information of this write in detail, including but not limited to timestamp, operation type, DGS value, etc., which is crucial for subsequent auditing, fault recovery and data tracking. At the same time, the storage unit will send a successful confirmation response to the client through the network to inform that the write operation has been completed, ensuring the closed loop of request and response and improving the reliability of the system.
[0189] Step 202: In response to a snapshot request, determine a target data block where target data corresponding to the snapshot request is located, and create a snapshot of the target data block based on a current snapshot creation mode.
[0190] In the embodiments of the present disclosure, the storage unit needs to accurately identify the location of the "target data" referred to by the snapshot request. This usually involves a complex metadata management system that records the allocation of all data blocks and the mapping relationship between data blocks and files or volumes. The system searches and locks the specific data block where the target data is located in the metadata by parsing the identifier (such as file path, volume ID or logical unit number LUN) in the snapshot request. This process ensures the pertinence and efficiency of the snapshot operation. The specific implementation method of the snapshot is determined according to the preset snapshot creation policy or the mode specified by the user in the request. According to the selected snapshot mode, the system performs the corresponding operation. During the snapshot creation process, the system will also adopt a variety of strategies to optimize storage efficiency, such as compression and deduplication technology to reduce data redundancy, or use intelligent caching strategies to balance I / O performance. In addition, in order to manage many snapshots for a long time, the system will regularly perform merge and cleanup operations to delete old snapshots that are no longer needed or integrate duplicate blocks to maintain the sustainable use of storage resources.
[0191] Step 203: Receive a stop instruction sent by the client, and stop creating a snapshot of the target data.
[0192] In an embodiment of the present disclosure, the management module in the storage unit or the component directly responsible for the snapshot operation first receives a stop command sent by the client via the network. The storage unit immediately checks the current snapshot task status of the target data to confirm whether the snapshot creation is in progress. If the snapshot task has been completed or has not yet started, the system will record this and return corresponding feedback to the client. For ongoing tasks, the system will lock its execution status to prevent any new write operations from interfering with the snapshot process and ensure the frozen state of the data. The storage unit can respond quickly to the client's stop command, effectively terminate the snapshot creation process, and maximize the protection of data consistency and system stability.
[0193] Further, as a refinement of the above embodiment, in response to a snapshot request, determining a target data block where target data corresponding to the snapshot request is located, and creating a snapshot of the target data block based on a current snapshot creation mode includes:
[0194] If the current snapshot creation mode is the write-time redirection mode, the write data set received when the snapshot is created is stored in the first data block created by the data block index service;
[0195] If the current snapshot creation mode is the copy-on-write mode, the target data block corresponding to the write data set received when creating the snapshot is determined, the data in the target data block is copied to the second data block, and the write data set is stored in the target data block.
[0196] Specifically, in the disclosed embodiment, in ROW (Record Update) mode, combined with the snapshot creation strategy of redirection on write (WriteRedirect), the entire process focuses more on dynamic adjustment in data write operations to adapt to the generation of snapshots, ensuring data consistency and write efficiency. When preparing for a snapshot operation, the client or data block index service (metadata service) will first update the metadata related to the target data set, mark the upcoming snapshot operation, and record the current data set version sequence number (DGS). This step ensures that all relevant components in the system have a unified understanding of the snapshot plan. Based on the requirements of ROW mode and redirection on write, the system configuration specifies that during snapshot creation, all new write requests do not directly overwrite the original data blocks, but are redirected to new data blocks. This requires the client or metadata service to be able to instantly identify and process such requests.
[0197] When the snapshot creation command is activated and the system is in write-time redirection mode, the data block index service comes into play. It is responsible for allocating storage nodes and corresponding data blocks for the new data to be written according to predefined distributed algorithms (such as consistent hashing, load balancing strategies, etc.). This new data block is specially created for the new DGS (such as Figure 5 As shown, the client is prepared for the write operation corresponding to the new DGS. After learning that the new DGS is 1, the client no longer directly writes to the original data block 5 for any new write request, but obtains the information of the first data block (i.e., the first data block) allocated for the new DGS through interaction with the data block index service. The client or the metadata service requests the client to perform this operation to ensure that the new data is accurately redirected to the newly allocated data block. The storage node creates a new data block according to the instructions of the data block index service and is ready to accept write data from the client. The client then writes the new data directly to this first data block, realizing write-time redirection, ensuring the integrity of the original data block during snapshot creation, and also guaranteeing the immediate storage needs of new data.
[0198] As the new data is successfully written to the first data block, the client or metadata service will update all metadata records related to the snapshot, including the location, size, and relationship of the new data block to the original data block, to ensure the integrity and traceability of the snapshot information. After the snapshot is created and confirmed to be correct, the system may release or adjust the resources temporarily occupied during the snapshot process, and restore the normal write path so that subsequent write requests can be directly written back to the original data block (unless a new snapshot operation requires continued redirection). Through the above detailed steps, the ROW mode combined with the snapshot strategy of redirection on write not only effectively avoids the problem of data overwriting during the snapshot period, but also ensures the continuity of data writing and the efficient operation of the system.
[0199] The process of creating a snapshot in COW (Copy-On-Write) mode, especially when processing newly written data, adopts a mechanism to protect the integrity of the original data to ensure the consistency of the snapshot data. The system starts to build a snapshot mapping table to record the status and correspondence of the current data blocks. See Figure 7 , when creating a snapshot of the storage unit, a write request to data block 4 (DGS 0) is detected. This means that the content of data block 4 will be copied to a new data block 5 (DGS 0) to keep the content of data block 4 unchanged at the snapshot time.
[0200] After the copy is complete, it indicates that all operations on block 4 of DGS0 should be directed to the newly created block 5. In this way, the original content of block 4 is preserved as part of the snapshot, and the newly written data is stored in block 4 and associated with DGS1. Once block 5 is ready, any new writes to block 4 will occur directly on the original block 4, but these writes will be treated as data from DGS1. This means that the DGS flag on block 4 is updated to 1, reflecting that this part of the data is modified after the snapshot was created.
[0201] Fig.13 A flowchart of a snapshot generation method provided in an embodiment of the present disclosure.
[0202] like Fig.13 As shown, the method comprises the following steps:
[0203] Step 301: Send snapshot requests to all clients.
[0204] In an embodiment of the present disclosure, when a system administrator or an automated operation and maintenance process decides to perform a snapshot operation on a data set or volume group, the central control node or data management service will initiate a global snapshot request. This request contains detailed information about the snapshot task, such as the name of the snapshot, the planned execution time, the scope of the target data set, and the snapshot type (such as full snapshot, incremental snapshot, etc.). Subsequently, the snapshot request is broadcast to all relevant client nodes in the system through an efficient and reliable communication mechanism (such as a message queue, a broadcast message, or a direct RPC call). The client here can be a server, a virtual machine instance, a container directly connected to the storage volume, or any component that has access to and may read and write the target data.
[0205] Step 302, receiving the snapshot reply message sent by the client, and after receiving all the snapshot reply messages from the client, incrementing the dataset version serial number, wherein the dataset version serial number is the time version serial number of writing the dataset in the distributed system.
[0206] In an embodiment of the present disclosure, in a distributed storage system, a data storage management component is responsible for coordinating and supervising the entire process of snapshot operations to ensure accurate increment of a data set version sequence number (DGS) to maintain data consistency and clear tracking of historical versions.
[0207] The data storage management component sends snapshot requests to all clients involved in the snapshot operation. After receiving the request, each client will perform a local data consistency check, prepare for snapshot creation, and send a confirmation message to the data storage management component that the snapshot preparation is complete. This step ensures that all participating nodes are ready to avoid data inconsistency or loss during the snapshot process.
[0208] The data storage management component monitors and collects snapshot reply messages from all clients. These messages usually contain the execution status (success or failure) of the snapshot creation, the time taken, and possible exception information. By aggregating these feedbacks, the management component can fully understand the overall progress of the snapshot operation.
[0209] After receiving snapshot replies from all clients, the data storage management component performs a comprehensive consistency verification. This includes confirming that all snapshots were created successfully, with no missed or failed reports. If any client reports that snapshot creation failed, the system may need to perform a rollback or retry mechanism to ensure data integrity.
[0210] Once the snapshot operations of all clients are verified to be completed successfully, the data storage management component will safely perform the increment operation of the dataset version sequence number (DGS). The increase of DGS marks the birth of a new dataset version. It not only records the completion of the snapshot, but also serves as a timestamp mark for the evolution history of the dataset. Incrementing DGS is a global operation, which requires ensuring that it is updated synchronously in all relevant components of the distributed system to avoid version confusion.
[0211] After incrementing the DGS, the data storage management component will update the system metadata, record the new DGS value, and may notify the administrator or related services through logs, monitoring panels, or emails to inform them of the successful DGS update and the availability of the new snapshot. At the same time, the system may also release or adjust temporary resources used during the snapshot creation process to optimize resource utilization.
[0212] Step 303: Receive the serial number confirmation request sent by the client, and reply to the client with the serial number of the current data set version.
[0213] In an embodiment of the present disclosure, in a distributed storage system, maintaining the accuracy and timeliness of the data set version sequence number (DGS) is the key to ensuring data consistency and management efficiency. The data storage management component receives a sequence number confirmation request sent by the client over the network. This request usually contains a client identifier, a target data set identifier, and possible authentication information. The system immediately verifies the request to ensure that the request comes from an authorized client and checks whether the format and content of the request meet the expected standards.
[0214] After verifying that the request is valid, the data storage management component turns to internal status checks, especially the current version sequence number of the target dataset. This step involves interacting with the metadata storage system to quickly find and extract the latest DGS value of the dataset. The metadata storage system maintains version information for all datasets and is the core of ensuring accurate data version control.
[0215] After obtaining the current dataset version serial number, the management component constructs a reply message. The message contains at least two key elements: one is a confirmation flag, indicating that the request has been successfully processed; the other is the current DGS value, which is the information that the client cares about most and indicates the latest version status of the dataset. In addition, the reply may also contain other auxiliary information, such as timestamps, operation status codes, etc., for further processing by the client.
[0216] Further, as a refinement of the above embodiment, the data storage management component includes: a snapshot coordination component, wherein the snapshot coordination component is used to assist the data storage management component in performing snapshot processing on the client;
[0217] The receiving the snapshot reply message sent by the client, and after receiving all the snapshot reply messages from the client, incrementing the data set version serial number, includes:
[0218] After receiving the snapshot reply messages from all the clients, the snapshot coordination component records the snapshot information and determines that the snapshot creation is completed;
[0219] After it is determined that the snapshot is created, the data set version serial number is incremented.
[0220] Specifically, in the embodiment of the present disclosure, in order to better coordinate the process of snapshot creation, a snapshot coordination component is added in the embodiment of the present disclosure, and the component provides a unified snapshot data version consensus for a volume (Volume, Vol) or a consistency group. Fig.14A system architecture diagram of a distributed system provided in an embodiment of the present disclosure, the snapshot coordination component (module) first initiates a snapshot creation instruction to all participating clients. Each client completes the consistency freeze of the data state locally, that is, generates a snapshot, and sends the result of this operation - the snapshot reply message, back to the snapshot coordination component. These replies contain the success status of the snapshot creation, timestamp, and possible error information.
[0221] After receiving the snapshot reply messages from all clients, the snapshot coordination component will perform summary analysis to verify whether all participating nodes have successfully completed the snapshot creation. Once it is confirmed that the snapshot operations of all clients have been successfully completed, the coordination component will record the summarized snapshot information in the system metadata database. This information includes but is not limited to the creation time of the snapshot, the list of participating clients, the data set range, and the snapshot version of each data shard, etc., providing the necessary basis for subsequent management and recovery operations.
[0222] Furthermore, as an achievable manner of the embodiment of the present disclosure, the method further includes:
[0223] Receive the clone reply message and the rollback reply message sent by the client, and after receiving the clone reply message and the rollback reply message, increment the data set version serial number.
[0224] Specifically, in the embodiments of the present disclosure, a snapshot is essentially a data set based on a time version. Consider linking each written data to a time version (not an absolute time, but used to indicate the order). We call this time version the data set version serial number. The DGS of each data written on a volume or group is determined by the current DGS of its volume or group information. In the initial state, the DGS of a volume or group when it is just created is 0. The DGS will increase after certain operations. The snapshot operation will cause the DGS to increase by 1, and the cloning and rollback will also increase. The DGS will eventually be recorded in the local disk of the storage node along with the write path.
[0225] It should be noted that the embodiments of the present disclosure may include multiple steps. For the convenience of description, these steps are numbered, but these numbers do not limit the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.
[0226] Corresponding to the above-mentioned snapshot-based generation method, the present invention also proposes a snapshot generation device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.
[0227] Fig.15A schematic diagram of a snapshot generation device provided in an embodiment of the present disclosure, wherein the device is applied to a client, such as Fig.15 As shown, including:
[0228] A generating unit 41 is used to combine the first write request sent by the service application with the data set version serial number to generate a second write request, wherein the data set version serial number is a time version serial number of writing the data set in the distributed system;
[0229] a first sending unit 42, configured to send the second write request to a corresponding storage unit, so as to store the write data set corresponding to the first write request in the storage unit;
[0230] The first creation unit 43 is used to respond to the snapshot request, use the data set version serial number to create a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and send snapshot reply information to the data storage management component after the snapshot is created;
[0231] The first determination unit 44 is used to send a serial number confirmation request to the data storage management component to determine whether the data set version serial number has changed; after the data set version serial number has changed, send a stop instruction to the target storage unit to stop creating a snapshot of the target storage unit.
[0232] The present disclosure provides a device for snapshot generation, which combines a first write request sent by a service application with a data set version serial number to generate a second write request, wherein the data set version serial number is a time version serial number of a data set written in a distributed system; sends the second write request to a corresponding storage unit so as to store the written data set corresponding to the first write request in the storage unit; in response to the snapshot request, uses the data set version serial number to create a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and after the snapshot creation is performed, sends a snapshot reply message to a data storage management component; sends a sequence number confirmation request to the data storage management component to determine whether the data set version serial number has changed; after the data set version serial number has changed, sends a stop instruction to the target storage unit to stop creating a snapshot of the target storage unit. Compared with the related art, the embodiment of the present disclosure ensures the consistency of the written data with the current data set version by combining the write request of the service application with the data set version serial number to generate the second write request. The dataset version serial number serves as a timestamp, reflecting the latest status of the dataset, which helps avoid data conflicts and inconsistencies, especially in high-concurrency write scenarios. By sending serial number confirmation requests to the data storage management component, the changes in the dataset version serial number are dynamically monitored, and timely responses can be made after the version is updated, such as stopping unnecessary snapshot creation, avoiding snapshots based on outdated data, and ensuring the timeliness and consistency of the data. After detecting a change in the dataset version serial number, a stop command is immediately sent to the target storage unit, effectively preventing resource waste and avoiding client I / O blocking.
[0233] Furthermore, in a possible implementation of this embodiment, as Fig.16 As shown, the data storage management component includes: a snapshot coordination component, wherein the snapshot coordination component is used to assist the data storage management component in performing snapshot processing on the client;
[0234] The device also includes:
[0235] A monitoring unit 45 is configured to monitor a processing time for creating a snapshot of a target storage unit where target data corresponding to the snapshot request is located after the creation unit creates a snapshot of the target storage unit in response to the snapshot request;
[0236] The second sending unit 46 is configured to send snapshot failure information to the snapshot coordination component and stop creating a snapshot for the target storage unit when it is determined that the processing time exceeds a timeout threshold.
[0237] Furthermore, in a possible implementation of this embodiment, as Fig.16 As shown, the first creation unit 43 includes:
[0238] A generating module 431, configured to generate a data sequence range of the target data based on the current data set version sequence number in response to the snapshot request sent by the data storage management component, wherein the data sequence range is composed of a starting data set version sequence number and an ending data set version sequence number of the data in the storage unit;
[0239] The creation module 432 is used to create a snapshot of the data blocks in the target storage unit according to the current snapshot creation mode, wherein the created snapshot includes the data sequence range and reference relationship; and after the snapshot is created, a snapshot reply message is sent to the data storage management component.
[0240] Furthermore, in a possible implementation of this embodiment, the creation module 432 is further configured to:
[0241] In the case where the client includes a data block index service, if the current snapshot creation mode is a write-time redirection mode, based on the data block index service, a first data block is created for a write data set received when creating the snapshot and redirection processing is performed on the first data block;
[0242] If the current snapshot creation mode is the copy-on-write mode, then based on the data block index service, determining a target data block corresponding to the write data set received when creating the snapshot, and creating a second data block, so as to copy the data in the target data block to the second data block;
[0243] In a case where the client does not include a data block index service, the second write request is forwarded to the data block index service; the data block index service compares the data set version serial number in the second write request with the data set version serial number of the data block in the target storage unit to determine whether to create the first data block or the second data block according to the current snapshot creation mode.
[0244] Furthermore, in a possible implementation of this embodiment, as Fig.16 As shown, the device also includes:
[0245] The second creation unit 47 is used to, before creating a snapshot, if a snapshot is deleted, use the data corresponding to the data sequence range of the deleted snapshot and the data corresponding to the version serial number of the current data set as target data for snapshot creation, and create a snapshot of the storage unit where the target data is located.
[0246] Furthermore, in a possible implementation of this embodiment, as Fig.16 As shown, the device also includes:
[0247] A second determining unit 48 is configured to determine, in response to the cloning request, a target clone storage unit that needs to be cloned, and determine a target snapshot corresponding to the target clone storage unit;
[0248] The cloning unit 49 is used to obtain the termination data set version serial number in the data sequence range of the target snapshot, and after incrementing the termination data set version serial number, mount the clone storage unit; and clone the target source data to the clone storage unit;
[0249] The third sending unit 410 is used to send a clone reply message to the data storage management component after completing the cloning process.
[0250] Furthermore, in a possible implementation of this embodiment, as Fig.16 As shown, the device also includes:
[0251] A fourth determining unit 411 is configured to determine, in response to the rollback request, a rollback snapshot corresponding to the rollback process, and determine a range of dataset version serial numbers that need to be covered based on the current dataset version serial number and the rollback snapshot;
[0252] A deleting unit 412, configured to determine target coverage data corresponding to the data set version serial number range, and delete the target coverage data;
[0253] The fourth sending unit 413 is used to send a rollback reply message to the data storage management component after completing the rollback process.
[0254] Fig.17 A schematic diagram of a device for generating a snapshot according to an embodiment of the present disclosure is provided. The device is applied to a storage unit. The storage unit includes: a data block, such as Fig.17 As shown, including:
[0255] The storage unit 51 is used to receive the second write request sent by the client, read the first write request in the second write request, and store the write data set corresponding to the first write request and the data set version serial number in the second write request together, wherein the data set version serial number is a time version serial number of writing the data set in the distributed system;
[0256] A creation unit 52, configured to determine, in response to a snapshot request, a target data block where target data corresponding to the snapshot request is located, and create a snapshot of the target data block based on a current snapshot creation mode;
[0257] The receiving unit 53 is configured to receive a stop instruction sent by the client and stop creating a snapshot of the target data.
[0258] Furthermore, in a possible implementation of this embodiment, as Fig.17 As shown, the creation unit 52 includes:
[0259] A first creation module 521, configured to store the write data set received when creating the snapshot into the first data block created by the data block index service when the current snapshot creation mode is the write-time redirection mode;
[0260] The second creation module 522 is used to determine the target data block corresponding to the write data set received when creating the snapshot when the current snapshot creation mode is the write-time copy mode, copy the data in the target data block to the second data block, and store the write data set in the target data block.
[0261] Fig.18 A schematic diagram of a device for generating a snapshot according to an embodiment of the present disclosure is provided. The device is applied to a data storage management component, such as Fig.18 As shown, including:
[0262] A sending unit 61, configured to send the snapshot request to all clients;
[0263] The first receiving unit 62 is used to receive the snapshot reply message sent by the client, and after receiving all the snapshot reply messages of the client, increment the data set version serial number, wherein the data set version serial number is the time version serial number of writing the data set in the distributed system;
[0264] The reply unit 63 is used to receive the serial number confirmation request sent by the client and reply the serial number of the current data set version to the client.
[0265] Furthermore, in a possible implementation of this embodiment, as Fig.18 As shown, the data storage management component includes: a snapshot coordination component, wherein the snapshot coordination component is used to assist the data storage management component in performing snapshot processing on the client;
[0266] The first receiving unit 62 includes:
[0267] Determination module 621, configured to record snapshot information and determine that snapshot creation is completed after the snapshot coordination component receives snapshot reply messages from all the clients;
[0268] The processing module 622 is used to increment the version serial number of the data set after determining that the snapshot creation is completed.
[0269] Furthermore, in a possible implementation of this embodiment, as Fig.18As shown, the device also includes:
[0270] The second receiving unit 64 is configured to receive the clone reply message and the rollback reply message sent by the client, and after receiving the clone reply message and the rollback reply message, increment the data set version serial number.
[0271] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and the principle is the same, which is no longer limited in this embodiment.
[0272] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0273] Fig.19 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0274] like Fig.19 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 702 or a computer program loaded from a storage unit 708 to a RAM (Random Access Memory) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An I / O (Input / Output) interface 705 is also connected to the bus 704.
[0275] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0276] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the method of snapshot generation. For example, in some embodiments, the method of snapshot generation may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the aforementioned snapshot generation method in any other appropriate manner (eg, by means of firmware).
[0277] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a special purpose or general purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0278] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0279] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory) or a flash memory, an optical fiber, a CD-ROM (Compact Dis sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0280] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0281] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.
[0282] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0283] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0284] The various numerical numbers such as first and second involved in the present disclosure are only for the convenience of description and are not used to limit the scope of the embodiments of the present disclosure, but also indicate the order of precedence.
[0285] At least one in the present disclosure may also be described as one or more, and a plurality may be two, three, four or more, which is not limited in the present disclosure. In the embodiments of the present disclosure, for a technical feature, the technical features in the technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", etc., and there is no order of precedence or size between the technical features described by the "first", "second", "third", "A", "B", "C" and "D".
[0286] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0287] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for generating a snapshot, characterized in that: The method is applied to a client, and the method comprises: Combining the first write request sent by the service application with the data set version serial number to generate a second write request, wherein the data set version serial number is a time version serial number of writing the data set in the distributed system; sending the second write request to a corresponding storage unit, so as to store the write data set corresponding to the first write request in the storage unit; In response to the snapshot request, using the data set version serial number, creating a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and after the snapshot creation is completed, sending snapshot reply information to the data storage management component; A serial number confirmation request is sent to the data storage management component to determine whether the data set version serial number has changed; after the data set version serial number has changed, a stop instruction is sent to the target storage unit to stop creating a snapshot of the target storage unit.
2. The method according to claim 1, characterized in that The data storage management component includes: a snapshot coordination component, wherein the snapshot coordination component is used to assist the data storage management component in performing snapshot processing on the client; After creating a snapshot of the target storage unit where the target data corresponding to the snapshot request is located in response to the snapshot request, the method further includes: Monitoring the processing time of creating a snapshot of the target storage unit; When it is determined that the processing time exceeds a timeout threshold, snapshot failure information is sent to the snapshot coordination component, and snapshot creation for the target storage unit is stopped.
3. The method according to claim 1, characterized in that The method of responding to the snapshot request, using the data set version serial number, creating a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and sending snapshot reply information to the data storage management component after the snapshot is created, includes: In response to the snapshot request sent by the data storage management component, generating a data sequence range of the target data based on the current data set version sequence number, wherein the data sequence range is composed of a starting data set version sequence number and an ending data set version sequence number of the data in the storage unit; According to the current snapshot creation mode, a snapshot is created for the data blocks in the target storage unit, wherein the created snapshot includes the data sequence range and the reference relationship; and after the snapshot is created, snapshot reply information is sent to the data storage management component.
4. The method according to claim 3, characterized in that The step of creating a snapshot of the data block in the target storage unit according to the current snapshot creation mode includes: In the case where the client includes a data block index service, if the current snapshot creation mode is a write-time redirection mode, based on the data block index service, a first data block is created for a write data set received when creating the snapshot and redirection processing is performed on the first data block; If the current snapshot creation mode is the copy-on-write mode, then based on the data block index service, determining a target data block corresponding to the write data set received when creating the snapshot, and creating a second data block, so as to copy the data in the target data block to the second data block; In a case where the client does not include a data block index service, the second write request is forwarded to the data block index service; the data block index service compares the data set version serial number in the second write request with the data set version serial number of the data block in the target storage unit to determine whether to create the first data block or the second data block according to the current snapshot creation mode.
5. The method according to claim 1, characterized in that The method further comprises: Before creating a snapshot, if a snapshot is deleted, the data corresponding to the data sequence range of the deleted snapshot and the data corresponding to the current data set version serial number are used as the target data for snapshot creation, and a snapshot is created for the storage unit where the target data is located.
6. The method according to claim 1, characterized in that The method further comprises: In response to the cloning request, determining a target clone storage unit that needs to be cloned, and determining a target snapshot corresponding to the target clone storage unit; Obtaining a termination data set version serial number in the data sequence range of the target snapshot, incrementing the termination data set version serial number, and mounting a clone storage unit; and cloning the target source data to the clone storage unit; After the cloning process is completed, a clone reply message is sent to the data storage management component.
7. The method according to claim 1, characterized in that The method further comprises: In response to the rollback request, determine the rollback snapshot corresponding to the rollback process, and determine the range of dataset version serial numbers that need to be covered based on the current dataset version serial number and the rollback snapshot; Determine the target coverage data corresponding to the data set version serial number range, and delete the target coverage data; After the rollback process is completed, a rollback reply message is sent to the data storage management component.
8. A method for generating a snapshot, characterized in that: The method is applied to a storage unit, the storage unit comprising: a data block, and the method comprises: Receive a second write request sent by the client, read the first write request in the second write request, and store the write data set corresponding to the first write request and the data set version serial number in the second write request together, wherein the data set version serial number is a time version serial number of writing the data set in the distributed system; In response to a snapshot request, determining a target data block where target data corresponding to the snapshot request is located, and creating a snapshot of the target data block based on a current snapshot creation mode; A stop instruction sent by the client is received, and snapshot creation of the target data is stopped.
9. The method according to claim 8, characterized in that The step of determining, in response to the snapshot request, a target data block where the target data corresponding to the snapshot request is located, and creating a snapshot of the target data block based on a current snapshot creation mode includes: If the current snapshot creation mode is the write-time redirection mode, the write data set received when the snapshot is created is stored in the first data block created by the data block index service; If the current snapshot creation mode is the copy-on-write mode, the target data block corresponding to the write data set received when creating the snapshot is determined, the data in the target data block is copied to the second data block, and the write data set is stored in the target data block.
10. A method for generating a snapshot, characterized in that: The method is applied to a data storage management component, and the method comprises: Send snapshot requests to all clients; Receive the snapshot reply message sent by the client, and after receiving all the snapshot reply messages from the client, increment the dataset version serial number, wherein the dataset version serial number is the time version serial number of writing the dataset in the distributed system; Receive the serial number confirmation request sent by the client, and reply to the client with the current data set version serial number.
11. The method according to claim 10, characterized in that The data storage management component includes: a snapshot coordination component, wherein the snapshot coordination component is used to assist the data storage management component in performing snapshot processing on the client; The receiving the snapshot reply message sent by the client, and after receiving all the snapshot reply messages from the client, incrementing the data set version serial number, includes: After receiving the snapshot reply messages from all the clients, the snapshot coordination component records the snapshot information and determines that the snapshot creation is completed; After it is determined that the snapshot is created, the data set version serial number is incremented.
12. The method according to claim 10, characterized in that The method further comprises: Receive the clone reply message and the rollback reply message sent by the client, and after receiving the clone reply message and the rollback reply message, increment the data set version serial number.
13. A snapshot generation device, characterized in that: The device is applied to a client, and comprises: A generating unit, configured to combine a first write request sent by a service application with a data set version serial number to generate a second write request, wherein the data set version serial number is a time version serial number of writing a data set in a distributed system; a first sending unit, configured to send the second write request to a corresponding storage unit, so as to store a write data set corresponding to the first write request in the storage unit; A first creation unit is used to respond to the snapshot request, use the data set version serial number to create a snapshot of the target storage unit where the target data corresponding to the snapshot request is located, and send snapshot reply information to the data storage management component after the snapshot is created; The first determination unit is used to send a serial number confirmation request to the data storage management component to determine whether the data set version serial number has changed; after the data set version serial number has changed, a stop instruction is sent to the target storage unit to stop creating a snapshot of the target storage unit.
14. A device for generating a snapshot, characterized in that: The device is applied to a storage unit, the storage unit includes: a data block, and the device includes: A storage unit, used to receive a second write request sent by a client, read the first write request in the second write request, and store the write data set corresponding to the first write request and the data set version serial number in the second write request together, wherein the data set version serial number is a time version serial number of writing the data set in the distributed system; a creating unit, configured to, in response to a snapshot request, determine a target data block where target data corresponding to the snapshot request is located, and create a snapshot of the target data block based on a current snapshot creation mode; The receiving unit is used to receive a stop instruction sent by the client and stop creating a snapshot of the target data.
15. A device for generating a snapshot, characterized in that: The device is applied to a data storage management component, and the device comprises: A sending unit, used for sending the snapshot request to all clients; A first receiving unit is used to receive the snapshot reply message sent by the client, and after receiving all the snapshot reply messages from the client, increment the data set version serial number, wherein the data set version serial number is a time version serial number of writing the data set in the distributed system; The reply unit is used to receive the serial number confirmation request sent by the client and reply the serial number of the current data set version to the client.
16. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.
17. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.
18. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Snapshot method for distributed storage
CN107798130A
Snapshot rollback method and apparatus
CN108255638A
Method and device for creating consistent snapshot group and computer readable storage medium
CN109582496A
Data snapshot method and device, electronic equipment and computer readable storage medium
CN112559243A
Snapshot creation method, computing device and storage medium
CN114328018A
Cited By
Data processing method and corresponding apparatus
WO2026144277A1
