Snapshot information processing method, distributed storage system, device, medium and product

Abstract

The invention discloses a snapshot information processing method, a distributed storage system, a device, a medium and a product, and relates to the technical field of data processing. The first incremental snapshot information is determined based on the first snapshot information generated based on the update request of the target object and the second snapshot information pre-stored by the target object, the first incremental snapshot information reduces retransmission of most of the snapshot information, and the transmission rate is improved. The second incremental snapshot information is determined based on the locally stored snapshot information in the same directory to which the target object belongs and the first incremental snapshot information, and the situation that the network load is increased due to the fact that the snapshot information of other target objects is obtained from the client side again is avoided through update storage of the snapshot information of the other target objects. In the same directory, the second incremental snapshot information is further determined based on the snapshot information under the directory, the snapshot information is ensured to be as simple as possible, the network load problem caused by acquisition from the client is reduced, and the performance and efficiency of the storage system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for processing snapshot information, a distributed storage system, a device, a medium, and a product. Background Art

[0002] The distributed storage system is one of the key technologies in the current big data processing and cloud computing fields. The transmission and caching of snapshot information have become important influencing factors for improving the performance of the distributed storage system. Conventionally, when an object is updated and modified, the system generates snapshot information of the object's state for transmission to the storage pool. However, during the transmission process, each time an object is modified, the full snapshot information is transmitted and stored, resulting in low transmission efficiency. Moreover, each update request in the storage pool needs to carry the full snapshot list, leading to an increase in network load during transmission and reducing the system's response speed.

[0003] Therefore, how to improve the transmission efficiency of snapshot information and the system response speed when an object is modified is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for processing snapshot information, a distributed storage system, a device, a medium, and a product to solve the problem of low transmission efficiency and increased network load caused by the transmission of full snapshot information when a conventional object is modified.

[0005] To solve the above technical problems, the present invention provides a method for processing snapshot information, including: Responding to an update request of a target object issued by a client and generating corresponding first snapshot information according to the update request; Reading second snapshot information pre-stored for the target object and determining first incremental snapshot information according to the first snapshot information and the second snapshot information; Determining second incremental snapshot information based on the snapshot information locally stored under the same directory to which the target object belongs and the first incremental snapshot information; Storing the second incremental snapshot information into the storage pool to perform an update process on the target object.

[0006] On the one hand, determining first incremental snapshot information according to the first snapshot information and the second snapshot information includes: Obtaining a first lower limit identifier in the first snapshot information and a second upper limit identifier in the second snapshot information; Determining first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier; Determining the first incremental snapshot information according to the first updated snapshot information.

[0007] On the other hand, determining the first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier includes: In the case where the first lower limit identifier is greater than the second upper limit identifier, it is determined that there is no intersection between the first snapshot information and the second snapshot information, and the first snapshot information is the first updated snapshot information; In the case where the first lower limit identifier is less than or equal to the second upper limit identifier, it is determined that there is an intersection between the first snapshot information and the second snapshot information, and a partial snapshot information of the first snapshot information is the first updated snapshot information.

[0008] On the other hand, determining the first incremental snapshot information based on the first updated snapshot information includes: In the case where the first lower limit identifier is greater than the second upper limit identifier, taking the first updated snapshot information as the first incremental snapshot information; In the case where the first lower limit identifier is less than or equal to the second upper limit identifier, obtaining the first upper limit identifier in the first snapshot information and the second lower limit identifier in the second snapshot information; Determining second updated snapshot information based on the first upper limit identifier and the second lower limit identifier, and determining the first incremental snapshot information based on the second updated snapshot information.

[0009] On the other hand, determining the second incremental snapshot information based on the snapshot information locally stored under the same directory to which the target object belongs and the first incremental snapshot information includes: When there is an update of other target objects under the same directory to which the target object belongs, when the identifier of the snapshot information locally stored is the same as the identifier of the second snapshot information, and the first lower limit identifier is greater than the second upper limit identifier, taking the first incremental snapshot information as the second incremental snapshot information; When the identifier of the snapshot information locally stored is the same as the identifier of the second snapshot information, and the first lower limit identifier is less than or equal to the second upper limit identifier, comparing the identifier of the first incremental snapshot information with the identifier of the snapshot information locally stored; If the identifier of the first incremental snapshot information is included in the identifier of the snapshot information locally stored, calling the snapshot information locally stored as the second incremental snapshot information; If a part of the identifier of the first incremental snapshot information is included in the identifier of the snapshot information locally stored, fusing the first incremental snapshot information and the snapshot information locally stored as the second incremental snapshot information.

[0010] On the other hand, when the identifier of the snapshot information stored locally is different from the identifier of the second snapshot information, the method further includes: Obtaining a third upper limit identifier corresponding to the snapshot information stored locally; Determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier.

[0011] On the other hand, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is less than or equal to the third upper limit identifier, determining a missing target identifier according to the identifier of the snapshot information stored locally and the identifier of the first snapshot information; using the snapshot information corresponding to the target identifier as the second incremental snapshot information.

[0012] On the other hand, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, filling the identifier of the snapshot information stored locally into the identifier corresponding to the first snapshot information of the update request to determine the second incremental snapshot information.

[0013] On the other hand, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first upper limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, using the first snapshot information as the second incremental snapshot information.

[0014] On the other hand, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first upper limit identifier is greater than the second upper limit identifier and the first lower limit identifier is less than or equal to the third upper limit identifier and the first upper limit identifier is greater than the third upper limit identifier, updating the first snapshot information corresponding to the update request to the snapshot information stored locally to use the updated snapshot information stored locally as the second incremental snapshot information.

[0015] On the other hand, when the target object is the first object, the second snapshot information is full - volume snapshot information, and the full - volume snapshot information is stored in the storage pool.

[0016] To solve the above technical problems, the present invention further provides a distributed storage system, including a master node and at least one storage node; The master node is connected to at least one storage node, and the master node is configured to execute the steps of the snapshot information processing method described above to complete the update of the snapshot information.

[0017] To solve the above technical problems, the present invention further provides a snapshot information processing device, including: A memory for storing a computer program; A processor for implementing the steps of the snapshot information processing method as described when executing the computer program.

[0018] To solve the above technical problems, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the snapshot information processing method as described.

[0019] To solve the above technical problems, the present invention further provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the snapshot information processing method are implemented.

[0020] The beneficial effects of the present invention are as follows. On the one hand, the first incremental snapshot information is determined based on the first snapshot information generated by the update request of the target object and the second snapshot information pre-stored in the target object. When the target object update request is made, only the first incremental snapshot information is determined, avoiding the subsequent transmission of the full snapshot information of the target object as long as there is an update request in the conventional technical solution. Compared with the entire snapshot list, the first incremental snapshot information reduces the retransmission of most of the snapshot information, reduces the data volume, and also facilitates the subsequent improvement of the transmission rate. On the other hand, the second incremental snapshot information is determined based on the snapshot information locally stored under the same directory as the target object and the first incremental snapshot information, considering that the snapshot information of other target objects under the same directory as the target object has been transmitted to the local cache before. If they are in the same directory, the update storage of the snapshot information of other target objects is used to avoid the increase in network load caused by re-obtaining from the client. In the same directory, based on the snapshot information under the directory, the second incremental snapshot information is further determined, ensuring that the snapshot information is as concise as possible while reducing the network load problem caused by obtaining from the client, and directly obtaining the final incremental snapshot information from the local cache to achieve cache optimization and improve the performance and efficiency of the storage system.

[0021] Secondly, by determining the first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier, the data range between two snapshots can be quickly located without a full scan of the entire data set. This significantly reduces the computational overhead. At the same time, using the upper and lower limit identifiers as boundary conditions helps the system quickly determine which data blocks or objects may have changed. By accurately determining the incremental range, the system can capture only the data part that has changed between two snapshots without backing up the unchanged data, which significantly reduces the storage space occupancy. Based on the comparison of the upper and lower limit identifiers of the two types of snapshot information, it is possible to avoid scanning the entire data during the data scanning process and quickly determine the incremental range. It can accurately capture the changes at the time point between two snapshots, thus ensuring consistency during data recovery, being able to quickly capture data changes, and reducing the risk of data loss caused by system failures or unexpected interruptions. During the determination process of the first incremental snapshot information, after comparing the first lower limit identifier and the second upper limit identifier, in different cases, the first upper limit identifier and the second lower limit identifier are further compared to narrow the constraint range while further ensuring the accuracy of the incremental snapshot information. During the determination process of the second incremental snapshot information, there is a prerequisite that when there are updates to other target objects in the same directory to which the target object belongs and the identifier of the snapshot information stored locally is the same as the identifier of the second snapshot information, on this basis, redundant data is further reduced to ensure that while streamlining the incremental data, the snapshot information stored locally can be called nearby, reducing the network load and improving the data transmission efficiency.

[0022] In addition, the present invention also provides a distributed storage system, a snapshot information processing device, a computer-readable storage medium, and a computer program product, which have the same beneficial effects as the above-mentioned snapshot information processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] To more clearly illustrate the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0024] Figure 1 It is a flowchart of a snapshot information processing method provided by an embodiment of the present invention; Figure 2 It is a flowchart of another snapshot information processing method provided by an embodiment of the present invention; Figure 3 It is a schematic structural diagram of a distributed storage system provided by an embodiment of the present invention; Figure 4 It is a structural diagram of a snapshot information processing device provided by an embodiment of the present invention; Figure 5 This is a structural diagram of a snapshot information processing device provided by an embodiment of the present invention. Detailed implementation manners

[0025] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present invention.

[0026] The core of the present invention is to provide a snapshot information processing method, a distributed storage system, a device, a medium and a product to solve the problems of low transmission efficiency and increased network load caused by the transmission of full - volume snapshot information when a conventional object is modified.

[0027] To enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0028] The distributed storage system is one of the key technologies in the current fields of big data processing and cloud computing. With the continuous growth of the data volume, the performance and efficiency of the storage system have become an important bottleneck. The snapshot technology is an important means for data protection and recovery in the distributed storage system, but the transmission and caching of snapshot information have become important factors affecting the system performance. The existing snapshot information transmission methods usually adopt the full - volume transmission method, which will lead to low transmission efficiency when the number of snapshots is large, further affecting the overall performance of the system. In addition, there are also problems with the caching of snapshot information in the storage pool. Each request needs to carry a full - volume snapshot list, which not only increases the network load but also reduces the system response speed. The snapshot information processing method provided by the present invention can solve the above - mentioned technical problems.

[0029] Figure 1 This is a flowchart of a snapshot information processing method provided by an embodiment of the present invention. As Figure 1 shown, the method includes: S11: Respond to the update request of the target object issued by the client and generate corresponding first snapshot information according to the update request; S12: Read the second snapshot information pre - stored for the target object and determine the first incremental snapshot information according to the first snapshot information and the second snapshot information; S13: Determine the second incremental snapshot information based on the snapshot information locally stored under the same directory to which the target object belongs and the first incremental snapshot information; S14: Store the second incremental snapshot information into the storage pool to perform update processing on the target object.

[0030] Specifically, when the target object is modified, it is necessary to respond to the update request of the target object sent by the client. The client is a user of the distributed storage system and runs on the user's device or application program, responsible for initiating various requests to the distributed storage system, including operations such as reading, writing, updating, and deleting. The target object refers to the specific data object that the client hopes to operate on. In a distributed storage system, data is usually stored in the form of objects, such as files, data blocks, key-value pairs, etc. The target object can be a part of a file, the value in a key-value pair, or a record in a database. The update request refers to the operation initiated by the client to modify the target object, which can be the identifier of the target object, the updated content, or metadata, etc., and is not limited here. Responding to the request can be that the master node or storage node of the distributed storage system receives the update request from the client, or it can be a verification request to execute the update operation.

[0031] In a distributed storage system, when the client initiates an update request, the system needs to generate snapshot information corresponding to the update request while performing the update operation. The purpose is to capture the current state before the data is updated so that it can be restored to the state before the update when needed. Regarding how to generate the first snapshot information, first generate a snapshot request to capture the current state of the target object, mainly including the identifier of the target object, the snapshot timestamp, the snapshot version number, and the current data state. The system needs to capture the current state of the target object before the update, which can be captured through strategies such as Copy-on-Write (COW) or Redirect-on-Write (ROW). The COW strategy is that when the update request arrives, the system will copy the current data of the target object to a new storage location, and this copied data is the snapshot content. The ROW strategy is that the system will record the metadata of the current state of the target object, such as the pointer of the data block, the offset of the file, etc. When performing the update operation, the system will write the new data to a new location and update the pointer. The snapshot information will point to the old data location before the update.

[0032] The second snapshot information pre-stored for the target object in step S12 is the second snapshot information that has been stored before the target object is modified. Determine the first incremental snapshot information based on the first snapshot information and the second snapshot information. The determination process can capture the state of the snapshot information, such as the identifier of the data block, the incremental data, and the metadata, by comparing the pointers or hash values, version numbers, or timestamps of the data blocks in the two snapshot information. It can also be through the corresponding identifier in the snapshot information, such as the modification identifier corresponding to the object. The snapshot information stores the sequence number corresponding to the snapshot identifier representing the object, and the first incremental snapshot information can be determined by comparing the identifier sequence numbers corresponding to the respective snapshot information.

[0033] The determination of the second incremental snapshot information in step S13, compared with the conventional method of only determining the incremental snapshot information, also takes into account the snapshot information corresponding to the client when caching locally. Therefore, in this case, first check whether the snapshot information stored locally corresponding to the target object in the same directory has been updated. If it has been updated, the snapshot information stored locally in the same directory can be fully called as the final second incremental snapshot information. If the snapshot information stored locally in the directory to which the target object belongs is different from the first incremental snapshot information, the first incremental snapshot information corresponding to the client needs to be re-obtained. In addition, there is also a situation where if the snapshot information corresponding to the local storage fails to be updated in a timely manner due to a node failure, the first incremental snapshot information corresponding to the client needs to be obtained as the second incremental snapshot information at this time. This is to illustrate that in the same directory, the snapshot information cached this time is directly called to reduce the network load corresponding to obtaining from the client.

[0034] In step S14, the finally generated second incremental snapshot information is stored in the storage pool to facilitate subsequent update processing of the target object.

[0035] The beneficial effects of the embodiments of the present invention are as follows. On the one hand, the first incremental snapshot information is determined based on the first snapshot information generated based on the update request of the target object and the second snapshot information pre-stored for the target object. When determining the first incremental snapshot information according to the update request of the target object, only the first incremental snapshot information is determined, avoiding the subsequent transmission of the full snapshot information of the target object as long as there is an update request in the conventional technical solution. Compared with the entire snapshot list, the first incremental snapshot information reduces the retransmission of most of the snapshot information, reduces the data volume, and also facilitates subsequent improvement of the transmission rate. On the other hand, the second incremental snapshot information is determined based on the snapshot information stored locally in the same directory as the target object and the first incremental snapshot information, considering that the snapshot information of other target objects in the same directory as the target object has been transmitted to the local cache before. If they are in the same directory, through the update storage of the snapshot information of other target objects, the situation of increased network load caused by re-obtaining from the client is avoided. In the same directory, based on the snapshot information in the directory, the second incremental snapshot information is further determined to ensure that the snapshot information is as concise as possible while reducing the network load problem caused by obtaining from the client, and directly obtain the final incremental snapshot information from the local cache to achieve cache optimization and improve the performance and efficiency of the storage system.

[0036] In some embodiments, determining the first incremental snapshot information according to the first snapshot information and the second snapshot information includes: Obtain the first lower limit identifier in the first snapshot information and the second upper limit identifier in the second snapshot information; Determine the first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier; Determine the first incremental snapshot information according to the first updated snapshot information.

[0037] Considering that there is at least one snapshot information corresponding to the snapshot information identifier, the first lower limit identifier of the first snapshot information is the smallest identifier of the first snapshot information; similarly, the second upper limit identifier of the second snapshot information is the largest identifier of the second snapshot information. Based on the comparison of the two, determine whether there is an intersection between the two snapshot informations. If there is an intersection, it means that there is partial update. If there is no intersection, it is necessary to check which snapshot information has identifiers all greater than those of the other snapshot information among the first snapshot information and the second snapshot information. If all the identifiers of the first snapshot information are greater than all the identifiers of the second snapshot information, it means that all the snapshot identifiers of the first snapshot information are updated snapshot informations. Determine the first incremental snapshot information based on the first updated snapshot information, that is, determine the first incremental snapshot information based on the above different situations.

[0038] The method of determining the first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier provided in this embodiment can quickly locate the data range between two snapshots without performing a full scan of the entire data set. This greatly reduces the computational overhead. At the same time, using the upper and lower limit identifiers as boundary conditions helps the system quickly determine which data blocks or objects may have changed. By accurately determining the incremental range, the system can only capture the data part that has changed between two snapshots without backing up the unchanged data, which significantly reduces the storage space occupancy.

[0039] In some embodiments, determining the first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier includes: In the case where the first lower limit identifier is greater than the second upper limit identifier, determine that there is no intersection between the first snapshot information and the second snapshot information, and the first snapshot information is the first updated snapshot information; In the case where the first lower limit identifier is less than or equal to the second upper limit identifier, determine that there is an intersection between the first snapshot information and the second snapshot information, and a partial snapshot information of the first snapshot information is the first updated snapshot information.

[0040] Specifically, in the case where the first lower limit identifier is greater than the second upper limit identifier, it means that there is no intersection between the two snapshot informations, and all of the first snapshot information are the latest snapshot informations. Therefore, the first snapshot information is the first updated snapshot information.

[0041] When the first lower limit identifier is less than or equal to the second upper limit identifier, it is determined that there is an intersection between the first snapshot information and the second snapshot information, and there are partial updates in the first snapshot information. In this case, part of the snapshot information of the first snapshot information needs to be used as the first updated snapshot information.

[0042] The comparison of the upper and lower limit identifiers based on the two snapshot information provided in this embodiment avoids full scanning during the data scanning process and can quickly determine the incremental range. It can accurately capture the changes at the time point between the two snapshots, thus ensuring consistency during data recovery, being able to quickly capture data changes, and reducing the risk of data loss caused by system failures or accidental interruptions.

[0043] In some embodiments, determining the first incremental snapshot information according to the first updated snapshot information includes: When the first lower limit identifier is greater than the second upper limit identifier, the first updated snapshot information is used as the first incremental snapshot information; When the first lower limit identifier is less than or equal to the second upper limit identifier, obtain the first upper limit identifier in the first snapshot information and the second lower limit identifier in the second snapshot information; Determine the second updated snapshot information according to the first upper limit identifier and the second lower limit identifier, and determine the first incremental snapshot information based on the second updated snapshot information.

[0044] Combining the two situations in the above embodiments, in the first case, directly use the first updated snapshot information as the first incremental snapshot information.

[0045] In the second case, further obtain the first upper limit identifier and the second lower limit identifier to determine the second updated snapshot information based on these two identifiers. In this process, it has been determined that there is an intersection between the two snapshot information (the first snapshot information and the second snapshot information), and there is updated snapshot information, so as to determine the first incremental snapshot information.

[0046] In the process of determining the first incremental snapshot information provided in this embodiment, after comparing the first lower limit identifier and the second upper limit identifier, continue to compare the first upper limit identifier and the second lower limit identifier in different situations, which not only narrows the constraint range but also further ensures the accuracy of the incremental snapshot information.

[0047] In some embodiments, determining the second incremental snapshot information based on the snapshot information locally stored in the same directory to which the target object belongs and the first incremental snapshot information includes: When there are updates to other target objects in the same directory to which the target object belongs, when the identifier of the snapshot information locally stored is the same as the identifier of the second snapshot information, and the first lower limit identifier is greater than the second upper limit identifier, the first incremental snapshot information is used as the second incremental snapshot information; When the identifier of the snapshot information stored locally is the same as the identifier of the second snapshot information, and the first lower limit identifier is less than or equal to the second upper limit identifier, compare the identifier of the first incremental snapshot information with the identifier of the snapshot information stored locally; If the identifier of the first incremental snapshot information is included in the identifier of the snapshot information stored locally, then call the snapshot information stored locally as the second incremental snapshot information; If a part of the identifier of the first incremental snapshot information is included in the identifier of the snapshot information stored locally, then fuse the first incremental snapshot information and the snapshot information stored locally as the second incremental snapshot information.

[0048] Specifically, in this embodiment, when considering that there are updates of other target objects in the same directory to which the target object belongs, it indicates that the identifier of the snapshot information stored locally needs to be compared at present. Usually, the identifier of the snapshot information stored locally is the same as the identifier of the second snapshot information to ensure consistency. Therefore, in this case, compare the first lower limit identifier and the second upper limit identifier. If the first lower limit identifier is greater than the second upper limit identifier, it means that the currently updated snapshot information does not exist in the local storage, and directly use the first incremental snapshot information as the final second incremental snapshot information.

[0049] If the first lower limit identifier is less than or equal to the second upper limit identifier, it means that there is an intersection between the identifier of the first incremental snapshot information and the identifier of the snapshot information stored locally, and further comparison and verification are needed to ensure the accuracy of the second incremental snapshot information.

[0050] If the identifier of the first incremental snapshot information is included in the identifier of the snapshot information stored locally, then directly call the snapshot information stored locally as the second incremental snapshot information of the target object. If a part of the identifier of the first incremental snapshot information is included in the identifier of the snapshot information stored locally, then fusion is required. The fusion process here can be to retain a complete set and delete one copy of the duplicate snapshot information. It is also possible to select the snapshot information that is not stored locally as the second incremental snapshot information.

[0051] In the process of determining the second incremental snapshot information provided in this embodiment, there is a prerequisite that when there are updates of other target objects in the same directory to which the target object belongs and the identifier of the snapshot information stored locally is the same as the identifier of the second snapshot information. On this basis, redundant data is further reduced to ensure that while streamlining the incremental data, the snapshot information stored locally is called nearby, reducing the network load and improving the data transmission efficiency.

[0052] In some other embodiments, when there is no update of other target objects in the same directory to which the target object belongs when determining the second incremental snapshot information, the first incremental snapshot information of the target object is updated to the snapshot information stored locally.

[0053] It should be noted that when there is no update of other target objects in the same directory where the target object is located, that is, the target object is the first updated object in a directory. At this time, the determined first incremental snapshot information is directly updated to the snapshot information stored locally, so as to be used as the second incremental snapshot information subsequently.

[0054] In this embodiment, when the target object is the first updated object in the directory where it is located, the first incremental snapshot information of the target object needs to be updated to the snapshot information stored locally, so that the comparison information stored locally can be directly called when other target objects in the same directory are updated subsequently.

[0055] In some other embodiments, when the identifier of the snapshot information stored locally is different from the identifier of the second snapshot information, the method further includes: According to a preset period, compare the identifier of the first snapshot information with the identifier of the snapshot information stored locally; If the identifier of the first snapshot information is different from the identifier of the snapshot information stored locally, update the identifier of the first snapshot information to the identifier of the snapshot information stored locally to complete the update process of the snapshot information stored locally.

[0056] Considering that when the identifier of the snapshot information stored locally is different from the identifier of the second snapshot information, it may be caused by a node failure of the storage node. Therefore, currently, according to a preset period, the identifier of the first snapshot information needs to be compared with the identifier of the snapshot information stored locally. When they are different, the identifier of the first snapshot information is updated to the identifier of the snapshot information stored locally to ensure that the snapshot information stored locally is the latest snapshot information.

[0057] In this embodiment, when the identifier of the snapshot information stored locally is different from the identifier of the second snapshot information, the snapshot information stored locally is updated in a timely manner to ensure that the comparison information stored locally can be directly called when other target objects in the same directory are updated subsequently, reducing the network load.

[0058] In some embodiments, when the identifier of the snapshot information stored locally is different from the identifier of the second snapshot information, the method further includes: Obtain the third upper limit identifier corresponding to the snapshot information stored locally; Determine the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier.

[0059] It should be noted that when the identifier of the snapshot information stored locally is different from the identifier of the second snapshot information, in addition to updating the snapshot information stored locally in a timely manner, it is also necessary to compare the currently locally stored snapshot information with the first snapshot information to determine the second incremental snapshot information and ensure the update process of the current target object.

[0060] Specifically, it is necessary to compare the first lower limit identifier of the first snapshot information with the third upper limit identifier of the snapshot information stored this time. Since it is necessary to distinguish two cases for comparison based on the two cases of comparing the first lower limit identifier of the first snapshot information with the second upper limit identifier of the second snapshot information, the second incremental snapshot information is determined.

[0061] In the process of determining the second incremental snapshot information provided in this embodiment, the comparison scheme of the first lower limit identifier of the first snapshot information with the third upper limit identifier of the snapshot information stored this time is carried out when the identifier of the snapshot information stored locally is different from the identifier of the second snapshot information, which improves the diversity and flexibility of the process of determining the second incremental snapshot information to ensure the update of the current target object.

[0062] In some embodiments, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is less than or equal to the third upper limit identifier, determine the missing target identifier according to the identifier of the snapshot information stored locally and the identifier of the first snapshot information; use the snapshot information corresponding to the target identifier as the second incremental snapshot information.

[0063] Specifically, when the first lower limit identifier is greater than the second upper limit identifier, it indicates that there is no intersection between the first snapshot information and the second snapshot information, and when the first lower limit identifier is less than or equal to the third upper limit identifier, it indicates that there is an intersection between the first snapshot information and the snapshot information stored locally. Then, it is necessary to determine the missing target identifier according to the identifier of the snapshot information stored locally and the identifier of the first snapshot information as the second incremental snapshot information.

[0064] In addition, the intersection relationship between the first upper limit identifier and the second upper limit identifier can be continuously judged to improve the comparison progress while ensuring the update.

[0065] In this embodiment, when there is no intersection between the first snapshot information and the second snapshot information but there is an intersection with the snapshot information stored locally, the missing part of the snapshot information in the update request is used as the updated snapshot information for local storage to improve the accuracy of the second incremental snapshot information while ensuring the update of the snapshot stored locally.

[0066] In some other embodiments, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, filling the identifier of the snapshot information stored locally into the identifier corresponding to the first snapshot information of the update request to determine the second incremental snapshot information.

[0067] Specifically, when the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, it indicates that there is no intersection among the first snapshot information, the second snapshot information, and the snapshot information stored locally. Then, filling the identifier of the snapshot information stored locally into the identifier corresponding to the first snapshot information of the update request can ensure the consistency of the snapshot information between the local and the client while improving the timeliness of the update of the second incremental snapshot information.

[0068] In some other embodiments, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first upper limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, using the first snapshot information as the second incremental snapshot information.

[0069] Specifically, when the first upper limit identifier is greater than the second upper limit identifier, it indicates that there is an intersection between the first snapshot information and the second snapshot information. Further, when the first lower limit identifier is greater than the third upper limit identifier, it indicates that there is no intersection between the first snapshot information and the snapshot information stored locally. Then, directly using the first snapshot information as the second incremental snapshot information.

[0070] In this embodiment, when there is an intersection between the first snapshot information and the second snapshot information and there is no intersection with the snapshot information stored locally, determining the second incremental snapshot information can ensure the timely update of the second incremental snapshot information while ensuring the timely update of the snapshot information stored locally.

[0071] In some other embodiments, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first upper limit identifier is greater than the second upper limit identifier, the first lower limit identifier is less than or equal to the third upper limit identifier, and the first upper limit identifier is greater than the third upper limit identifier, updating the first snapshot information corresponding to the update request to the snapshot information stored locally, and using the updated snapshot information stored locally as the second incremental snapshot information.

[0072] Specifically, when the first upper limit identifier is greater than the second upper limit identifier and the first lower limit identifier is less than or equal to the third upper limit identifier, it indicates that there is an intersection between the first snapshot information and the second snapshot information, and there is an intersection between the first snapshot information and the snapshot information stored locally. In this case, the first snapshot information needs to be directly updated to the local storage as the second incremental snapshot information.

[0073] During the determination process of the second incremental snapshot information provided in this embodiment, considering that there is an intersection between the first snapshot information and the second snapshot information, and there is an intersection between the first snapshot information and the snapshot information stored locally, there is no need to rescan. The first snapshot information is directly stored in the snapshot information stored locally to improve the update progress of the snapshot information.

[0074] In some embodiments, when the target object is the first object, the second snapshot information is the full - volume snapshot information, and the full - volume snapshot information is stored in the storage pool.

[0075] Specifically, when the first object is updated and initialized, the full - volume snapshot information needs to be used as the second snapshot information of the target object, and the full - volume snapshot information is stored in the storage pool, which is convenient for subsequent transmission and cache optimization using incremental snapshot information.

[0076] Figure 2 The flowchart of another snapshot information processing method provided for the embodiments of the present invention is as Figure 2 shown, including: S21: Respond to the write request sent by the client; S22: Parse the first snapshot information in the write request; S23: Read the second snapshot information pre - stored in the target object; S24: Determine whether the first lower limit identifier is greater than the second upper limit identifier; if so, go to step S25, if not, go to step S26; S25: Find the identifier of the snapshot information stored locally according to the directory or volume where the target object is located; S27: Determine whether the first lower limit identifier is greater than the third upper limit identifier; if so, go to step S28, if not, go to step S29; S28: Fill the third upper limit identifier stored locally into the response to request the client to resend, and return to step S21; S29: Update the identifiers of the missing partial snapshots in the write request from the local cache, and enter step S26; S26: Determine whether the first upper limit identifier is greater than the second upper limit identifier; if so, go to step S30, if not, go to step S31; S30: Find the identifier of the snapshot information stored locally according to the directory or volume where the target object is located; S31: Write the target object normally and proceed to step S36; S32: Determine whether the first lower limit identifier is greater than the third upper limit identifier; if so, proceed to step S33, if not, proceed to step S34; S33: Copy the write object and perform a write operation; S34: Determine whether the first upper limit identifier is greater than the third upper limit identifier; if so, proceed to step S35; if not, proceed to step S33; S35: Update the first snapshot information to the local and proceed to step S33; S36: Give a feedback response to the client.

[0077] Initialize the snapshot list: Initialize a snapshot list in the storage pool to store all snapshot identity (Identity, ID) (i.e., the identifier corresponding to the snapshot information).

[0078] Incremental transmission of the snapshot list: When updating an object in the storage pool, carry the latest snapshot ID and transmit it to the storage pool.

[0079] Local cache of the storage pool: The storage pool locally caches the full snapshot ID list and updates the cached snapshot ID list according to the request. When it is found that the saved snapshot list is insufficient for the modification of other objects, the snapshot list in the local cache can be used.

[0080] Request processing: When a write request is received, check the snapshot ID list carried in the request and compare it with the snapshot ID list cached locally in the storage pool and the snapshot ID list saved by the target object. If there is a changed snapshot ID, update the snapshot ID list cached locally in the storage pool. If there are many changes to the snapshot ID and the snapshot ID carried in the request is insufficient, trigger a request retransmission and require the client to send an incremental snapshot list.

[0081] Initialize the snapshot list: snapshot_list = [1, 2, 3, 4, 5]; Incremental transmission of the snapshot list: incremental_snapshot_list = [6, 7, 8]; Local cache of the storage pool: cache_snapshot_list = [1, 2, 3, 4, 5]; Save the modified object: object_snapshot_list = [1, 2, 3]; Write request: request_snapshot_list = [8]; Resend request: request_snapshot_list = [6, 7, 8]; Updated local cache of storage pool: cache_snapshot_list = [1, 2, 3, 4, 5, 6, 7, 8]; Modified object saving: object_snapshot_list = [1, 2, 3, 4, 5, 6, 7, 8]; Other modified object write requests: request_snapshot_list = [8]; Other modified object saving: object_snapshot_list = [1, 2]; Local cache of storage pool: cache_snapshot_list = [1, 2, 3, 4, 5, 6, 7, 8]; Other modified object saving: object_snapshot_list = [1, 2, 3, 4, 5, 6, 7, 8]; The corresponding numbers above represent the identifier sequence numbers of snapshot information.

[0082] Furthermore, the present invention also provides a distributed storage system. Figure 3 It is a schematic structural diagram of a distributed storage system provided by an embodiment of the present invention, as Figure 3 shown, including a master control node and at least one storage node; The master control node is connected to at least one storage node, and the master control node is used to execute the steps of the above snapshot information processing method to complete the update of snapshot information.

[0083] In an actual distributed storage system, the deployment relationships among the client, the master control node, and the storage node can have the following several common modes: Fully separated: The client, the master control node, and the storage node are respectively deployed on different machines. This is the most commonly recommended deployment method and is suitable for large-scale and high-performance distributed storage systems.

[0084] Partially coexist: The client or the master control node can be deployed on the same machine as the storage node. This method is suitable for small or lightweight distributed storage systems and can reduce hardware costs and network overhead.

[0085] Hybrid deployment: In some scenarios, the client, the master control node, and the storage node can be hybridly deployed on multiple machines. For example, in some small test environments, the master control node and some storage nodes can be deployed on the same machine, while the client is deployed on other machines.

[0086] The structural settings of this embodiment can be any of the above, without limitation.

[0087] For the introduction of a distributed storage system provided by the present invention, please refer to the above method embodiments, and the present invention will not repeat them here. It has the same beneficial effects as the above snapshot information processing method.

[0088] The above has described in detail each embodiment corresponding to the snapshot information processing method. On this basis, the present invention also discloses a snapshot information processing device corresponding to the above method. Figure 4 It is a structural diagram of a snapshot information processing device provided by an embodiment of the present invention. As Figure 4 shown, the snapshot information processing device includes: A response module 11, configured to respond to an update request of a target object sent by a client, and generate corresponding first snapshot information according to the update request; A reading module 12, configured to read pre-stored second snapshot information of the target object, and determine first incremental snapshot information according to the first snapshot information and the second snapshot information; A determination module 13, configured to determine second incremental snapshot information based on the snapshot information locally stored under the same directory to which the target object belongs and the first incremental snapshot information; A storage module 14, configured to store the second incremental snapshot information into a storage pool to perform an update process on the target object.

[0089] Since the embodiments of the device part correspond to the above embodiments, the embodiments of the device part please refer to the embodiments of the above method part for description, and will not be repeated here.

[0090] For the introduction of a snapshot information processing device provided by the present invention, please refer to the above method embodiments, and the present invention will not repeat them here. It has the same beneficial effects as the above snapshot information processing method.

[0091] Figure 5 It is a structural diagram of a snapshot information processing device provided by an embodiment of the present invention. As Figure 5 shown, the device includes: A memory 21, configured to store a computer program; A processor 22, configured to implement the steps of the snapshot information processing method when executing the computer program.

[0092] The snapshot information processing device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a notebook computer, or a desktop computer, etc.

[0093] Among them, the processor 22 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 22 may be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array. The processor 22 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 22 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 22 may further include an artificial intelligence (AI) processor, and the AI processor is used to process computational operations related to machine learning.

[0094] The memory 21 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 21 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In this embodiment, the memory 21 is at least used to store the following computer program 211. After the computer program is loaded and executed by the processor 22, it can implement the relevant steps of the snapshot information processing method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 21 may further include an operating system 212 and data 213, etc., and the storage method may be transient storage or permanent storage. Among them, the operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, the data involved in the snapshot information processing method, etc.

[0095] In some embodiments, the snapshot information processing device may further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.

[0096] Those skilled in the art can understand that Figure 5 the structure shown in

[0097] does not constitute a limitation on the snapshot information processing device, and it may include more or fewer components than shown in the figure.

[0098] For the introduction of a snapshot information processing device provided by the present invention, please refer to the above method embodiments. The present invention will not be elaborated herein, and it has the same beneficial effects as the above snapshot information processing method.

[0099] Furthermore, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor 22, the steps of the above snapshot information processing method are implemented.

[0100] It can be understood that if the method in the above embodiments is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in various embodiments of the present invention. The foregoing storage media include: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks, etc., which can store program codes.

[0101] For the introduction of a computer-readable storage medium provided by the present invention, please refer to the above method embodiments. The present invention will not be elaborated herein, and it has the same beneficial effects as the above snapshot information processing method.

[0102] Furthermore, the present invention also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the snapshot information processing method are implemented.

[0103] For the introduction of a computer program product provided by the present invention, please refer to the above method embodiments. The present invention will not be elaborated herein, and it has the same beneficial effects as the above snapshot information processing method.

[0104] The above has introduced in detail a snapshot information processing method, a distributed storage system, a device, a medium, and a product provided by the present invention. The various embodiments in the specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple. For the relevant parts, please refer to the description of the method part. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

[0105] It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

Claims

1. A snapshot information processing method, characterized in that, Including: Responding to an update request of a target object sent by a client, and generating corresponding first snapshot information according to the update request; Reading second snapshot information pre-stored for the target object, and determining first incremental snapshot information according to the first snapshot information and the second snapshot information; Determining second incremental snapshot information based on snapshot information locally stored under the same directory to which the target object belongs and the first incremental snapshot information; Storing the second incremental snapshot information into a storage pool to perform an update process on the target object.

2. The snapshot information processing method according to claim 1, wherein Determining the first incremental snapshot information according to the first snapshot information and the second snapshot information includes: Obtaining a first lower limit identifier in the first snapshot information and a second upper limit identifier in the second snapshot information; Determining first updated snapshot information corresponding to a snapshot information identifier based on the first lower limit identifier and the second upper limit identifier; Determining the first incremental snapshot information according to the first updated snapshot information.

3. The snapshot information processing method according to claim 2, wherein Determining first updated snapshot information corresponding to a snapshot information identifier based on the first lower limit identifier and the second upper limit identifier includes: When the first lower limit identifier is greater than the second upper limit identifier, determining that there is no intersection between the first snapshot information and the second snapshot information, and the first snapshot information is the first updated snapshot information; When the first lower limit identifier is less than or equal to the second upper limit identifier, determining that there is an intersection between the first snapshot information and the second snapshot information, and a partial snapshot of the first snapshot information is the first updated snapshot information.

4. The snapshot information processing method according to claim 3, wherein Determining the first incremental snapshot information according to the first updated snapshot information includes: When the first lower limit identifier is greater than the second upper limit identifier, using the first updated snapshot information as the first incremental snapshot information; When the first lower limit identifier is less than or equal to the second upper limit identifier, obtaining a first upper limit identifier in the first snapshot information and a second lower limit identifier in the second snapshot information; Determining second updated snapshot information according to the first upper limit identifier and the second lower limit identifier, and determining the first incremental snapshot information based on the second updated snapshot information.

5. The snapshot information processing method according to claim 4, wherein Determining second incremental snapshot information based on snapshot information locally stored under the same directory to which the target object belongs and the first incremental snapshot information includes: When there is an update to another target object under the same directory to which the target object belongs, when the identifier of the locally stored snapshot information is the same as the identifier of the second snapshot information and the first lower limit identifier is greater than the second upper limit identifier, using the first incremental snapshot information as the second incremental snapshot information; When the identifier of the locally stored snapshot information is the same as the identifier of the second snapshot information and the first lower limit identifier is less than or equal to the second upper limit identifier, comparing the identifier of the first incremental snapshot information with the identifier of the locally stored snapshot information; If the identifier of the first incremental snapshot information is included in the identifier of the locally stored snapshot information, then the locally stored snapshot information is called as the second incremental snapshot information; If a part of the identifier of the first incremental snapshot information is included in the identifier of the locally stored snapshot information, then the first incremental snapshot information and the locally stored snapshot information are fused as the second incremental snapshot information.

6. The snapshot information processing method according to claim 4, characterized in that When the identifier of the locally stored snapshot information is different from the identifier of the second snapshot information, the method further includes: Obtaining a third upper limit identifier corresponding to the locally stored snapshot information; Determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier.

7. The snapshot information processing method according to claim 6, characterized in that, Determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is less than or equal to the third upper limit identifier, determining a missing target identifier according to the identifier of the locally stored snapshot information and the identifier of the first snapshot information; and using the snapshot information corresponding to the target identifier as the second incremental snapshot information.

8. The snapshot information processing method according to claim 6, wherein Determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, filling the identifier of the locally stored snapshot information into the identifier corresponding to the first snapshot information of the update request to determine the second incremental snapshot information.

9. The snapshot information processing method according to claim 6, wherein Determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first upper limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, using the first snapshot information as the second incremental snapshot information.

10. The snapshot information processing method according to claim 6, characterized in that, Determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first upper limit identifier is greater than the second upper limit identifier, the first lower limit identifier is less than or equal to the third upper limit identifier, and the first upper limit identifier is greater than the third upper limit identifier, updating the first snapshot information corresponding to the update request to the locally stored snapshot information, and using the updated locally stored snapshot information as the second incremental snapshot information.

11. The snapshot information processing method according to claim 1, wherein When the target object is the first object, the second snapshot information is a full - volume snapshot information, and the full - volume snapshot information is stored in the storage pool.

12. A distributed storage system, characterized in that, Including a master node and at least one storage node; The master node is connected to at least one storage node, and the master node is configured to execute the steps of the snapshot information processing method according to any one of claims 1 to 11 above to complete the update of the snapshot information.

13. A snapshot information processing device, characterized in that, Including: A memory for storing a computer program; A processor, configured to implement the steps of the snapshot information processing method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the snapshot information processing method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising computer programs / instructions, characterized in that, When the computer program / instructions are executed by a processor, the steps of the snapshot information processing method according to any one of claims 1 to 11 are implemented.

Images