Snapshot information processing method, distributed storage system, device, medium and product

Abstract

The present invention discloses a snapshot information processing method, a distributed storage system, a device, a medium and a product, and relates to the field of data processing technology. The first incremental snapshot information is determined based on the first snapshot information generated by the update request of the target object and the second snapshot information pre-stored by the target object. The first incremental snapshot information reduces the retransmission of most snapshot information and improves the transmission rate. The second incremental snapshot information is determined based on the locally stored snapshot information and the first incremental snapshot information in the same directory to which the target object belongs. By updating and storing the snapshot information of other target objects, the increase in network load caused by re-acquisition from the client is avoided. In the same directory, the second incremental snapshot information is further determined based on the snapshot information in the directory, ensuring that the snapshot information is as streamlined as possible while reducing the network load problem caused by acquisition from the client, thereby improving the performance and efficiency of the storage system.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a snapshot information processing method, distributed storage system, device, medium and product. Background Art

[0002] Distributed storage systems are a key technology in the current fields of big data processing and cloud computing. The transmission and caching of snapshot information have become crucial factors in improving the performance of distributed storage systems. Conventionally, when an object is updated or modified, the system generates a snapshot of the object's state and transmits it to the storage pool. However, during this transmission process, the full snapshot information is transmitted and stored for each object modification, resulting in low transmission efficiency. Furthermore, each update request within the storage pool must include a list of the full snapshots, increasing the network load during transmission and reducing system responsiveness.

[0003] Therefore, how to improve the transmission efficiency of snapshot information and the system response speed when an object is modified is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of the present invention is to provide a snapshot information processing method, distributed storage system, device, medium and product to solve the problems of low transmission efficiency and increased network load caused by full snapshot information transmission when conventional objects are modified.

[0005] To solve the above technical problems, the present invention provides a snapshot information processing method, comprising:

[0006] Responding to an update request for a target object issued by a client, and generating corresponding first snapshot information according to the update request;

[0007] Reading the second snapshot information pre-stored for the target object, and determining first incremental snapshot information according to the first snapshot information and the second snapshot information;

[0008] Determine second incremental snapshot information based on the locally stored snapshot information in the same directory to which the target object belongs and the first incremental snapshot information;

[0009] The second incremental snapshot information is stored in a storage pool to update the target object.

[0010] On the one hand, determining first incremental snapshot information according to the first snapshot information and the second snapshot information includes:

[0011] Obtaining a first lower limit identifier in the first snapshot information and a second upper limit identifier in the second snapshot information;

[0012] determining first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier;

[0013] The first incremental snapshot information is determined according to the first updated snapshot information.

[0014] On the other hand, determining first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier includes:

[0015] If the first lower limit identifier is greater than the second upper limit identifier, determining that the first snapshot information and the second snapshot information have no intersection, and the first snapshot information is the first updated snapshot information;

[0016] When the first lower limit identifier is less than or equal to the second upper limit identifier, it is determined that the first snapshot information and the second snapshot information have an intersection, and part of the snapshot information of the first snapshot information is the first updated snapshot information.

[0017] On the other hand, determining the first incremental snapshot information according to the first update snapshot information includes:

[0018] In a case where the first lower limit identifier is greater than the second upper limit identifier, using the first updated snapshot information as the first incremental snapshot information;

[0019] When the first lower limit identifier is less than or equal to the second upper limit identifier, obtaining the first upper limit identifier in the first snapshot information and the second lower limit identifier in the second snapshot information;

[0020] Second updated snapshot information is determined according to the first upper limit identifier and the second lower limit identifier, and the first incremental snapshot information is determined based on the second updated snapshot information.

[0021] On the other hand, determining second incremental snapshot information based on the locally stored snapshot information in the same directory to which the target object belongs and the first incremental snapshot information includes:

[0022] When there is an update of another target object in the same directory to which the target object belongs, when the identifier of the locally stored snapshot information is the same as the identifier of the second snapshot information, and when the first lower limit identifier is greater than the second upper limit identifier, use the first incremental snapshot information as the second incremental snapshot information;

[0023] When the identifier of the locally stored snapshot information is the same as the identifier of the second snapshot information, and when the first lower limit identifier is less than or equal to the second upper limit identifier, comparing the identifier of the first incremental snapshot information with the identifier of the locally stored snapshot information;

[0024] If the identifier of the first incremental snapshot information is included in the identifier of the locally stored snapshot information, calling the locally stored snapshot information as the second incremental snapshot information;

[0025] If the identifier portion of the first incremental snapshot information is included in the identifier of the locally stored snapshot information, the first incremental snapshot information and the locally stored snapshot information are merged to form the second incremental snapshot information.

[0026] On the other hand, when the identifier of the locally stored snapshot information is different from the identifier of the second snapshot information, the method further includes:

[0027] Obtaining a third upper limit identifier corresponding to the locally stored snapshot information;

[0028] The second incremental snapshot information is determined according to the third upper limit identifier and the first lower limit identifier.

[0029] On the other hand, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes:

[0030] When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is less than or equal to the third upper limit identifier, the missing target identifier is determined based on the identifier of the locally stored snapshot information and the identifier of the first snapshot information; and the snapshot information corresponding to the target identifier is used as the second incremental snapshot information.

[0031] On the other hand, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes:

[0032] When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, the identifier of the locally stored snapshot information is filled into the identifier corresponding to the first snapshot information of the update request to determine the second incremental snapshot information.

[0033] On the other hand, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes:

[0034] When the first upper limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, the first snapshot information is used as the second incremental snapshot information.

[0035] On the other hand, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes:

[0036] When the first upper limit identifier is greater than the second upper limit identifier, and the first lower limit identifier is less than or equal to the third upper limit identifier, and the first upper limit identifier is greater than the third upper limit identifier, the first snapshot information corresponding to the update request is updated to the locally stored snapshot information, so that the updated locally stored snapshot information is used as the second incremental snapshot information.

[0037] On the other hand, when the target object is the first object, the second snapshot information is full snapshot information, and the storage pool stores the full snapshot information.

[0038] To solve the above technical problems, the present invention further provides a distributed storage system, comprising a master control node and at least one storage node;

[0039] The master control node is connected to at least one storage node, and the master control node is used to execute the steps of the snapshot information processing method described above to complete the update of the snapshot information.

[0040] To solve the above technical problems, the present invention further provides a snapshot information processing device, comprising:

[0041] memory for storing computer programs;

[0042] A processor is configured to implement the steps of the snapshot information processing method when executing the computer program.

[0043] To solve the above technical problem, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the snapshot information processing method are implemented.

[0044] To solve the above technical problem, the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the snapshot information processing method when executed by a processor.

[0045] The beneficial effects of the present invention are that, on the one hand, the first incremental snapshot information is determined based on the first snapshot information generated by the update request of the target object and the second snapshot information pre-stored for the target object. When the target object is updated, only the first incremental snapshot information is determined, avoiding the subsequent transmission of the full snapshot information of the target object as long as there is an update request in the conventional technical solution. Compared with the entire snapshot list, the first incremental snapshot information reduces the retransmission of most snapshot information, reduces the amount of data, and also facilitates the subsequent improvement of the transmission rate. On the other hand, the second incremental snapshot information is determined based on the locally stored snapshot information and the first incremental snapshot information in the same directory to which the target object belongs. This is because the snapshot information of other target objects in the same directory that the target object belongs to is previously transmitted to the local cache. If they are in the same directory, the snapshot information of other target objects is updated and stored to avoid the increase in network load caused by re-acquisition from the client. In the same directory, the second incremental snapshot information is further determined based on the snapshot information in the directory, ensuring that the snapshot information is as streamlined as possible while reducing the network load problem caused by acquisition from the client. The final incremental snapshot information is directly obtained from the local cache to achieve cache optimization and improve the performance and efficiency of the storage system.

[0046] Secondly, by determining the first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier, the data range between the two snapshots can be quickly located without having to scan the entire data set in full. This greatly reduces computational overhead. At the same time, using the upper and lower limit identifiers as boundary conditions helps the system quickly determine which data blocks or objects may have changed. By accurately determining the incremental range, the system can capture only the data that has changed between the two snapshots without having to back up the unmodified data, which significantly reduces storage space usage. Based on the comparison of the upper and lower limit identifiers of the two snapshot information, a full scan is avoided during the data scanning process, and the incremental range is determined more quickly. The time point changes between the two snapshots can be accurately captured, thereby ensuring consistency during data recovery, quickly capturing data changes, and reducing the risk of data loss due to system failures or unexpected interruptions. In the process of determining the first incremental snapshot information, after comparing the first lower limit identifier and the second upper limit identifier, the first upper limit identifier and the second lower limit identifier are continued to be compared in different situations, narrowing the constraint range while further ensuring the accuracy of the incremental snapshot information. In the process of determining the second incremental snapshot information, there is a prerequisite that when there is an update of other target objects in the same directory to which the target object belongs, the identifier of the locally stored snapshot information is the same as the identifier of the second snapshot information. On this basis, redundant data is further reduced to ensure that while streamlining the incremental data, the locally stored snapshot information is called nearby, reducing network load and improving data transmission efficiency.

[0047] In addition, the present invention also provides a distributed storage system, a snapshot information processing device, a computer-readable storage medium, and a computer program product, which have the same beneficial effects as the above-mentioned snapshot information processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A flowchart of a snapshot information processing method provided by an embodiment of the present invention;

[0050] Figure 2 A flowchart of another snapshot information processing method provided by an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the structure of a distributed storage system provided by an embodiment of the present invention;

[0052] Figure 4 A structural diagram of a snapshot information processing device provided by an embodiment of the present invention;

[0053] Figure 5 This is a structural diagram of a snapshot information processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0055] The core of the present invention is to provide a snapshot information processing method, distributed storage system, device, medium and product to solve the problems of low transmission efficiency and increased network load caused by full snapshot information transmission when conventional objects are modified.

[0056] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0057] Distributed storage systems are one of the key technologies in the current fields of big data processing and cloud computing. With the continuous growth of data volume, the performance and efficiency of storage systems have become a major bottleneck. Snapshot technology is an important means for data protection and recovery in distributed storage systems, but the transmission and caching of snapshot information have become important factors affecting system performance. Existing snapshot information transmission methods usually adopt a full transmission method, which will lead to low transmission efficiency when the number of snapshots is large, further affecting the overall performance of the system. In addition, there are also problems with the storage pool's caching of snapshot information. Each request needs to carry a full snapshot list, which not only increases the network load, but also reduces the system's response speed. The snapshot information processing method provided by the present invention can solve the above technical problems.

[0058] Figure 1 A flowchart of a snapshot information processing method provided by an embodiment of the present invention is shown in FIG. Figure 1 The method shown includes:

[0059] S11: responding to an update request for a target object sent by a client, and generating corresponding first snapshot information according to the update request;

[0060] S12: Reading the second snapshot information pre-stored for the target object, and determining the first incremental snapshot information according to the first snapshot information and the second snapshot information;

[0061] S13: Determine second incremental snapshot information based on the locally stored snapshot information and the first incremental snapshot information in the same directory to which the target object belongs;

[0062] S14: Storing the second incremental snapshot information in a storage pool to update the target object.

[0063] Specifically, when the target object is modified, it is necessary to respond to the target object update request issued by the client. The client is the user of the distributed storage system. It runs on the user's device or application and is responsible for initiating various requests to the distributed storage system, including read, write, update, delete and other operations. The target object refers to the specific data object that the client wants to operate. In a distributed storage system, data is usually stored in the form of objects, such as files, data blocks, key-value pairs, etc. The target object can be a part of a file, a value in a key-value pair, or a record in a database. An update request refers to an operation initiated by the client to modify the target object, which can be the identifier of the target object, updated content or metadata, etc., which are not limited here. The response request can be the master node or storage node of the distributed storage system receiving the update request from the client, or it can be a verification request to perform the update operation.

[0064] In a distributed storage system, when a client initiates an update request, the system needs to generate snapshot information corresponding to the update request while executing the update operation. This is to capture the current state of the data before the update, allowing for restoration to the pre-update state when needed. Regarding how to generate the first snapshot information, a snapshot request is first generated to capture the current state of the target object. This information primarily includes the target object's identifier, snapshot timestamp, snapshot version number, and the current data state. The system needs to capture the current state of the target object before the update. This can be achieved through strategies such as Copy-on-Write (COW) or Redirect-on-Write (ROW). With the COW strategy, when an update request arrives, the system copies the current data of the target object to a new storage location. This copied data becomes the snapshot content. With the ROW strategy, the system records metadata about the current state of the target object, such as data block pointers and file offsets. During an update operation, the system writes the new data to the new location and updates the pointer. The snapshot information points to the old data location before the update.

[0065] The second snapshot information pre-stored for the target object in step S12 is the second snapshot information stored before the target object is modified. The first incremental snapshot information is determined based on the first snapshot information and the second snapshot information. The determination process can capture the state of the snapshot information, such as the identifier of the data block, incremental data, and metadata, by comparing the pointers or hash values, version numbers, or timestamps of the data blocks in the two snapshot information. It can also be determined through corresponding identifiers in the snapshot information, such as the modification identifier corresponding to the object. The snapshot information stores the serial number corresponding to the snapshot identifier representing the object, and the first incremental snapshot information can be determined by comparing the serial numbers of the identifiers corresponding to the respective snapshot information.

[0066] The determination of the second incremental snapshot information in step S13, compared to the conventional determination of only the incremental snapshot information, also takes into account the snapshot information corresponding to the client when the local cache is obtained. Therefore, in this case, first check whether the locally stored snapshot information corresponding to the target object in the same directory is updated. If it is updated, the locally stored snapshot information in the same directory can be called as the final second incremental snapshot information. If the locally stored snapshot information in the directory to which the target object belongs is different from the first incremental snapshot information, it is necessary to re-acquire the first incremental snapshot information corresponding to the client. In addition, there is also a situation where the snapshot information corresponding to the local storage is not updated in time due to a node failure. At this time, it is necessary to obtain the first incremental snapshot information corresponding to the client as the second incremental snapshot information. This point is to illustrate that in the same directory, directly calling the cached snapshot information reduces the corresponding network load when obtaining from the client.

[0067] In step S14, the finally generated second incremental snapshot information is stored in the storage pool to facilitate subsequent update processing of the target object.

[0068] The beneficial effects of the embodiments of the present invention are that, on the one hand, the first incremental snapshot information is determined based on the first snapshot information generated by the update request of the target object and the second snapshot information pre-stored for the target object. When the target object is updated, only the first incremental snapshot information is determined, avoiding the subsequent transmission of the full snapshot information of the target object as long as there is an update request in the conventional technical solution. Compared with the entire snapshot list, the first incremental snapshot information reduces the retransmission of most snapshot information, reducing the amount of data while also facilitating the subsequent improvement of the transmission rate. On the other hand, the second incremental snapshot information is determined based on the locally stored snapshot information and the first incremental snapshot information in the same directory to which the target object belongs. This takes into account that the snapshot information of other target objects in the same directory that the target object belongs to has been previously transmitted to the local cache. If they are in the same directory, the snapshot information of other target objects is updated and stored to avoid the increase in network load caused by re-acquisition from the client. In the same directory, the second incremental snapshot information is further determined based on the snapshot information in the directory, ensuring that the snapshot information is as streamlined as possible while reducing the network load problem caused by acquisition from the client. The final incremental snapshot information is directly obtained from the local cache to achieve cache optimization and improve the performance and efficiency of the storage system.

[0069] In some embodiments, determining first incremental snapshot information based on the first snapshot information and the second snapshot information includes:

[0070] Obtaining a first lower limit identifier in the first snapshot information and a second upper limit identifier in the second snapshot information;

[0071] determining first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier;

[0072] First incremental snapshot information is determined according to the first updated snapshot information.

[0073] Considering that there is at least one snapshot information corresponding to the serial number of the snapshot information identifier of the snapshot information, the first lower limit identifier of the first snapshot information is the minimum identifier of the first snapshot information; similarly, the second upper limit identifier of the second snapshot information is the maximum identifier of the second snapshot information. Based on the comparison of the two, it is determined whether the two snapshot information have an intersection. If there is an intersection, it means that there is a partial update. If there is no intersection, it is necessary to check which snapshot information in the first snapshot information and the second snapshot information has an identifier greater than the identifier of the other snapshot information. If all the identifiers of the first snapshot information are greater than all the identifiers of the second snapshot information, it means that all the snapshot identifiers of the first snapshot information are updated snapshot information. Determine the first incremental snapshot information based on the first updated snapshot information, that is, determine the first incremental snapshot information based on the above-mentioned different situations.

[0074] The first updated snapshot information corresponding to the snapshot information identifier provided in this embodiment is determined based on the first lower limit identifier and the second upper limit identifier, which can quickly locate the data range between the two snapshots without having to scan the entire data set in full. This greatly reduces computational overhead. At the same time, using the upper and lower limit identifiers as boundary conditions helps the system quickly determine which data blocks or objects may have changed. By accurately determining the incremental range, the system can capture only the portion of the data that has changed between the two snapshots without having to back up the unmodified data, which significantly reduces storage space usage.

[0075] In some embodiments, determining first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier includes:

[0076] When the first lower limit identifier is greater than the second upper limit identifier, determining that the first snapshot information and the second snapshot information have no intersection, and the first snapshot information is the first updated snapshot information;

[0077] When the first lower limit identifier is less than or equal to the second upper limit identifier, it is determined that the first snapshot information and the second snapshot information have an intersection, and part of the snapshot information of the first snapshot information is the first updated snapshot information.

[0078] Specifically, when the first lower limit identifier is greater than the second upper limit identifier, it means that the two snapshot information have no intersection, and all the first snapshot information are the latest snapshot information, so the first snapshot information is regarded as the first updated snapshot information.

[0079] When the first lower limit identifier is less than or equal to the second upper limit identifier, it is determined that the first snapshot information and the second snapshot information have an intersection, and the first snapshot information is partially updated, and the partial snapshot information of the first snapshot information needs to be used as the first updated snapshot information.

[0080] This embodiment provides a comparison of upper and lower limit identifiers based on two snapshots, avoiding the need for a complete data scan and quickly determining the incremental range. It accurately captures the point-in-time changes between the two snapshots, ensuring consistency during data recovery. It can quickly capture data changes and reduce the risk of data loss due to system failures or unexpected interruptions.

[0081] In some embodiments, determining first incremental snapshot information based on first updated snapshot information includes:

[0082] In a case where the first lower limit identifier is greater than the second upper limit identifier, using the first updated snapshot information as the first incremental snapshot information;

[0083] When the first lower limit identifier is less than or equal to the second upper limit identifier, obtaining the first upper limit identifier in the first snapshot information and the second lower limit identifier in the second snapshot information;

[0084] Second updated snapshot information is determined according to the first upper limit identifier and the second lower limit identifier, and first incremental snapshot information is determined based on the second updated snapshot information.

[0085] Combining the two situations of the above embodiment, in the first situation, the first updated snapshot information is directly used as the first incremental snapshot information.

[0086] In the second case, the first upper limit identifier and the second lower limit identifier are further obtained to determine the second updated snapshot information based on the two identifiers. In this process, it has been determined that there is an intersection between the two snapshot information (the first snapshot information and the second snapshot information) and there is updated snapshot information, thereby determining the first incremental snapshot information.

[0087] In the process of determining the first incremental snapshot information provided by this embodiment, after comparing the first lower limit identifier and the second upper limit identifier, the first upper limit identifier and the second lower limit identifier are continued to be compared under different circumstances, thereby narrowing the constraint range while further ensuring the accuracy of the incremental snapshot information.

[0088] In some embodiments, determining the second incremental snapshot information based on the locally stored snapshot information and the first incremental snapshot information in the same directory to which the target object belongs includes:

[0089] When there is an update of another target object in the same directory to which the target object belongs, if the identifier of the locally stored snapshot information is the same as the identifier of the second snapshot information, and if the first lower limit identifier is greater than the second upper limit identifier, the first incremental snapshot information is used as the second incremental snapshot information;

[0090] When the identifier of the locally stored snapshot information is the same as the identifier of the second snapshot information, and when the first lower limit identifier is less than or equal to the second upper limit identifier, comparing the identifier of the first incremental snapshot information with the identifier of the locally stored snapshot information;

[0091] If the identifier of the first incremental snapshot information is included in the identifier of the locally stored snapshot information, calling the locally stored snapshot information as the second incremental snapshot information;

[0092] If the identifier portion of the first incremental snapshot information is included in the identifier of the locally stored snapshot information, the first incremental snapshot information and the locally stored snapshot information are merged to form the second incremental snapshot information.

[0093] Specifically, in this embodiment, if updates to other target objects exist within the same directory as the target object, a comparison must be performed using the identifier of the locally stored snapshot information. Typically, the identifier of the locally stored snapshot information and the identifier of the second snapshot information are identical to ensure consistency. Therefore, in this case, the first lower limit identifier and the second upper limit identifier are compared. If the first lower limit identifier is greater than the second upper limit identifier, this indicates that the currently updated snapshot information is not locally stored, and the first incremental snapshot information is directly used as the final second incremental snapshot information.

[0094] If the first lower limit identifier is less than or equal to the second upper limit identifier, it means that there is an intersection between the identifier of the first incremental snapshot information and the identifier of the locally stored snapshot information, and further comparison and verification are required to ensure the accuracy of the second incremental snapshot information.

[0095] If the identifier of the first incremental snapshot information is included in the identifier of the locally stored snapshot information, the locally stored snapshot information is directly used as the second incremental snapshot information of the target object. If the identifier of the first incremental snapshot information is partially included in the identifier of the locally stored snapshot information, it needs to be merged. The merging process here can be to retain a full set and delete the duplicate snapshot information. Alternatively, snapshot information not stored locally can be selected as the second incremental snapshot information.

[0096] In the process of determining the second incremental snapshot information provided by this embodiment, there is a premise that when there is an update of other target objects in the same directory to which the target object belongs and the identifier of the locally stored snapshot information is the same as the identifier of the second snapshot information, redundant data is further reduced on this basis to ensure that while streamlining the incremental data, the locally stored snapshot information is called nearby, reducing network load and improving data transmission efficiency.

[0097] In other embodiments, when determining the second incremental snapshot information, if no update of other target objects exists in the same directory to which the target object belongs, the first incremental snapshot information of the target object is updated to the locally stored snapshot information.

[0098] It should be noted that when there is no update of other target objects in the same directory where the target object is located, that is, the target object is the first updated object in a directory, the determined first incremental snapshot information is directly updated to the locally stored snapshot information so that it can be used as the second incremental snapshot information later.

[0099] This embodiment provides that when the first target object is updated in the directory where the target object is located, the first incremental snapshot information of the target object needs to be updated to the locally stored snapshot information, so that when other target objects in the same directory are subsequently updated, the locally stored comparison information can be directly called.

[0100] In some other embodiments, when the identifier of the locally stored snapshot information is different from the identifier of the second snapshot information, the method further includes:

[0101] Comparing the identifier of the first snapshot information with the identifier of the locally stored snapshot information according to a preset period;

[0102] If the identifier of the first snapshot information is different from the identifier of the locally stored snapshot information, the identifier of the first snapshot information is updated to the identifier of the locally stored snapshot information to complete the update process of the locally stored snapshot information.

[0103] Taking into account that the identifier of the locally stored snapshot information is different from the identifier of the second snapshot information, it may be caused by a node failure of the storage node. Therefore, it is currently necessary to compare the identifier of the first snapshot information with the identifier of the locally stored snapshot information according to a preset period. If they are different, the identifier of the first snapshot information is updated to the identifier of the locally stored snapshot information to ensure that the locally stored snapshot information is the latest snapshot information.

[0104] This embodiment provides that when the identifier of the locally stored snapshot information is different from the identifier of the second snapshot information, the locally stored snapshot information is updated in a timely manner to ensure that the locally stored comparison information is directly called when other target objects in the same directory are subsequently updated, thereby reducing network load.

[0105] In some embodiments, when the identifier of the locally stored snapshot information is different from the identifier of the second snapshot information, the method further includes:

[0106] Obtaining a third upper limit identifier corresponding to the locally stored snapshot information;

[0107] Second incremental snapshot information is determined according to the third upper limit identifier and the first lower limit identifier.

[0108] It should be noted that when the identifier of the locally stored snapshot information is different from the identifier of the second snapshot information, in addition to timely updating the locally stored snapshot information, it is also necessary to compare the current locally stored snapshot information with the first snapshot information to determine the second incremental snapshot information to ensure the update process of the current target object.

[0109] Specifically, it is necessary to compare the first lower limit identifier of the first snapshot information with the third upper limit identifier of the snapshot information stored this time. Based on the two situations of comparing the first lower limit identifier of the first snapshot information with the second upper limit identifier of the second snapshot information, it is necessary to distinguish the comparisons carried out in the two situations to determine the second incremental snapshot information.

[0110] In the process of determining the second incremental snapshot information provided by this embodiment, a scheme is used to compare the first lower limit identifier of the first snapshot information expanded when the identifier of the locally stored snapshot information and the identifier of the second snapshot information are different with the third upper limit identifier of the snapshot information stored this time, thereby improving the diversity and flexibility of the process of determining the second incremental snapshot information to ensure the update of the current target object.

[0111] In some embodiments, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes:

[0112] When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is less than or equal to the third upper limit identifier, the missing target identifier is determined based on the identifier of the locally stored snapshot information and the identifier of the first snapshot information; the snapshot information corresponding to the target identifier is used as the second incremental snapshot information.

[0113] Specifically, when the first lower limit identifier is greater than the second upper limit identifier, it means that the first snapshot information and the second snapshot information have no intersection, and the first limit identifier is less than or equal to the third upper limit identifier, it means that the first snapshot information and the locally stored snapshot information have an intersection, then it is necessary to determine the missing target identifier based on the identifier of the locally stored snapshot information and the identifier of the first snapshot information as the second incremental snapshot information.

[0114] In addition, the intersection relationship between the first upper limit identifier and the second upper limit identifier may be further determined to ensure updating while improving the comparison progress.

[0115] This embodiment provides that when the first snapshot information and the second snapshot information have no intersection but have an intersection with the locally stored snapshot information, part of the snapshot information missing in the update request will be used as the locally stored updated snapshot information to ensure that the locally stored snapshot is updated while improving the accuracy of the second incremental snapshot information.

[0116] In some other embodiments, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes:

[0117] When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, the identifier of the locally stored snapshot information is filled into the identifier corresponding to the first snapshot information of the update request to determine the second incremental snapshot information.

[0118] Specifically, when the first lower limit identifier is greater than the second upper limit identifier, and the first lower limit identifier is greater than the third upper limit identifier, it means that the first snapshot information, the second snapshot information and the locally stored snapshot information have no intersection, then the identifier of the locally stored snapshot information is filled in with the identifier corresponding to the first snapshot information of the update request, so as to ensure the consistency of the snapshot information between the local and the client while improving the update timeliness of the second incremental snapshot information.

[0119] In some other embodiments, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes:

[0120] When the first upper limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, the first snapshot information is used as the second incremental snapshot information.

[0121] Specifically, when the first upper limit identifier is greater than the second upper limit identifier, it means that there is an intersection between the first snapshot information and the second snapshot information. Furthermore, when the first lower limit identifier is greater than the third upper limit identifier, it means that there is no intersection between the first snapshot information and the locally stored snapshot information. In this case, the first snapshot information is directly used as the second incremental snapshot information.

[0122] This embodiment provides a method for determining the second incremental snapshot information when there is an intersection between the first snapshot information and the second snapshot information, but no intersection with the locally stored snapshot information, so as to ensure timely update of the second incremental snapshot information while ensuring timely update of the locally stored snapshot information.

[0123] In some other embodiments, determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes:

[0124] When the first upper limit identifier is greater than the second upper limit identifier, and the first lower limit identifier is less than or equal to the third upper limit identifier, and the first upper limit identifier is greater than the third upper limit identifier, the first snapshot information corresponding to the update request is updated to the locally stored snapshot information, so that the updated locally stored snapshot information is used as the second incremental snapshot information.

[0125] Specifically, when the first upper limit identifier is greater than the second upper limit identifier, and the first lower limit identifier is less than or equal to the third upper limit identifier, it means that the first snapshot information and the second snapshot information have an intersection, and the first snapshot information has an intersection with the locally stored snapshot information, then the first snapshot information needs to be directly updated to the local storage as the second incremental snapshot information.

[0126] In the process of determining the second incremental snapshot information provided by this embodiment, considering that the first snapshot information and the second snapshot information have an intersection, and the first snapshot information has an intersection with the locally stored snapshot information, there is no need to rescan, and the first snapshot information is directly stored in the locally stored snapshot information to improve the update progress of the snapshot information.

[0127] In some embodiments, when the target object is the first object, the second snapshot information is full snapshot information, and the full snapshot information is stored in the storage pool.

[0128] Specifically, when the first object is updated, initialization is performed and the full snapshot information needs to be used as the second snapshot information of the target object. The full snapshot information is stored in the storage pool to facilitate subsequent transmission and cache optimization with incremental snapshot information.

[0129] Figure 2 A flowchart of another snapshot information processing method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, including:

[0130] S21: responding to the write request sent by the client;

[0131] S22: Parsing the first snapshot information in the write request;

[0132] S23: Reading the second snapshot information pre-stored on the target object;

[0133] S24: Determine whether the first lower limit identifier is greater than the second upper limit identifier; if so, proceed to step S25; if not, proceed to step S26;

[0134] S25: searching for an identifier of the locally stored snapshot information according to the directory or volume where the target object is located;

[0135] S27: Determine whether the first lower limit identifier is greater than the third upper limit identifier; if so, proceed to step S28; if not, proceed to step S29;

[0136] S28: Fill the locally stored third upper limit identifier into the response to request the client to resend, and return to step S21;

[0137] S29: Update the identifiers of the snapshots missing from the write request from the local cache, and proceed to step S26;

[0138] S26: Determine whether the first upper limit identifier is greater than the second upper limit identifier. If so, proceed to step S30; if not, proceed to step S31;

[0139] S30: searching for an identifier of the locally stored snapshot information according to the directory or volume where the target object is located;

[0140] S31: Write the target object normally and proceed to step S36;

[0141] S32: Determine whether the first lower limit identifier is greater than the third upper limit identifier; if so, proceed to step S33; if not, proceed to step S34;

[0142] S33: Copy the write object and perform a write operation;

[0143] S34: Determine whether the first upper limit identifier is greater than the third upper limit identifier; if so, proceed to step S35; if not, proceed to step S33;

[0144] S35: Update the first snapshot information to the local computer and proceed to step S33;

[0145] S36: Feedback a response to the client.

[0146] Initialize snapshot list: Initialize a snapshot list in the storage pool to store all snapshot identities (IDs) (i.e., identifiers corresponding to snapshot information).

[0147] Incremental transfer of snapshot lists: When updating objects in the storage pool, the latest snapshot ID is included and transferred to the storage pool.

[0148] Storage pool local cache: The storage pool locally caches the full snapshot ID list and updates the cached snapshot ID list based on requests. If the saved snapshot list is insufficient for modifications to other objects, the snapshot list in the local cache can be used.

[0149] Request processing: When a write request is received, the system checks the snapshot ID list in the request and compares it with the snapshot ID lists stored in the local storage pool cache and the target object. If any snapshot IDs have changed, the system updates the snapshot ID list in the local storage pool cache. If the snapshot IDs have changed significantly and the number of snapshot IDs in the request is insufficient, the system triggers a request retransmission, requiring the client to send an incremental snapshot list.

[0150] Initialize the snapshot list: snapshot_list=[1, 2, 3, 4, 5];

[0151] Incremental transfer of snapshot list: incremental_snapshot_list=[6, 7, 8];

[0152] Storage pool local cache: cache_snapshot_list=[1, 2, 3, 4, 5];

[0153] Modify the object and save: object_snapshot_list=[1, 2, 3];

[0154] Write request: request_snapshot_list=[8];

[0155] Resend request: request_snapshot_list=[6, 7, 8];

[0156] Updated storage pool local cache: cache_snapshot_list=[1, 2, 3, 4, 5, 6, 7, 8];

[0157] Modify the object and save: object_snapshot_list=[1, 2, 3, 4, 5, 6, 7, 8];

[0158] Other modification object write requests: request_snapshot_list=[8];

[0159] Save other modified objects: object_snapshot_list=[1,2];

[0160] Storage pool local cache: cache_snapshot_list=[1, 2, 3, 4, 5, 6, 7, 8];

[0161] Save other modified objects: object_snapshot_list=[1, 2, 3, 4, 5, 6, 7, 8];

[0162] The corresponding numbers above represent the identifier serial numbers of the snapshot information.

[0163] Furthermore, the present invention also provides a distributed storage system. Figure 3 A structural diagram of a distributed storage system provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, it includes a master control node and at least one storage node;

[0164] The master control node is connected to at least one storage node, and is used to execute the steps of the above snapshot information processing method to complete the update of the snapshot information.

[0165] In actual distributed storage systems, the deployment relationships among clients, master nodes, and storage nodes can have the following common patterns:

[0166] Complete separation: Clients, master nodes, and storage nodes are deployed on separate machines. This is the most common and recommended deployment method, suitable for large-scale, high-performance distributed storage systems.

[0167] Partial co-location: Clients or master nodes can be deployed on the same machine as storage nodes. This approach is suitable for small or lightweight distributed storage systems and can reduce hardware costs and network overhead.

[0168] Hybrid deployment: In some scenarios, clients, master nodes, and storage nodes can be deployed on multiple machines. For example, in a small test environment, the master node and some storage nodes can be deployed on the same machine, while clients can be deployed on other machines.

[0169] The structural setting of this embodiment can be any one of the above-mentioned ones, without limitation.

[0170] For an introduction to a distributed storage system provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. The system has the same beneficial effects as the above snapshot information processing method.

[0171] The above detailed descriptions of various embodiments corresponding to the snapshot information processing method are based on which the present invention further discloses a snapshot information processing device corresponding to the above method. Figure 4 This is a structural diagram of a snapshot information processing device provided by an embodiment of the present invention. Figure 4 As shown, the snapshot information processing device includes:

[0172] The response module 11 is configured to respond to a target object update request issued by a client and generate corresponding first snapshot information according to the update request;

[0173] A reading module 12 is configured to read the second snapshot information pre-stored on the target object and determine the first incremental snapshot information based on the first snapshot information and the second snapshot information;

[0174] A determination module 13 is configured to determine second incremental snapshot information based on the locally stored snapshot information and the first incremental snapshot information in the same directory to which the target object belongs;

[0175] The storage module 14 is configured to store the second incremental snapshot information into a storage pool to update the target object.

[0176] Since the embodiments of the device part correspond to the above embodiments, the embodiments of the device part please refer to the description of the embodiments of the method part, and will not be repeated here.

[0177] For an introduction to a snapshot information processing device provided by the present invention, please refer to the above method embodiment, and the present invention will not be described in detail here. It has the same beneficial effects as the above snapshot information processing method.

[0178] Figure 5 A structural diagram of a snapshot information processing device provided by an embodiment of the present invention, such as Figure 5 As shown, the device includes:

[0179] Memory 21, for storing computer programs;

[0180] The processor 22 is configured to implement the steps of the snapshot information processing method when executing a computer program.

[0181] The snapshot information processing device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0182] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented in at least one of the following hardware forms: a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array. The processor 22 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 22 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 22 may also include an artificial intelligence (AI) processor, which is responsible for processing computing operations related to machine learning.

[0183] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 21 is at least used to store the following computer program 211, wherein, after the computer program is loaded and executed by the processor 22, it can implement the relevant steps of the snapshot information processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include but is not limited to data involved in the snapshot information processing method, etc.

[0184] In some embodiments, the snapshot information processing device may further include a display screen 23 , an input / output interface 24 , a communication interface 25 , a power supply 26 , and a communication bus 27 .

[0185] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the snapshot information processing device, and may include more or fewer components than shown in the figure.

[0186] The processor 22 implements the snapshot information processing method provided by any of the above embodiments by calling instructions stored in the memory 21 .

[0187] For an introduction to a snapshot information processing device provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above snapshot information processing method.

[0188] Furthermore, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by the processor 22 , the steps of the above-mentioned snapshot information processing method are implemented.

[0189] It is understood that if the methods in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0190] For an introduction to a computer-readable storage medium provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. The present invention has the same beneficial effects as the above snapshot information processing method.

[0191] Furthermore, the present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the snapshot information processing method when executed by a processor.

[0192] For an introduction to a computer program product provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. The computer program product has the same beneficial effects as the above snapshot information processing method.

[0193] The above is a detailed introduction to a snapshot information processing method, distributed storage system, device, medium and product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0194] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A snapshot information processing method, characterized in that: include: Responding to an update request for a target object issued by a client, and generating corresponding first snapshot information according to the update request; Reading the second snapshot information pre-stored for the target object, and determining first incremental snapshot information according to the first snapshot information and the second snapshot information; Determine second incremental snapshot information based on the locally stored snapshot information in the same directory to which the target object belongs and the first incremental snapshot information; storing the second incremental snapshot information in a storage pool to update the target object; Correspondingly, determining first incremental snapshot information according to the first snapshot information and the second snapshot information includes: Obtaining a first lower limit identifier in the first snapshot information and a second upper limit identifier in the second snapshot information; determining first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier; determining the first incremental snapshot information according to the first update snapshot information; Correspondingly, determining the first incremental snapshot information according to the first update snapshot information includes: In a case where the first lower limit identifier is greater than the second upper limit identifier, using the first updated snapshot information as the first incremental snapshot information; When the first lower limit identifier is less than or equal to the second upper limit identifier, obtaining the first upper limit identifier in the first snapshot information and the second lower limit identifier in the second snapshot information; determining second update snapshot information according to the first upper limit identifier and the second lower limit identifier, and determining the first incremental snapshot information based on the second update snapshot information; Correspondingly, when the identifier of the locally stored snapshot information is different from the identifier of the second snapshot information, the method further includes: Obtaining a third upper limit identifier corresponding to the locally stored snapshot information; The second incremental snapshot information is determined according to the third upper limit identifier and the first lower limit identifier.

2. The snapshot information processing method according to claim 1, characterized in that: Determining first updated snapshot information corresponding to the snapshot information identifier based on the first lower limit identifier and the second upper limit identifier includes: If the first lower limit identifier is greater than the second upper limit identifier, determining that the first snapshot information and the second snapshot information have no intersection, and the first snapshot information is the first updated snapshot information; When the first lower limit identifier is less than or equal to the second upper limit identifier, it is determined that the first snapshot information and the second snapshot information have an intersection, and part of the snapshot information of the first snapshot information is the first updated snapshot information.

3. The snapshot information processing method according to claim 2, characterized in that: Determining second incremental snapshot information based on the locally stored snapshot information in the same directory to which the target object belongs and the first incremental snapshot information includes: When there is an update of another target object in the same directory to which the target object belongs, when the identifier of the locally stored snapshot information is the same as the identifier of the second snapshot information, and when the first lower limit identifier is greater than the second upper limit identifier, use the first incremental snapshot information as the second incremental snapshot information; When the identifier of the locally stored snapshot information is the same as the identifier of the second snapshot information, and when the first lower limit identifier is less than or equal to the second upper limit identifier, comparing the identifier of the first incremental snapshot information with the identifier of the locally stored snapshot information; If the identifier of the first incremental snapshot information is included in the identifier of the locally stored snapshot information, calling the locally stored snapshot information as the second incremental snapshot information; If the identifier portion of the first incremental snapshot information is included in the identifier of the locally stored snapshot information, the first incremental snapshot information and the locally stored snapshot information are merged to serve as the second incremental snapshot information.

4. The snapshot information processing method according to claim 1, characterized in that: Determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is less than or equal to the third upper limit identifier, the missing target identifier is determined based on the identifier of the locally stored snapshot information and the identifier of the first snapshot information; and the snapshot information corresponding to the target identifier is used as the second incremental snapshot information.

5. The snapshot information processing method according to claim 1, characterized in that: Determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first lower limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, the identifier of the locally stored snapshot information is filled into the identifier corresponding to the first snapshot information of the update request to determine the second incremental snapshot information.

6. The snapshot information processing method according to claim 1, characterized in that: Determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first upper limit identifier is greater than the second upper limit identifier and the first lower limit identifier is greater than the third upper limit identifier, the first snapshot information is used as the second incremental snapshot information.

7. The snapshot information processing method according to claim 1, characterized in that: Determining the second incremental snapshot information according to the third upper limit identifier and the first lower limit identifier includes: When the first upper limit identifier is greater than the second upper limit identifier, and the first lower limit identifier is less than or equal to the third upper limit identifier, and the first upper limit identifier is greater than the third upper limit identifier, the first snapshot information corresponding to the update request is updated to the locally stored snapshot information, so that the updated locally stored snapshot information is used as the second incremental snapshot information.

8. The snapshot information processing method according to claim 1, characterized in that: When the target object is the first object, the second snapshot information is full snapshot information, and the storage pool stores the full snapshot information.

9. A distributed storage system, characterized in that: including a master control node and at least one storage node; The master control node is connected to at least one storage node, and the master control node is used to execute the steps of the snapshot information processing method described in any one of claims 1 to 8 above to complete the update of the snapshot information.

10. A snapshot information processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the snapshot information processing method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the snapshot information processing method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the snapshot information processing method according to any one of claims 1 to 8 are implemented.

Images