Backup set execution processing system, backup set usage terminal, and lock manager

Abstract

The application relates to the technical field of data storage management, and provides an execution processing system of a backup set, a backup set use end and a lock manager. The system comprises the backup set use end and the lock manager; the backup set use end is used for determining a lock type of an operation type application according to a job type and an operation type of a target backup set; the backup set use end is also used for generating a job request of the target backup set according to the lock type and the operation type and sending the job request to the lock manager; the lock manager is used for determining the lock type and the operation type according to the job request, determining a lock response result according to conflict detection logic of various operation requests for locking which is preset, and feeding back the lock response result to the backup set use end; and the backup set use end is used for executing processing on the target backup set according to the lock response result. The system can improve job operation efficiency and can coordinate conflicts caused by concurrent operation sharing metadata of multiple jobs.

Description

Technical Field

[0001] This application relates to the field of data storage management technology, and in particular to an execution processing system for backup sets, a backup set user terminal, and a lock manager. Background Technology

[0002] In data backup scenarios, virtual storage pools are typically used to logically aggregate resources. Within these virtual storage pools, a set of related files is further aggregated using the concept of backup sets, which are managed through backup set metadata. Reading and writing files within the backup set requires access to this metadata. Against this backdrop, conflicts arising from concurrent operations by multiple jobs (backup jobs / copy jobs / recovery jobs / reclaim jobs) sharing metadata are becoming increasingly prominent, including read-write conflicts, write-write conflicts, and long-running transaction blocking. Summary of the Invention

[0003] Therefore, it is necessary to provide a backup set execution processing system, a backup set user terminal, and a lock manager to address the aforementioned technical problems.

[0004] Firstly, this application provides an execution processing system for backup sets.

[0005] The system includes a backup set user terminal and a lock manager;

[0006] The backup set user terminal is used to determine the lock type requested for the operation type based on the job type and operation type for the target backup set;

[0007] The backup set user is also used to generate a job request for the target backup set and send it to the lock manager based on the lock type and the operation type.

[0008] The lock manager is used to determine the lock type and the operation type according to the job request, determine the lock response result according to the pre-set conflict detection logic of various operation request locks, and feed it back to the backup set user terminal.

[0009] The backup set user terminal is used to perform processing on the target backup set based on the lock response result.

[0010] In one embodiment, determining the lock type requested for the operation type based on the job type and operation type for the target backup set includes:

[0011] When the job type for the target backup set is a recovery job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock;

[0012] When the job type for the target backup set is a copy job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock;

[0013] When the job type for the target backup set is a copy job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock;

[0014] When the job type for the target backup set is a backup job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock.

[0015] When the job type for the target backup set is a recycling job and the operation type is a delete operation, the lock type requested by the operation type is determined to be an exclusive lock.

[0016] In one embodiment, the step of determining the lock type and the operation type based on the job request, and determining the lock response result according to a pre-set conflict detection logic for various operation request locks, includes:

[0017] When the lock type is determined to be a shared lock and the operation type to be a read operation based on the job request, if a shared lock or a copy-on-write lock exists on the target backup set, the lock response result is determined to be to send a shared lock back to the backup set user.

[0018] When the lock type is determined to be a shared lock and the operation type to be a read operation based on the job request, if an exclusive lock exists on the target backup set, the lock response result is determined to be a refusal to send the lock back to the backup set user.

[0019] Based on the lock response result, the target backup set is processed, including:

[0020] When the lock manager determines that the shared lock is being fed back based on the lock response result, a read operation is performed on the target backup set;

[0021] When it is determined from the lock response result that the lock manager refuses to provide feedback on the lock, no read operation is performed on the target backup set.

[0022] In one embodiment, the step of determining the lock type and operation type based on the job request, and determining the lock response result according to the pre-set conflict detection logic for various operation request locks, includes:

[0023] When the lock type is determined to be an exclusive lock and the operation type to be a write operation based on the job request, if a shared lock exists on the target backup set, the lock response result is determined to be to send a write-on-replication lock back to the backup set user.

[0024] When the lock type is determined to be an exclusive lock and the operation type to be a write operation based on the job request, if an exclusive lock or a copy-on-write lock exists on the target backup set, the lock response result is determined to be to refuse to send the lock back to the backup set user.

[0025] Based on the lock response result, the target backup set is processed, including:

[0026] When the lock manager determines that a copy-on-write lock has been returned based on the lock response result, the conflicting data in the target backup set is copied to obtain a copy of the target backup set; the conflicting data is data that has been simultaneously subjected to write and read operations.

[0027] Perform write operations on the target backup set copy and read operations on the conflicting data;

[0028] When write and read operations are completed, the metadata pointer of the target backup set replica is atomically updated, and the conflicting data is deleted.

[0029] When it is determined from the lock response result that the lock manager refuses to provide feedback on the lock, no write operation is performed on the target backup set.

[0030] In one embodiment, the step of determining the lock type and operation type based on the job request, and determining the lock response result according to the pre-set conflict detection logic for various operation request locks, includes:

[0031] When the lock type is determined to be an exclusive lock and the operation type is a delete operation based on the job request, if there is no shared lock, exclusive lock or copy-on-write lock on the target backup set, the lock response result is determined to be to send an exclusive lock back to the backup set user.

[0032] When the lock type is determined to be an exclusive lock and the operation type is a delete operation based on the job request, if a shared lock, exclusive lock or copy-on-write lock exists on the target backup set, the lock response result is determined to be to refuse to send the lock back to the backup set user.

[0033] Based on the lock response result, the target backup set is processed, including:

[0034] When it is determined from the lock response result that the lock manager has returned an exclusive lock, the target backup set is deleted.

[0035] When it is determined from the lock response result that the lock manager refuses to provide feedback on the lock, the target backup set is not deleted.

[0036] In one embodiment, the lock manager is further configured to maintain a lock timeout period for the job request, and release the lock corresponding to the job request after the lock timeout period has expired.

[0037] In one embodiment, the lock manager is further configured to maintain a lock timeout for the job request according to the specified lock timeout when the job request specifies a lock timeout time;

[0038] When the job request does not specify a lock timeout, the lock timeout for the job request is maintained according to the preset lock timeout.

[0039] In one embodiment, the backup set user is further configured to call the lock renewal interface to send a lock timeout extension request to the lock manager when the lock timeout period of the job request is about to be reached;

[0040] The lock manager is also configured to extend the lock timeout period of the job request in response to the lock timeout extension request.

[0041] Secondly, this application also provides a backup set user terminal, which is the backup set user terminal included in the system described in the above embodiments.

[0042] Thirdly, this application also provides a lock manager, which is the lock manager included in the system described in the above embodiments.

[0043] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, the computer program being executed by a processor of the steps described in the above embodiments.

[0044] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that is executed by a processor through the steps described in the above embodiments.

[0045] The aforementioned backup set execution processing system includes a backup set user and a lock manager. The backup set user is used to determine the lock type requested for the operation type based on the job type and operation type for the target backup set. The backup set user is also used to generate a job request for the target backup set based on the lock type and operation type and send it to the lock manager. The lock manager is used to determine the lock type and operation type based on the job request, determine the lock response result according to pre-set conflict detection logic for various operation request locks, and feed it back to the backup set user. The backup set user is used to perform processing on the target backup set based on the lock response result. This application's backup set user generates a job request for the target backup set based on the lock type and operation type and sends it to the lock manager; the lock manager determines the lock type and operation type based on the job request, determines the lock response result according to pre-set conflict detection logic for various operation request locks, and feeds it back to the backup set user; the backup set user performs processing on the target backup set based on the lock response result, which can improve job execution efficiency and coordinate conflicts caused by multiple concurrent job operations sharing metadata. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the backup job in one embodiment;

[0048] Figure 2 This is a flowchart illustrating the recovery process in one embodiment;

[0049] Figure 3 This is a flowchart illustrating the replication job in one embodiment;

[0050] Figure 4 This is a flowchart illustrating the recycling operation in one embodiment;

[0051] Figure 5 This is an application environment diagram of the backup set execution processing system in one embodiment;

[0052] Figure 6 This is a schematic diagram of the conflict detection logic for various operation request locks in one embodiment;

[0053] Figure 7 This is a schematic diagram of the locking process in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that the terms "comprising" and "having," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusion. The term "multiple" as used in this application refers to two or more. The term "and / or" as used in this application refers to one of the solutions, or any combination of multiple solutions.

[0056] To better understand the backup set execution processing system, the following details the four types of jobs that perform processing on backup sets: backup jobs, copy jobs, recovery jobs, and recycle jobs.

[0057] The process of reading data files from the client, generating a backup set, and writing it to the storage pool is called a backup job. The specific execution process of a backup job is as follows: Figure 1 As shown. A backup set is a collection of physical files representing the data set generated by a backup operation. Stored on a storage server, it is the smallest recoverable unit in the backup system and logically belongs to a storage pool. A storage pool is a logical storage unit managed by the backup server, used to organize and store backup sets. Each backup set has corresponding metadata describing its status, such as its size, storage time, and physical file locations.

[0058] The process of reading backup data from a storage pool and restoring it to a specified location on the client is called a recovery job. The specific execution process of a recovery job is as follows: Figure 2 As shown.

[0059] The process of copying a backup set from a source storage pool to a destination storage pool to create a new backup set is called a replication job. The specific execution process of a replication job is as follows: Figure 3 As shown.

[0060] The process of deleting a backup set from a storage pool and clearing its associated metadata is called a recycling job. The specific execution process of a recycling job is as follows: Figure 4 As shown.

[0061] The backup set execution processing system provided in this application embodiment can be applied to, for example... Figure 5In the application environment shown, the backup set execution processing system includes a backup set user and a lock manager. The backup set user is used to determine the lock type requested for the operation type based on the job type and operation type for the target backup set. The backup set user is also used to generate a job request for the target backup set based on the lock type and operation type and send it to the lock manager. The lock manager is used to determine the lock type and operation type based on the job request, determine the lock response result according to the pre-set conflict detection logic for various operation request locks, and feed it back to the backup set user. The backup set user is used to perform processing on the target backup set based on the lock response result.

[0062] The entity that performs processing on the backup set can be called the backup set user. For example... Figures 1-4 As shown, in backup, copy, restore, and recycle scenarios, the backup set user is the backup server. The backup set to be processed on the backup set user can be used as the target backup set.

[0063] The job types for the target backup set include backup jobs, copy jobs, restore jobs, and reclaim jobs. Each job type has a corresponding operation type: backup jobs correspond to write operations, copy jobs correspond to read and write operations, restore jobs correspond to read operations, and reclaim jobs correspond to delete operations.

[0064] The backup set user can determine the lock type to request based on the job type and operation type for the target backup set.

[0065] Specifically, when the job type for the target backup set is a recovery job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock; when the job type for the target backup set is a replication job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock; when the job type for the target backup set is a replication job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock; when the job type for the target backup set is a backup job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock; when the job type for the target backup set is a recycling job and the operation type is a delete operation, the lock type requested by the operation type is determined to be an exclusive lock. Exclusive locks include reentrant exclusive locks and non-reentrant exclusive locks, and the backup set user can specify the type of exclusive lock when sending the job request.

[0066] The backup set user can generate a job request for the target backup set and send it to the lock manager based on the lock type and operation type.

[0067] When the backup set user needs to perform processing on the target backup set, it can generate a universally unique identifier (uuid) and, based on the universally unique identifier, lock type, and operation type, generate a job request for the target backup set and send it to the lock manager.

[0068] The lock manager can determine the lock type and operation type requested by the backup set user based on the job request. The lock manager can determine whether to send a lock back to the backup set user and the type of lock to send back when sending a lock back to the backup set user according to the pre-set conflict detection logic for various operation request locks, thereby obtaining the lock response result.

[0069] The lock manager can feed back the lock response results to the backup set user, and the backup set user can perform processing on the target backup set based on the lock response results.

[0070] After the lock manager sends the lock response results back to the backup set user, it maintains information such as lock type, lock owner's universal unique identifier, and corresponding lock timeout for the lock corresponding to the job request.

[0071] The aforementioned backup set execution processing system involves the backup set user generating a job request for the target backup set based on the lock type and operation type, and sending it to the lock manager. The lock manager determines the lock type and operation type based on the job request, and determines the lock response result according to the pre-set conflict detection logic for various operation request locks, and feeds it back to the backup set user. The backup set user then processes the target backup set based on the lock response result, which can improve job execution efficiency and coordinate conflicts caused by multiple concurrent job operations sharing metadata.

[0072] In one embodiment, the lock type requested by the operation type is determined based on the job type and operation type for the target backup set, and the specific steps are as follows: when the job type for the target backup set is a recovery job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock; when the job type for the target backup set is a replication job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock; when the job type for the target backup set is a replication job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock; when the job type for the target backup set is a backup job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock; when the job type for the target backup set is a recycling job and the operation type is a delete operation, the lock type requested by the operation type is determined to be an exclusive lock.

[0073] The operation type for the target backup set can be determined based on the job type for the target backup set. The lock type requested by the operation type can be determined based on the setting rules for requesting shared locks for read operations and exclusive locks for write and delete operations.

[0074] When the job type for the target backup set is a recovery job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock; when the job type for the target backup set is a replication job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock.

[0075] When the job type for the target backup set is a copy job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock; when the job type for the target backup set is a backup job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock; when the job type for the target backup set is a recycle job and the operation type is a delete operation, the lock type requested by the operation type is determined to be an exclusive lock.

[0076] In this embodiment, the lock type requested by the operation type is determined according to the job type and operation type for the target backup set, so as to prepare the data for the job request to generate the target backup set.

[0077] In one embodiment, based on the job request, the lock type and operation type are determined. According to the pre-set conflict detection logic for various operation request locks, the lock response result is determined. The specific steps are as follows: When the lock type is determined to be a shared lock and the operation type to be a read operation based on the job request, if a shared lock or a copy-on-write lock exists on the target backup set, the lock response result is determined to be to provide feedback of the shared lock to the backup set user. A copy-on-write lock is a locking mechanism that allows multiple job requests to hold the lock simultaneously, perform read or write operations on the target backup set simultaneously, and allow other job requests to request and obtain the shared lock of the target backup set during the holding period. When the lock type is determined to be a shared lock and the operation type to be a read operation based on the job request, if an exclusive lock exists on the target backup set, the lock response result is determined to be to refuse to provide feedback of the lock to the backup set user.

[0078] Based on the lock response result, the target backup set is processed in the following steps: when it is determined from the lock response result that the lock manager is returning a shared lock, a read operation is performed on the target backup set; when it is determined from the lock response result that the lock manager is refusing to return a lock, no read operation is performed on the target backup set.

[0079] Copy-on-write locks are a locking mechanism that allows multiple jobs to request to hold a shared lock on the target backup set simultaneously, perform read or write operations on the target backup set at the same time, and allow other jobs to request and acquire the target backup set while it is being held.

[0080] When the lock type is determined to be a shared lock and the operation type to be a read operation based on the job request, if a shared lock or a copy-on-write (CoW) lock exists on the target backup set, it indicates that other jobs are requesting to perform read or write operations on the target backup set. In this case, read operations on the target backup set are allowed, and the lock response result can be determined to be a shared lock feedback to the backup set user, thereby ensuring that multiple backup set users can concurrently read or concurrently read and write the target backup set.

[0081] When the lock type is determined to be a shared lock and the operation type to be a read operation based on the job request, if an exclusive lock exists on the target backup set, it indicates that other jobs are requesting to perform write operations on the target backup set. In this case, read operations on the target backup set are not allowed, and the lock response result can be determined to be a refusal to send the lock back to the backup set user. This ensures that write operations performed on the target backup set by other backup set users are not interfered with.

[0082] When the lock response result indicates that the lock manager has granted read access to the target backup set, the backup set user can perform read operations on the target backup set. When the lock response result indicates that the lock manager has refused to grant read access to the target backup set, the backup set user has not granted read access to the target backup set, and the backup set user will not perform read operations on the target backup set.

[0083] Exclusive locks include reentrant exclusive locks and non-reentrant exclusive locks, while copy-on-write locks include reentrant copy-on-write locks and non-reentrant copy-on-write locks. Backup set users can specify the types of exclusive locks and copy-on-write locks when sending job requests.

[0084] For job requests with reentrant exclusive locks and reentrant copy-on-write locks, the lock manager will normally grant a lock if it detects that the job request's universally unique identifier already holds the lock. However, for job requests with non-reentrant exclusive locks and non-reentrant copy-on-write locks, since the same universally unique identifier is not allowed to request the lock repeatedly, the lock manager will refuse to grant a lock if it detects that the job request's universally unique identifier already holds the lock.

[0085] Reentrant exclusive locks and reentrant copy-on-write locks are suitable for scenarios where lock ownership needs to be transferred between different nodes. The reentrant design avoids the situation where multiple lock requests cause blocking failures. Non-reentrant exclusive locks and non-reentrant copy-on-write locks are suitable for scenarios where only a certain process is allowed to request the lock. Other processes will fail to request the lock even if they obtain the universally unique identifier.

[0086] In this embodiment, when the lock type is determined to be a shared lock and the operation type to be a read operation based on the job request, if a shared lock or a copy-on-write lock exists on the target backup set, the lock response result is determined to provide a shared lock to the backup set user. If an exclusive lock exists on the target backup set, the lock response result is determined to refuse to provide a lock to the backup set user. When the lock manager provides a shared lock based on the lock response result, a read operation is performed on the target backup set. When the lock manager refuses to provide a lock based on the lock response result, no read operation is performed on the target backup set. This ensures that multiple backup set users can concurrently read or concurrently read and write to the target backup set, and also ensures that write operations performed on the target backup set by other backup set users are not interfered with.

[0087] In one embodiment, based on the job request, the lock type and operation type are determined, and the lock response result is determined according to the pre-set conflict detection logic for various operation request locks. The specific steps are as follows: When the lock type is determined to be an exclusive lock and the operation type is a write operation based on the job request, if a shared lock exists on the target backup set, the lock response result is determined to send a copy-on-write lock to the backup set user; when the lock type is determined to be an exclusive lock and the operation type is a write operation based on the job request, if an exclusive lock or a copy-on-write lock exists on the target backup set, the lock response result is determined to refuse to send a lock to the backup set user.

[0088] Based on the lock response result, the target backup set is processed in the following steps: When the lock response result determines that the lock manager is responding with a write-on-copy lock, the conflicting data in the target backup set is copied to obtain a copy of the target backup set; the conflicting data is data that has been simultaneously written and read; write operations are performed on the copy of the target backup set, and read operations are performed on the conflicting data; when the write and read operations are completed, the metadata pointer of the copy of the target backup set is atomically updated, and the conflicting data is deleted; when the lock response result determines that the lock manager is rejecting the response lock, no write operations are performed on the target backup set.

[0089] When the lock type is determined to be an exclusive lock and the operation type to be a write operation based on the job request, if a shared lock exists on the target backup set, it indicates that other jobs are requesting to read the target backup set. In this case, write operations on the target backup set are allowed. The lock manager can upgrade the exclusive lock to a copy-on-write lock and determine the lock response result as a copy-on-write lock to be fed back to the backup set user, thereby ensuring that multiple backup set users can concurrently read and write to the target backup set.

[0090] When the lock type is determined to be an exclusive lock and the operation type to be a write operation based on the job request, if an exclusive lock or a copy-on-write lock exists on the target backup set, it indicates that other jobs are requesting to perform write operations on the target backup set. In this case, write operations on the target backup set are not allowed, and the lock response result can be determined to be a refusal to send the lock back to the backup set user. This ensures that write operations performed on the target backup set by other backup set users are not interfered with.

[0091] When the lock manager responds with a write-on-copy lock based on the lock response result, it indicates that the backup set user has obtained write access to the target backup set. The backup set user can copy conflicting data in the target backup set to obtain a copy of the target backup set. Conflicting data is data that has been subject to both write and read operations. The backup set user can write to the target backup set copy, while other backup set users can read the conflicting data. When the write and read operations are completed, the metadata pointer of the target backup set copy is atomically updated, and the conflicting data is deleted. This allows for physical isolation between jobs corresponding to read and write operations, ensuring that read operations on the target backup set are not interfered with by write operations. Furthermore, copying only conflicting data in the target backup set reduces storage redundancy.

[0092] When the lock manager refuses to provide a feedback lock based on the lock response result, it indicates that the backup set user has not obtained write operation permission for the target backup set, and the backup set user will not perform write operations on the target backup set.

[0093] In this embodiment, when the lock type is determined to be an exclusive lock and the operation type to be a write operation based on the job request, if a shared lock exists on the target backup set, the lock response result is determined to be to send a copy-on-write lock to the backup set user; if an exclusive lock or a copy-on-write lock exists on the target backup set, the lock response result is determined to refuse to send a lock to the backup set user; when the lock manager is determined to send a copy-on-write lock based on the lock response result, the conflicting data in the target backup set is copied to obtain a copy of the target backup set; a write operation is performed on the copy of the target backup set, and a read operation is performed on the conflicting data; when the write and read operations are completed, the metadata pointer of the copy of the target backup set is atomically updated, and the conflicting data is deleted; when the lock manager is determined to refuse to send a lock based on the lock response result, no write operation is performed on the target backup set. It can ensure that write operations performed on the target backup set by other backup set users are not interfered with, and it can also ensure that multiple backup set users can concurrently read and write to the target backup set. Furthermore, it can physically isolate the jobs corresponding to read operations and the jobs corresponding to write operations, ensuring that read operations on the target backup set are not interfered with by write operations. In addition, it can reduce storage redundancy by copying only conflicting data in the target backup set.

[0094] In one embodiment, based on the job request, the lock type and operation type are determined, and the lock response result is determined according to the pre-set conflict detection logic for various operation request locks. The specific steps are as follows: When the lock type is determined to be an exclusive lock and the operation type is a delete operation based on the job request, if there is no shared lock, exclusive lock, or copy-on-write lock on the target backup set, the lock response result is determined to send an exclusive lock to the backup set user; when the lock type is determined to be an exclusive lock and the operation type is a delete operation based on the job request, if there is a shared lock, exclusive lock, or copy-on-write lock on the target backup set, the lock response result is determined to refuse to send the lock to the backup set user.

[0095] Based on the lock response result, the target backup set is processed in the following steps: when the lock response result determines that the lock manager is returning an exclusive lock, the target backup set is deleted; when the lock response result determines that the lock manager is refusing to return a lock, the target backup set is not deleted.

[0096] When the lock type is determined to be an exclusive lock and the operation type to be a delete operation based on the job request, if there is no shared lock, exclusive lock, or copy-on-write lock on the target backup set, it indicates that no other backup set user is performing write or read operations on the target backup set. In this case, the delete operation on the target backup set is allowed, and the lock response result can be determined to be an exclusive lock feedback to the backup set user.

[0097] When the lock type is determined to be an exclusive lock and the operation type to be a delete operation based on the job request, if a shared lock, exclusive lock, or copy-on-write lock exists on the target backup set, it indicates that other jobs are requesting write or read operations on the target backup set. In this case, the delete operation on the target backup set is not allowed, and the lock response result can be determined to be a refusal to send the lock back to the backup set user. This ensures that the write or read operations performed on the target backup set by other backup set users are not interfered with.

[0098] When the lock response result indicates that the lock manager has granted an exclusive lock, it means that the backup set user has obtained the permission to delete the target backup set, and the backup set user can delete the target backup set. When the lock response result indicates that the lock manager has refused to grant the lock, it means that the backup set user has not obtained the permission to delete the target backup set, and the backup set user will not delete the target backup set.

[0099] In this embodiment, when the lock type is determined to be an exclusive lock and the operation type to be a delete operation based on the job request, if there is no shared lock, exclusive lock, or copy-on-write lock on the target backup set, the lock response result is determined to provide an exclusive lock to the backup set user. If there is a shared lock, exclusive lock, or copy-on-write lock on the target backup set, the lock response result is determined to refuse to provide the lock to the backup set user. When the lock response result indicates that the lock manager provides an exclusive lock, the target backup set is deleted. When the lock response result indicates that the lock manager refuses to provide the lock, the target backup set is not deleted. This allows the target backup set to be deleted without interference from write or read operations performed by other backup set users, thus avoiding the situation where deleting backup set data during job recovery causes other jobs to fail.

[0100] In one embodiment, the lock manager is also configured to maintain a lock timeout period for the job request, and release the lock corresponding to the job request after the lock timeout period has elapsed.

[0101] The lock manager can maintain a lock timeout period for job requests. When the lock timeout period is exceeded, the lock corresponding to the job request will be released, which can prevent the situation where the lock resources are not released due to the offline status of the backup set user.

[0102] In this embodiment, the lock manager can maintain a lock timeout period for job requests. When the lock timeout period is exceeded, the lock corresponding to the job request is released. This ensures that different processes on the same node can initiate lock requests independently without the need to maintain a heartbeat for each node, thus saving memory and network resources.

[0103] In one embodiment, the lock manager is further configured to maintain the lock timeout for the job request according to the specified lock timeout when the job request specifies a lock timeout; and to maintain the lock timeout for the job request according to a preset lock timeout when the job request does not specify a lock timeout.

[0104] When a job request specifies a lock timeout, the lock manager can maintain the lock timeout for the job request based on the specified lock timeout. When a job request does not specify a lock timeout, the lock manager can set a preset lock timeout based on the time situation and maintain the lock timeout for the job request based on the preset lock timeout.

[0105] In this embodiment, when a job request specifies a lock timeout period, the lock timeout period is maintained for the job request according to the specified lock timeout period; when a job request does not specify a lock timeout period, the lock timeout period is maintained for the job request according to the preset lock timeout period. This allows for flexible setting of the lock timeout period to meet job requests.

[0106] In one embodiment, the backup set user is further configured to call the lock renewal interface to send a lock timeout extension request to the lock manager when the lock timeout time of the job request is about to be reached; the lock manager is further configured to extend the lock timeout time of the job request in response to the lock timeout extension request.

[0107] When the lock timeout period for a job request is about to expire, the backup set user can call the lock renewal interface to send a lock timeout extension request to the lock manager. The lock manager can respond to the lock timeout extension request and extend the lock timeout period for the job request, thereby preventing the lock corresponding to the job request from being released due to timeout.

[0108] To better understand the execution processing system of the above backup set, an application embodiment of the execution processing system of the backup set of this application is described in detail below.

[0109] The backup set execution processing system focuses on the concurrent control of multiple jobs (backup jobs / copy jobs / recovery jobs / reclaim jobs) within the storage pool of the backup system. In the field of data backup, resources are often logically aggregated using the concept of virtual storage pools. A group of related files is then aggregated on the virtual storage pool using the concept of backup sets, which are managed through backup set metadata. File reading and writing require the use of metadata.

[0110] Against this backdrop, conflicts arising from concurrent operations of multiple jobs sharing metadata are becoming increasingly prominent, including read-write conflicts (e.g., when a recovery job reads data deleted by a recycling job, data inconsistency occurs and job recovery fails), write-write conflicts (e.g., when backup and copy jobs modify data simultaneously, data overwriting occurs, causing version inconsistencies), and long transaction blocking (e.g., global locks cause job serialization and reduced throughput).

[0111] To solve the above problems, the following method is currently used:

[0112] (1) Global lock scheme: When accessing metadata, a global lock is first added, but this will lead to a decrease in throughput and low job running efficiency in scenarios where jobs run frequently.

[0113] (2) Distributed lock solutions are difficult to avoid lock anomalies caused by split-brain and additional device resource deployment problems, and are also difficult to support the requirement of simultaneous reading and writing in data backup scenarios.

[0114] (3) Storage snapshot scheme, which adds a new version every time the metadata is changed, but this greatly increases the storage cost.

[0115] However, the above solutions are complex to implement, requiring the maintenance of heartbeats for each node, resulting in high memory and network consumption, and the lock granularity only supports the node level. Common distributed locks such as Remote Dictionary Server Lock (Redis) locks and Apache Zookeeper Lock (Zookeeper) locks are prone to split-brain scenarios and cannot support the concurrent read and write requirements of data backup scenarios. Distributed locks can easily become a bottleneck for backup operations. In addition, the lock design is not compatible with the product business characteristics in the data backup field, making it difficult to maximize the efficiency of the lock.

[0116] To address the aforementioned issues, this embodiment provides a backup data contention solution for backup, recovery, replication, and recycling scenarios based on an innovative mechanism of layered locking architecture and dynamic write-time replication. This solution can be applied to the execution processing system of the backup set. The system supports metadata locking layering, dynamic write-time replication optimization, and also supports reentrant and non-reentrant locks.

[0117] Metadata locking layering: Read operations (recovery jobs / copy jobs) request shared locks to achieve parallelism, while write operations and deletion operations (backup jobs / reclaim jobs) isolate conflicts through exclusive locks and the write-time copy lock triggering mechanism, introducing write-time copy locks to support read-write parallelism;

[0118] Dynamic write-time copy optimization: Data copies are created only on demand when a read lock is detected (non-full copying), combined with logical / physical deletion separation control (delayed cleanup of recycling jobs);

[0119] Supports both reentrant and non-reentrant locks, and supports lock permission transfer: For distributed task scenarios, it supports the same lock to be used collaboratively on the same task on different machines, which facilitates flexible business deployment.

[0120] To better understand the backup set execution processing system, the following details the four types of jobs that perform processing on backup sets: backup jobs, copy jobs, recovery jobs, and recycle jobs.

[0121] The process of reading data files from the client, generating a backup set, and writing it to the storage pool is called a backup job. The specific execution process of a backup job is as follows: Figure 1 As shown in the diagram. A backup set represents the data collection generated during a backup operation, stored on a storage server. It is the smallest recoverable unit of the backup system and logically belongs to a specific storage pool. A storage pool is a logical storage unit managed by the backup server, used to organize and store backup sets.

[0122] The process of reading backup data from a storage pool and restoring it to a specified location on the client is called a recovery job. The specific execution process of a recovery job is as follows: Figure 2 As shown.

[0123] The process of copying a backup set from a source storage pool to a destination storage pool to create a new backup set is called a replication job. The specific execution process of a replication job is as follows: Figure 3 As shown.

[0124] The process of deleting a backup set from a storage pool and clearing its associated metadata is called a recycling job. The specific execution process of a recycling job is as follows: Figure 4 As shown.

[0125] The original data lock design for the backup set is shown in Table 1. The operation type for the target backup set can be determined based on the job type of the target backup set. The lock type requested by the operation type can be determined based on the setting rules of requesting shared locks for read operations and exclusive locks for write and delete operations.

[0126] Table 1. Backup Set Source Data Lock Design

[0127]

[0128] When the job type for the target backup set is a recovery job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock; when the job type for the target backup set is a replication job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock.

[0129] When the job type for the target backup set is a copy job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock; when the job type for the target backup set is a backup job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock; when the job type for the target backup set is a recycle job and the operation type is a delete operation, the lock type requested by the operation type is determined to be an exclusive lock.

[0130] The lock type and operation type can be determined based on the job request, as shown in Table 2 and... Figure 6 The pre-set conflict detection logic for various operation request locks is shown to determine the lock response result.

[0131] Table 2. Mutual Exclusion Relationships of Locks and Lock Conflict Detection Logic

[0132]

[0133] The conflict detection logic for various operation request locks is as follows: Figure 6As shown. Backup set users are only allowed to request shared locks and exclusive locks. Copy-on-write locks are automatically detected and upgraded by the lock manager when responding to exclusive lock requests, based on potential conflicts.

[0134] Specifically, when the lock type is determined to be a shared lock and the operation type to be a read operation based on the job request, if a shared lock or a copy-on-write lock exists on the target backup set, the lock response result is to send a shared lock to the backup set user. When the lock type is determined to be a shared lock and the operation type to be a read operation based on the job request, if an exclusive lock exists on the target backup set, the lock response result is to refuse to send a lock to the backup set user. If the lock response result indicates that the lock manager is sending a shared lock, a read operation is performed on the target backup set; if the lock response result indicates that the lock manager is refusing to send a lock, no read operation is performed on the target backup set.

[0135] When the lock type is determined to be an exclusive lock and the operation type to be a write operation based on the job request, if a shared lock exists on the target backup set, the lock response result is to send a copy-on-write lock to the backup set user. When the lock type is determined to be an exclusive lock and the operation type to be a write operation based on the job request, if an exclusive lock or a copy-on-write lock exists on the target backup set, the lock response result is to refuse to send a lock to the backup set user. When the lock response result determines that the lock manager should send a copy-on-write lock, a dynamic copy-on-write mechanism is executed to copy conflicting data in the target backup set to obtain a copy of the target backup set. Conflicting data refers to data that has undergone both write and read operations. Write operations are performed on the copy of the target backup set, and read operations are performed on the conflicting data. When the write and read operations are completed, the metadata pointer of the copy of the target backup set is atomically updated, and the conflicting data is deleted. When the lock response result determines that the lock manager should refuse to send a lock, no write operation is performed on the target backup set.

[0136] Dynamic copy-on-write mechanism: When the backup set user requests an exclusive lock (E1 / E2), if an active shared lock (S0) is detected, the lock manager upgrades the exclusive lock to a copy-on-write lock and returns the lock information to the backup set user. After obtaining the copy-on-write lock, the backup set user needs to use the dynamic copy-on-write mechanism to write the data blocks of the backup set.

[0137] (1) Before writing the data blocks of the backup set, copy the conflicting data in the target backup set to obtain a copy of the target backup set (only copy the conflicting data files to reduce the amount of copying); where the conflicting data is the data that is simultaneously written and read.

[0138] (2) Backup set users can perform write operations on the target backup set copy, such as modifying the content of the target backup set copy; other backup set users can perform read operations on conflicting data;

[0139] (3) When write and read operations are completed, the metadata pointer of the target backup set replica is atomically updated; specifically, the metadata physical file of the conflicting data is pointed to the target backup set replica.

[0140] (4) Mark conflicting data (original data) as "logical deletion" and put it into the recycle bin for unified cleanup.

[0141] Steps (3) and (4) can be performed by the backup set manager after the job is completed and the copy-on-write lock is released.

[0142] The above methods ensure that multiple backup set users can concurrently read and write to the target backup set, and can physically isolate the jobs corresponding to read operations and the jobs corresponding to write operations, ensuring that read operations on the target backup set are not interfered with by write operations. In addition, only conflicting data in the target backup set is copied, which can reduce storage redundancy.

[0143] When the lock type is determined to be an exclusive lock and the operation type to be a delete operation based on the job request, if no shared lock, exclusive lock, or copy-on-write lock exists on the target backup set, the lock response result is to send an exclusive lock to the backup set user. Conversely, if the lock type is determined to be an exclusive lock and the operation type to be a delete operation based on the job request, and a shared lock, exclusive lock, or copy-on-write lock exists on the target backup set, the lock response result is to refuse to send the lock to the backup set user. If the lock response result indicates that the lock manager is sending an exclusive lock, the delete operation is performed on the target backup set; if the lock response result indicates that the lock manager refuses to send the lock, the delete operation is not performed on the target backup set.

[0144] The locking process is as follows Figure 7 As shown, when a backup set user needs to process a target backup set, it can generate a universally unique identifier (UUID) and, based on the UUID, lock type, and operation type, generate a job request for the target backup set and send it to the lock manager. The lock manager, based on the UUID, lock type, and operation type, confirms the returned lock type, lock information, or rejects the lock response. After the lock manager sends the lock response result back to the backup set user, it maintains information such as lock type, lock master's UUID, and corresponding lock timeout for the lock corresponding to the job request. The backup set user can use the lock information to know whether the target backup set has other services and the duration of the lock, enabling better decision-making and execution.

[0145] Exclusive locks include reentrant exclusive locks and non-reentrant exclusive locks, while copy-on-write locks include reentrant copy-on-write locks and non-reentrant copy-on-write locks. Backup set users can specify the types of exclusive locks and copy-on-write locks when sending job requests.

[0146] For job requests with reentrant exclusive locks and reentrant copy-on-write locks, the lock manager will normally grant a lock if it detects that the job request's universally unique identifier already holds the lock. However, for job requests with non-reentrant exclusive locks and non-reentrant copy-on-write locks, since the same universally unique identifier is not allowed to request the lock repeatedly, the lock manager will refuse to grant a lock if it detects that the job request's universally unique identifier already holds the lock.

[0147] Reentrant exclusive locks and reentrant copy-on-write locks are suitable for scenarios where lock ownership needs to be transferred between different nodes. The reentrant design avoids the situation where multiple lock requests cause blocking failures. Non-reentrant exclusive locks and non-reentrant copy-on-write locks are suitable for scenarios where only a certain process is allowed to request the lock. Other processes will fail to request the lock even if they obtain the universally unique identifier.

[0148] Lock timeout and lock renewal: The lock manager can maintain a lock timeout period for job requests. When the lock timeout period is exceeded, the lock corresponding to the job request will be released, which can prevent the situation where the lock resource is not released due to the backup set user being offline.

[0149] When a job request specifies a lock timeout, the lock manager can maintain the lock timeout for the job request based on the specified lock timeout. When a job request does not specify a lock timeout, the lock manager can set a preset lock timeout based on the time situation and maintain the lock timeout for the job request based on the preset lock timeout.

[0150] When the lock timeout period for a job request is about to expire, the backup set user can call the lock renewal interface to send a lock timeout extension request to the lock manager. The lock manager can respond to the lock timeout extension request and extend the lock timeout period for the job request, thereby preventing the lock corresponding to the job request from being released due to timeout.

[0151] Lock release and post-release operations: The lock can be passively released by the lock manager after a timeout, or it can be actively released.

[0152] When a lock is released, a post-operation can be specified, allowing the lock manager to perform the post-operation on the metadata before releasing the lock. For example, when a copy-on-write lock is released, the lock manager can point the physical metadata file of the conflicting data to the target backup set copy and move the conflicting data to the recycle bin, where it will be deleted in a unified manner.

[0153] The above-mentioned backup set execution processing system has the following beneficial effects:

[0154] (1) By adopting a lock timeout and renewal scheme, it is ensured that different processes on the same node can initiate lock requests independently, without the need to maintain a heartbeat for each node, which can save memory and network resources;

[0155] (2) Through shared locks, recovery jobs and copy jobs can concurrently read the same backup set from the same storage pool indefinitely;

[0156] (3) By using copy-on-write locks, it is ensured that when there is a read operation, there is also a write operation that can be performed at the same time, thereby supporting the backup job to write to the backup set while the copy job can read the same backup set in the same storage pool at the same time;

[0157] (4) By using exclusive locks, we can ensure that when there is a read or write operation, the recycling job will not delete data and cause other jobs to fail, thus improving the robustness of the system.

[0158] The aforementioned backup set execution processing system can improve job running efficiency, coordinate conflicts caused by multiple concurrent operations sharing metadata, avoid lock anomalies and additional device resource deployment issues caused by split-brain, reduce storage costs, and support the need for simultaneous reading and writing in data backup scenarios.

[0159] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0160] In one embodiment, a backup set user terminal is provided, which is the backup set user terminal included in the backup set execution processing system of the above embodiment.

[0161] In one embodiment, a lock manager is provided, which is the lock manager included in the execution processing system of the backup set described above.

[0162] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above embodiments.

[0163] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above embodiments.

[0164] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0165] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes described in the above embodiments. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0166] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0167] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A backup set execution processing system, characterized in that, The system includes a backup set user terminal and a lock manager; The backup set user terminal is used to determine the lock type requested for the operation type based on the job type and operation type for the target backup set; The backup set user is also used to generate a job request for the target backup set and send it to the lock manager based on the lock type and the operation type. The lock manager is configured to, when determining that the lock type is an exclusive lock and the operation type is a write operation based on the job request, if a shared lock exists on the target backup set, indicating that other jobs are requesting to perform read operations on the target backup set and allowing write operations on the target backup set, determine the lock response result as feeding back a write-time copy lock to the backup set user. When the lock type is determined to be an exclusive lock and the operation type to be a write operation based on the job request, if an exclusive lock or a copy-on-write lock exists on the target backup set, it indicates that there are other job requests to perform write operations on the target backup set, and write operations on the target backup set are not allowed. The lock response result is determined to be to refuse to send the lock back to the backup set user. The lock response result is then fed back to the backup set user. The backup set user is used to copy conflicting data in the target backup set to obtain a copy of the target backup set when the lock manager returns a copy-on-write lock based on the lock response result. The conflicting data refers to data that has been simultaneously written and read. Perform write operations on the target backup set copy and read operations on the conflicting data; When write and read operations are completed, the metadata pointer of the target backup set replica is atomically updated, and the conflicting data is deleted. When it is determined from the lock response result that the lock manager refuses to provide feedback on the lock, no write operation is performed on the target backup set; The copy-on-write lock is a locking mechanism that allows multiple jobs to request to hold the lock simultaneously, perform read or write operations on the target backup set at the same time, and allow other jobs to request and acquire the shared lock on the target backup set while it is held.

2. The system according to claim 1, characterized in that, The step of determining the lock type requested for the operation type based on the job type and operation type for the target backup set includes: When the job type for the target backup set is a recovery job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock; When the job type for the target backup set is a copy job and the operation type is a read operation, the lock type requested by the operation type is determined to be a shared lock; When the job type for the target backup set is a copy job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock; When the job type for the target backup set is a backup job and the operation type is a write operation, the lock type requested by the operation type is determined to be an exclusive lock. When the job type for the target backup set is a recycling job and the operation type is a delete operation, the lock type requested by the operation type is determined to be an exclusive lock.

3. The system according to claim 1, characterized in that, The lock manager is also configured to, when determining that the lock type is a shared lock and the operation type is a read operation based on the job request, determine that if a shared lock or a copy-on-write lock exists on the target backup set, the lock response result is to send a shared lock back to the backup set user. When the lock type is determined to be a shared lock and the operation type to be a read operation based on the job request, if an exclusive lock exists on the target backup set, the lock response result is determined to be a refusal to send the lock back to the backup set user. The backup set user is also configured to perform a read operation on the target backup set when it is determined from the lock response result that the lock manager is returning a shared lock; and not to perform a read operation on the target backup set when it is determined from the lock response result that the lock manager is rejecting the return of the lock.

4. The system according to claim 1, characterized in that, The lock manager is also used to determine that when the lock type is an exclusive lock and the operation type is a delete operation based on the job request, if there is no shared lock, exclusive lock or copy-on-write lock on the target backup set, the lock response result is to send an exclusive lock back to the backup set user. When the lock type is determined to be an exclusive lock and the operation type is a delete operation based on the job request, if a shared lock, exclusive lock or copy-on-write lock exists on the target backup set, the lock response result is determined to be to refuse to send the lock back to the backup set user. The backup set user terminal is also used to delete the target backup set when it is determined from the lock response result that the lock manager is returning an exclusive lock; and not to delete the target backup set when it is determined from the lock response result that the lock manager is refusing to return a lock.

5. The system according to claim 1, characterized in that, The target backup set includes backup jobs, copy jobs, recovery jobs, and recycle jobs.

6. The system according to claim 1, characterized in that, The lock manager is also used to maintain a lock timeout period for the job request, and release the lock corresponding to the job request after the lock timeout period has expired.

7. The system according to claim 6, characterized in that, The lock manager is also configured to maintain the lock timeout for the job request according to the specified lock timeout when the job request specifies the lock timeout time; When the job request does not specify a lock timeout, the lock timeout for the job request is maintained according to the preset lock timeout.

8. The system according to claim 7, characterized in that, The backup set user is also used to send a lock timeout extension request to the lock manager by calling the lock renewal interface when the lock timeout period of the job request is about to be reached. The lock manager is also configured to extend the lock timeout period of the job request in response to the lock timeout extension request.

Images