Space recovery method and device for online increment reconstruction process

Abstract

The embodiment of the invention provides a space recovery method and device for an online incremental reconstruction process, and relates to the technical field of data processing.The method specifically comprises the steps that a target object where a to-be-deleted data block is located is determined in a first target copy according to a target index key, and deletion operation is executed on the target index key in the target object; after the target index key is deleted, checking the target object, and if other effective index keys do not exist in the target object, configuring a minimum deletion timestamp for the target object; when the distributed system background traverses each object in the first target copy, obtaining a latest operation timestamp corresponding to the target object; if a minimum deletion timestamp which is the latest operation timestamp of the target object exists, deleting the target object and performing space recovery on the target object; and if the write operation timestamp is greater than the minimum deletion timestamp, reserving the target object. According to the invention, the overall performance and data security of the distributed storage framework can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a space recovery method and device in an online incremental reconstruction process. Background Art

[0002] With the globalization of data, the size of metadata will also expand rapidly. At present, distributed systems are often used to expand storage capacity and processing capabilities by adding new storage nodes to meet the growing demand for metadata. In a distributed storage framework, effective management of metadata is critical to system performance and data integrity. In order to facilitate the location and operation of metadata data blocks, metadata indexes usually use the format of object name-object key-object offset-len. This method can more clearly identify the location and size of specific data blocks in the storage object, such as object1-key1-offset1-len1.

[0003] In terms of metadata recovery mechanism, it mainly relies on two methods: background aggregation process and client-sent deletion operation. The background aggregation process is to check metadata regularly to determine whether the data block is overwritten or deleted, so as to reclaim the old metadata space overwritten by new data. It is suitable for scenarios where data blocks are frequently updated. The client-sent deletion operation is that the user initiates the deletion instruction through the client, marks the deletion timestamp on the key of the data block corresponding to the metadata, and the background aggregation process will then actually delete the data block and reclaim the space according to the key. This greatly reduces the latency of the client deletion operation, especially for scenarios where users actively delete data blocks.

[0004] However, the existing mechanism has obvious defects in special scenarios. When deleting a key in an object, the system will check whether all keys under the object have been deleted. If all are deleted, the object is considered to be deletable and the latest deletion timestamp is stamped. However, in the online incremental reconstruction scenario, some keys under the object may not be in place because they are being migrated. At this time, the system may mistakenly identify that there are no valid keys under the object, and then stamp the object with a deletion timestamp. Even if the relevant keys are subsequently migrated, the write timestamp when they are written to the new copy during the migration process is smaller than the deletion timestamp of the object. Therefore, the metadata corresponding to the data blocks that have been migrated will not be visible to the outside world, resulting in data loss.

[0005] The above-mentioned risk of data loss seriously affects the data reliability and stability of the distributed storage framework in complex scenarios. A new technical solution is urgently needed to solve these problems in order to improve the overall performance and data security of the distributed storage framework. Summary of the invention

[0006] In view of this, an embodiment of the present application provides a method and device for space recovery in an online incremental reconstruction process, which can be used in a scenario of multi-person collaborative automated test case writing to prevent virus and Trojan samples from being detected and killed.

[0007] In a first aspect, an embodiment of the present application provides a space recovery method for an online incremental reconstruction process, including:

[0008] Receive a deletion instruction of a data block to be deleted issued by a client, and obtain a target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted;

[0009] Obtaining a first target replica in the distributed system; the first target replica is a replica in the distributed system that is in an online incremental reconstruction state;

[0010] Determine the target object where the to-be-deleted data block is located in the first target copy according to the target index key, and perform a deletion operation on the target index key in the target object; the target object includes multiple index keys including the target index key;

[0011] After deleting the target index key, the target object is checked. If no other valid index key exists in the target object, a minimum deletion timestamp is configured for the target object; the minimum deletion timestamp is the first timestamp starting from Greenwich Mean Time;

[0012] When the distributed system background traverses each object in the first target copy, the latest operation timestamp corresponding to the target object is obtained;

[0013] If the minimum deletion timestamp exists as the latest operation timestamp of the target object, the target object is deleted and the space is reclaimed;

[0014] If there is a write operation timestamp greater than the minimum deletion timestamp, the target object is retained.

[0015] As an optional implementation method of the embodiment of the present application, determining the target object where the data block to be deleted is located in the first target copy according to the target index key, and performing a deletion operation on the target index key in the target object, includes:

[0016] Determine the target object where the to-be-deleted data block is located in the first target replica according to the target index key, and mark a deletion tag for the target index key; the deletion tag is used to instruct the distributed system to delete the to-be-deleted data block corresponding to the target index key in the background;

[0017] When the distributed system background traverses the target object in the first target copy, the to-be-deleted data block corresponding to the target index key in the target object is deleted according to the deletion label on the target index key.

[0018] As an optional implementation of the embodiment of the present application, obtaining the first target replica in the distributed system includes:

[0019] The state labels corresponding to the multiple replicas in the distributed system are obtained, and the replica whose state label is a reconstruction label is used as the first target replica.

[0020] As an optional implementation of the embodiment of the present application, before obtaining the status labels corresponding to the multiple replicas in the distributed system and taking the replica whose status label is the reconstruction label as the first target replica, the method further includes:

[0021] The states corresponding to the multiple replicas in the distributed system are identified, and the replicas in the online incremental reconstruction state are marked with reconstruction tags.

[0022] As an optional implementation of the embodiment of the present application, after receiving a deletion instruction of the data block to be deleted issued by the client and obtaining a target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted, the method further includes:

[0023] Acquire a second target replica in the distributed system; the second target replica is a replica in a normal state in the distributed system;

[0024] Determine the target object where the to-be-deleted data block is located in the second target replica according to the target index key, and perform a deletion operation on the target index key in the target object;

[0025] After deleting the target index key, the target object is checked. If no other valid index key exists in the target object, a normal deletion timestamp is configured for the target object; the normal timestamp is a timestamp corresponding to the current time of the distributed system when it is checked that no other valid data blocks exist in the target object;

[0026] When the distributed system background traverses each object in the second target copy, the latest operation timestamp corresponding to the target object is obtained;

[0027] If the latest operation timestamp corresponding to the target object is the normal deletion timestamp, the target object is deleted and the space is reclaimed.

[0028] As an optional implementation of the embodiment of the present application, the method further includes:

[0029] When a write operation is performed on the target object, a write operation timestamp corresponding to when the write operation is issued is recorded, and the write operation timestamp is configured for the target object.

[0030] In a second aspect, an embodiment of the present application provides a space recovery device for an online incremental reconstruction process, including:

[0031] A receiving unit, configured to receive a deletion instruction of a data block to be deleted issued by a client, and obtain a target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted;

[0032] An acquisition unit, configured to acquire a first target replica in the distributed system; the first target replica is a replica in the distributed system that is in an online incremental reconstruction state;

[0033] a deleting unit, configured to determine, in the first target replica, a target object where the data block to be deleted is located according to the target index key, and perform a deletion operation on the target index key in the target object;

[0034] A checking unit, configured to check the target object after deleting the target index key, and if no other valid index key exists in the target object, configure a minimum deletion timestamp for the target object; the minimum deletion timestamp is the first timestamp starting from Greenwich Mean Time;

[0035] A traversal unit, configured to obtain a latest operation timestamp corresponding to the target object when the distributed system background traverses each object in the first target copy;

[0036] A processing unit, configured to delete the target object and reclaim space if the minimum deletion timestamp is the latest operation timestamp of the target object;

[0037] The processing unit is further configured to retain the target object if there is a write operation timestamp greater than the minimum deletion timestamp.

[0038] As an optional implementation party of an embodiment of the present application, the deletion unit is specifically used to determine the target object where the data block to be deleted is located in the first target copy according to the target index key, and mark a deletion tag for the target index key; the deletion tag is used to instruct the distributed system to delete the data block to be deleted corresponding to the target index key in the background; when the distributed system background traverses the target object in the first target copy, the data block to be deleted corresponding to the target index key in the target object is deleted according to the deletion tag on the target index key.

[0039] As an optional implementation of the embodiment of the present application, the acquisition unit is specifically used to obtain status labels corresponding to multiple replicas in the distributed system, and use the replica whose status label is a reconstruction label as the first target replica.

[0040] As an optional implementation of the embodiment of the present application, the acquisition unit is also used to identify the states corresponding to multiple replicas in the distributed system, and to identify the replicas in the online incremental reconstruction state with a reconstruction label.

[0041] As an optional implementation of the embodiment of the present application, the deletion unit is also used to obtain a second target copy in the distributed system; the second target copy is a copy in a normal state in the distributed system; the target object where the data block to be deleted is located is determined in the second target copy according to the target index key, and a deletion operation is performed on the target index key in the target object; after deleting the target index key, the target object is checked, and if there are no other valid index keys in the target object, a normal deletion timestamp is configured for the target object; the normal timestamp is the timestamp corresponding to the current time of the distributed system when it is checked that there are no other valid data blocks in the target object; when the distributed system background traverses each object in the second target copy, it obtains the latest operation timestamp corresponding to the target object; if the latest operation timestamp corresponding to the target object is the normal deletion timestamp, the target object is deleted and the space is reclaimed.

[0042] As an optional implementation of the embodiment of the present application, the processing unit is also used to record the write operation timestamp corresponding to when the write operation is issued when the write operation is performed on the target object, and configure the write operation timestamp for the target object.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the virus sample protection method for a test case described in any of the above embodiments when executing the computer program.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the virus sample protection method for the test case described in any of the above embodiments.

[0045] The virus sample protection method for the test case provided by the embodiment of the present application is specifically as follows: receiving a deletion instruction of a data block to be deleted issued by a client, and obtaining a target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted; obtaining a first target replica in the distributed system; the first target replica is a replica in the distributed system in an online incremental reconstruction state; determining a target object where the data block to be deleted is located in the first target replica according to the target index key, and performing a deletion operation on the target index key in the target object; multiple index keys including the target index key in the target object are deleted; key; after deleting the target index key, check the target object, if there is no other valid index key in the target object, configure a minimum deletion timestamp for the target object; the minimum deletion timestamp is the first timestamp starting from Greenwich Mean Time; when the distributed system background traverses each object in the first target copy, obtains the latest operation timestamp corresponding to the target object; if there is a minimum deletion timestamp that is the latest operation timestamp of the target object, delete the target object and reclaim space; if there is a write operation timestamp that is greater than the minimum deletion timestamp, retain the target object.

[0046] The beneficial effects of the present application are as follows: In order to avoid the phenomenon of data loss caused by the misidentification of objects and the addition of deletion timestamps due to the migration of some keys in the online incremental reconstruction scenario, the present application configures a minimum deletion timestamp for the target object, and then performs deletion and space recovery operations only when the minimum deletion timestamp is the latest operation timestamp of the target object, thereby avoiding the situation where objects are mistakenly deleted due to the incorrect judgment that there are no valid keys under the object, greatly reducing the risk of data loss, and improving the data reliability and stability of the distributed storage framework in complex scenarios. At the same time, for special scenarios such as online incremental reconstruction, the present application accurately identifies the status of the target object, avoids misjudgment during the key migration process, ensures the integrity and availability of data in complex scenarios, and enables the distributed storage framework to better adapt to complex and changing business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 One of the step flow charts of the virus sample protection method in the test case provided in the embodiment of the present application;

[0050] Figure 2 The second step flow chart of the method for protecting virus samples in a test case provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a framework of a virus sample protection method for a test case provided in an embodiment of the present application;

[0052] Figure 4 The third step flow chart of the method for protecting virus samples in a test case provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of a space recovery device for an online incremental reconstruction process provided by an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0057] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0058] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0059] Metadata contains detailed index information of objects and data blocks, such as object name, key, offset and length of data blocks. In a distributed storage environment, data is scattered among many nodes. When specific data needs to be accessed, the system can quickly determine the location of the data based on these indexes in the metadata, avoiding blindly searching the entire distributed storage space, greatly improving the speed and efficiency of data retrieval. For example, in a large-scale distributed file storage system, the file path and data block location information in the metadata can be used to quickly locate the file content required by the user, reducing the delay in data access.

[0060] The present application embodiment provides a space recovery method for an online incremental reconstruction process. Figure 1 As shown, the virus sample protection method for the test case includes the following steps S101-S105:

[0061] S101, receiving a deletion instruction of a data block to be deleted issued by a client, and acquiring a target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted.

[0062] In some embodiments, when the distributed system receives a deletion instruction for a data block to be deleted from a client, it will immediately parse the instruction to obtain the target index key corresponding to the data block to be deleted, and obtain the corresponding storage location of the data block in the distributed system based on the index key, thereby accurately locating the data to be deleted.

[0063] S102: Obtain a first target replica in the distributed system.

[0064] The first target replica is a replica in the distributed system that is in an online incremental reconstruction state.

[0065] In some embodiments, in a distributed system, a copy refers to redundant storage of data, that is, a copy of data is stored on multiple different nodes. The purpose of doing so is mainly to improve the reliability, fault tolerance and performance of the system. For example, in a distributed database system, a copy of the data may be stored on different servers. When one of the servers fails, the copies on other servers can still provide data services, avoiding data loss and service interruption.

[0066] Since the embodiment of the present application mainly solves the problem of avoiding data loss for the replica in the online incremental reconstruction state when receiving the deletion instruction of the data block to be deleted, in this step, it is necessary to first determine the replica in the distributed system that is in the online incremental reconstruction state, and then use the replica in the online incremental reconstruction state as the first target replica.

[0067] Among them, a replica in the online incremental reconstruction state refers to a state in which a replica is gradually recovering its data integrity after a node (such as a hard disk or server) where the replica is located fails in a distributed storage system. In this state, the replica is not completely unavailable, but partially available, and is recovering the lost data through incremental recovery, that is, only restoring the data objects that changed during the failure. Compared with full recovery, this method can use resources more efficiently, reduce recovery time and the impact on system performance.

[0068] S103: Determine the target object where the to-be-deleted data block is located in the first target copy according to the target index key, and perform a deletion operation on the target index key in the target object.

[0069] In a distributed system, metadata indexes usually use the format of object name-object key-object offset-len (offset and length), which can more clearly identify the location and size of specific data blocks in the storage object, such as the format of object1-key1-offset1-len1. After obtaining the target index key, the target object where the data block to be deleted is located can be determined in the first target copy according to the target index key, and the deletion operation can be performed on the target index key in the target object.

[0070] Specifically, the main function of marking deletion timestamps in distributed systems is to achieve delayed deletion. When a client initiates a deletion operation, the system marks the deletion timestamp in the key corresponding to the data block to be deleted instead of performing physical deletion immediately. This allows the client to quickly get feedback that the deletion operation has been accepted, greatly reducing the latency of the client's deletion operation. After initiating a deletion operation, the user will soon see the system prompt that the deletion is successful, without having to wait for a long time for the data to be completely removed from the storage device. At the same time, in a distributed system, data may be accessed and operated by multiple clients at the same time. Marking deletion timestamps can be used as a means of ensuring data consistency. Before the data block is actually deleted, the system can determine the status of the data block based on the timestamp and coordinate other operations (such as reading and updating).

[0071] Then, the data blocks marked with the deletion timestamp are deleted when the target object in the first target copy is traversed through the background aggregation process.

[0072] Specifically, the background aggregation process is a processing mechanism that runs in the background of the system. It is started periodically or under specific conditions and is specifically responsible for processing data blocks marked with deletion timestamps. The background aggregation process finds the corresponding actual data block based on the key recorded in the metadata. For example, in a distributed storage system, the key can be used to locate the specific node and location where the data block is stored. Then, the background aggregation process performs the actual data block deletion operation and reclaims the released storage space for subsequent reuse.

[0073] S104: After deleting the target index key, check the target object. If no other valid index key exists in the target object, configure a minimum deletion timestamp for the target object.

[0074] The minimum deletion timestamp is the first timestamp starting from Greenwich Mean Time.

[0075] It should be noted that in the data deletion process of a distributed system, deleting the target index key is only the first step. Further inspection of the target object and configuration of the minimum deletion timestamp under specific conditions are to manage system resources and data objects more finely. In a distributed system, data is organized in different objects in various forms, and each object contains multiple index keys and their corresponding data. After deleting a key, judging whether the target object still has other valid data helps decide whether the object needs to be completely cleaned up to avoid unnecessary resource occupation.

[0076] After deleting the target index key, the system will check the target object to determine whether there are other valid index keys. "Valid index keys" usually refer to data identifiers that are still associated with meaningful data and need to be accessed during system operation. For example, in a distributed system that stores user files, each file may consist of multiple data blocks, each of which is identified by an index key.

[0077] In combination with the embodiments of the present application, when the first target copy is in an online incremental reconstruction state, after the target index key in the target object in the first copy is deleted, there may be no other index keys in the target object, that is, there are no other valid index keys. In this case, since the first target copy is in an online incremental reconstruction state, it is also possible that the index key in the target object has not been migrated, resulting in the illusion that there are no other index keys in the current target object, which is easily deleted by mistake; it is also possible that there are indeed no other index keys in the current target object, that is, the current target object can be recycled.

[0078] Therefore, in order to avoid the above-mentioned situation where the index key in the target object is mistakenly deleted because it has not been migrated, when the copy is in the online incremental reconstruction state, when there are no other valid index keys in the target object after the key deletion operation is executed, the system will configure a minimum deletion timestamp for it, where the minimum deletion timestamp is defined as the first timestamp starting from Greenwich Mean Time (i.e., January 1, 1970 00:00:00 UTC). This specific time point is selected as the minimum deletion timestamp; and then through this minimum deletion timestamp, a clear and minimized time mark is used to facilitate the judgment basis in subsequent processing, indicating that the object has entered the final stage of "to be deleted". Specifically, the specific inspection method for checking the target object after deleting the target index key depends on the data structure of the target object. If the target object is a simple hash table structure, it may be possible to check whether there are other keys except the deleted key by traversing the key-value pairs of the hash table. If it is a more complex tree structure, it may be necessary to check the index key in the node according to the tree traversal algorithm (such as depth-first traversal or breadth-first traversal).

[0079] S105: When the distributed system background traverses each object in the first target copy, it obtains the latest operation timestamp corresponding to the target object.

[0080] In the daily maintenance of the distributed system, the background will regularly or irregularly traverse each object in the first target copy. This process is a key step for the system to check and optimize its own storage status. Through traversal, you can understand the current status of each object, and decide what operation to take on the target object based on different timestamps to ensure the effective use of system storage resources and data integrity.

[0081] Specifically, when traversing the objects in the first target copy in the background of the distributed system, the latest operation timestamp of each target object must first be determined. The timestamp is a mark that records the time when the data operation occurs in the system. For each target object, it records the time point of the most recent operation on the object. This way of obtaining the operation timestamp helps the system understand the state change history of the object. For example, in a distributed file storage system, each time a write or delete operation is performed on a file (target object), the corresponding timestamp is updated. By obtaining the latest operation timestamp, and then by judging the relationship between the latest operation timestamp and the minimum deletion timestamp, it is clear whether the target object needs to be deleted and recovered later.

[0082] S106: If the minimum deletion timestamp exists as the latest operation timestamp of the target object, the target object is deleted and the space is reclaimed.

[0083] Specifically, the minimum deletion timestamp is a special timestamp previously configured when the target object has no other valid index keys, indicating that the object is basically "useless" and is in a state of waiting for final cleanup. When the system finds that the minimum deletion timestamp is the latest operation timestamp of the target object, it means that no other new operations have changed the state of the object since the configuration of the timestamp, and the target object actually has no valid index keys, that is, the target object does not contain valid data. At this time, the system can perform a deletion operation to remove the target object from storage and reclaim the space it occupies for use by other data.

[0084] S107: If there is a write operation timestamp that is greater than the minimum deletion timestamp, retain the target object.

[0085] Specifically, the write operation timestamp records the time when the data is written to the target object. When the system detects that the write operation timestamp is greater than the minimum deletion timestamp, it indicates that new data has been written after the object was marked with the deletion tag, that is, the index key in the target object has not been migrated. After the target object was marked with the minimum deletion timestamp, a write operation was performed in the target object, and the existence of a write operation timestamp means that the target object still contains valid data, and deleting it will result in data loss. Therefore, the target object needs to be retained.

[0086] Through this timestamp-based judgment mechanism, the distributed system can intelligently manage storage resources, clean up useless objects in a timely manner while ensuring data integrity, and improve the storage efficiency and performance of the system.

[0087] In order to avoid the phenomenon of data loss caused by the misidentification of objects and the addition of deletion timestamps due to the migration of some keys in the online incremental reconstruction scenario, this application configures a minimum deletion timestamp for the target object, and then performs deletion and space recovery operations only when the minimum deletion timestamp is the latest operation timestamp of the target object, thereby avoiding the situation where objects are mistakenly deleted due to the incorrect judgment that there are no valid keys under the object, greatly reducing the risk of data loss, and improving the data reliability and stability of the distributed storage framework in complex scenarios. At the same time, for special scenarios such as online incremental reconstruction, this application accurately identifies the status of the target object to avoid misjudgment during the key migration process, ensures the integrity and availability of data in complex scenarios, and enables the distributed storage framework to better adapt to complex and changing business scenarios.

[0088] As an extension and refinement of the above embodiment, refer to Figure 1 As shown, another space recovery method for the online incremental reconstruction process is shown. The virus sample protection method for the test case includes the following steps S201-S208:

[0089] S201. Receive a deletion instruction of a data block to be deleted issued by a client, and obtain a target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted.

[0090] S202: Obtain status labels corresponding to multiple replicas in the distributed system, and use the replica whose status label is a reconstruction label as the first target replica.

[0091] In the embodiment of the present application, the reconstruction tag indicates that the replica is in online incremental reconstruction, which is a common maintenance operation in distributed systems, and may involve reorganizing data, repairing damaged data blocks, optimizing storage structures, etc. For example, when the system detects that the data of a replica is partially damaged, or the data layout of the replica needs to be adjusted to improve data access performance, the replica will be reconstructed.

[0092] Because the embodiment of the present application is directed to a space recovery method during an online incremental reconstruction process, it is necessary to first find a replica in an online incremental reconstruction state from multiple replicas in the distributed system, that is, the first target replica.

[0093] Specifically, before obtaining the status tags corresponding to the multiple replicas in the distributed system and taking the replica whose status tag is the reconstruction tag as the first target replica, the embodiment of the present application further needs to perform the following steps:

[0094] The states corresponding to the multiple replicas in the distributed system are identified, and the replicas in the online incremental reconstruction state are marked with reconstruction tags.

[0095] It should be noted that in a distributed system, a replica is a form of redundant data storage, and different replicas may be in multiple states. In order to facilitate subsequent special processing of replicas in the online incremental reconstruction state, it is necessary to first identify the states corresponding to multiple replicas in the distributed system, and identify the replicas in the online incremental reconstruction state with reconstruction tags. This will help the system to perform differentiated management of replicas, especially when performing operations such as data deletion, and subsequent specific strategies can be adopted for replicas in this special state.

[0096] Reference Figure 3 As shown, Figure 3 The replica 1.2 in the table is marked with a special flag: rebuilding, which means that the replica 1.2 is in the online incremental reconstruction state, and the other replicas 1.1 and 1.3 are in the non-online incremental reconstruction state.

[0097] S203: Determine the target object where the to-be-deleted data block is located in the first target copy according to the target index key, and mark a deletion tag for the target index key.

[0098] The deletion tag is used to instruct the distributed system to delete the to-be-deleted data block corresponding to the target index key in the background.

[0099] In some embodiments, in a distributed system, data is usually stored and organized through some data structure, such as a hash table, a tree structure, an object storage, etc. When the target index key is obtained, the system needs to find the target object corresponding to the key in the first target copy, and then mark the deletion tag for the target index key in the target object. The main purpose of marking the deletion tag is to delay the deletion operation, because directly deleting data may affect the performance and consistency of the system, especially in a distributed environment, which may involve multiple copies and complex operation processes. By marking the deletion tag, the system can delay the deletion operation to background processing, so that the client operation can be completed faster without waiting for the actual deletion operation to be completed. Doing so can avoid long lock waits or network delays, and improve the system's response speed and user experience.

[0100] Specifically, the implementation method of marking the target index key with a delete tag depends on the data storage structure. If the target object is a simple dictionary, a special key-value pair can be added to the dictionary item storing the target index key to indicate the delete tag.

[0101] S204: When the distributed system background traverses the target object in the first target copy, the to-be-deleted data block corresponding to the target index key in the target object is deleted according to the deletion tag on the target index key.

[0102] After the above step S203 is executed, when the distributed system traverses all keys in the target object in the first target copy in the background, the data block to be deleted corresponding to the target index key in the target object can be deleted according to the deletion label on the target index key.

[0103] It should be noted that in the embodiments of the present application, the traversal operation of the distributed system can be triggered by timing, such as setting a timer to start the background program and check the replicas at regular intervals; it can also be event-triggered, such as when the system load is low or certain specific conditions are met.

[0104] S205: After deleting the target index key, check the target object. If no other valid index key exists in the target object, configure a minimum deletion timestamp for the target object.

[0105] The minimum deletion timestamp is the first timestamp starting from Greenwich Mean Time.

[0106] S206: When the distributed system background traverses each object in the first target copy, it obtains the latest operation timestamp corresponding to the target object.

[0107] S207: If the minimum deletion timestamp exists as the latest operation timestamp of the target object, the target object is deleted and the space is reclaimed.

[0108] S208. If there is a write operation timestamp that is greater than the minimum deletion timestamp, retain the target object.

[0109] In an embodiment of the present application, if there is a write operation timestamp that is greater than the minimum deletion timestamp, it means that in the current online incremental reconstruction process, the index key in the target object in the first copy has not been migrated, but is marked with a minimum deletion timestamp, and after the target object is marked with the minimum deletion timestamp, a write operation is performed in the target object.

[0110] Therefore, the embodiment of the present application also needs to perform the following steps:

[0111] When a write operation is performed on the target object, a write operation timestamp corresponding to when the write operation is issued is recorded, and the write operation timestamp is configured for the target object.

[0112] Through the above steps, the write operation timestamp corresponding to the write operation is recorded, so as to facilitate subsequent comparison with the minimum deletion timestamp and avoid accidental deletion of the target object.

[0113] As an extension and refinement of the above embodiment, the embodiment of the present application illustrates that after receiving the deletion instruction of the data block to be deleted issued by the client and obtaining the target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted, when the replica status is normal, the following steps S401-S405 need to be performed:

[0114] S401. Obtain a second target copy in the distributed system.

[0115] The second target replica is a replica in a normal state in the distributed system.

[0116] Different from the embodiment provided by the above steps S101-S107, the embodiment of the present application mainly explains how to process the deletion instruction of the to-be-deleted data block issued by the client for the replica in the normal state in the distributed system.

[0117] Specifically, a replica in a normal state indicates that the replica is not in an online incremental reconstruction state, can normally receive and process various operations, including data writing and deletion operations, and has no ongoing abnormal operations or maintenance tasks.

[0118] S402: Determine the target object where the to-be-deleted data block is located in the second target copy according to the target index key, and perform a deletion operation on the target index key in the target object.

[0119] In the embodiment of the present application, similar to the previous operation in the first target copy, it is also necessary to find the target object where the data block to be deleted is located in the second target copy according to the target index key.

[0120] S403: After deleting the target index key, check the target object. If no other valid index key exists in the target object, configure a normal deletion timestamp for the target object.

[0121] The normal timestamp is a timestamp corresponding to the current time of the distributed system when it is checked that no other valid data blocks exist in the target object.

[0122] Specifically, in the second target copy, when there are no other valid index keys in the target object, a normal deletion timestamp is configured for the target object. The normal deletion timestamp is the time when it is checked that there are no other valid data blocks in the target object. It can use the current system time to generate the normal deletion timestamp.

[0123] S404: When the distributed system background traverses each object in the second target copy, it obtains the latest operation timestamp corresponding to the target object.

[0124] When traversing each object in the second target copy in the background of the distributed system, it is further determined whether to delete the target object and reclaim space based on the relationship between the latest operation timestamp of the target object and the normal deletion timestamp.

[0125] S405: If the latest operation timestamp corresponding to the target object is the normal deletion timestamp, the target object is deleted and the space is reclaimed.

[0126] Specifically, when the latest operation timestamp corresponding to the target object is a normal deletion timestamp, it indicates that there are no other valid index keys in the target object, that is, there is no valid data, and the target object is deleted and the space is reclaimed.

[0127] Furthermore, the biggest difference between the first target copy in the online incremental reconstruction state and the second target copy in the normal state after deleting the target index key is that when there are no other valid keys in the target object, the configured deletion timestamps are different. The first target copy is configured with a minimum deletion timestamp (the first timestamp starting from Greenwich Mean Time), and the second target copy is configured with a normal deletion timestamp, that is, the timestamp corresponding to when it is checked that there are no other valid data blocks in the target object.

[0128] This is because the first target replica is in an online incremental reconstruction state, and the system cannot immediately determine whether the data can be safely deleted. By setting a minimum deletion timestamp, the system will determine whether to delete the object based on this special timestamp and other conditions when traversing the replicas in the background. The data deletion operation is then postponed, giving the system enough time to ensure the integrity and consistency of the data during the reconstruction process. The second target replica is in a normal state and does not need to consider the complex reconstruction process like the first target replica, and there will be no phenomenon of index keys being migrated. Therefore, when the distributed system traverses the objects in the replicas, it can directly determine whether the data block can be deleted relatively quickly based on the normal deletion timestamp.

[0129] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a space recovery device for an online incremental reconstruction process. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will no longer repeat the details of the aforementioned method embodiment one by one, but it should be clear that a space recovery device for an online incremental reconstruction process in this embodiment can correspond to the implementation of all the contents in the aforementioned method embodiment.

[0130] The present application embodiment provides a space recovery device for an online incremental reconstruction process. Figure 5 is a schematic diagram of the structure of the space recovery device in the online incremental reconstruction process, such as Figure 5 As shown, the space recovery device 500 for the online incremental reconstruction process includes:

[0131] The receiving unit 501 is configured to receive a deletion instruction of a data block to be deleted issued by a client, and obtain a target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted;

[0132] An acquisition unit 502 is used to acquire a first target replica in the distributed system; the first target replica is a replica in the distributed system that is in an online incremental reconstruction state;

[0133] A deleting unit 503 is used to determine the target object where the to-be-deleted data block is located in the first target replica according to the target index key, and perform a deletion operation on the target index key in the target object;

[0134] A checking unit 504 is used to check the target object after deleting the target index key, and if there is no other valid index key in the target object, configure a minimum deletion timestamp for the target object; the minimum deletion timestamp is the first timestamp starting from Greenwich Mean Time;

[0135] A traversal unit 505, configured to obtain the latest operation timestamp corresponding to the target object when the distributed system background traverses each object in the first target copy;

[0136] The processing unit 506 is configured to delete the target object and reclaim space if the minimum deletion timestamp is the latest operation timestamp of the target object.

[0137] The processing unit 506 is further configured to retain the target object if there is a write operation timestamp greater than the minimum deletion timestamp.

[0138] As an optional implementation party of an embodiment of the present application, the deletion unit 503 is specifically used to determine the target object where the data block to be deleted is located in the first target copy according to the target index key, and mark a deletion tag for the target index key; the deletion tag is used to instruct the distributed system to delete the data block to be deleted corresponding to the target index key in the background; when the distributed system background traverses the target object in the first target copy, the data block to be deleted corresponding to the target index key in the target object is deleted according to the deletion tag on the target index key.

[0139] As an optional implementation of the embodiment of the present application, the acquisition unit 502 is specifically used to obtain status labels corresponding to multiple replicas in the distributed system, and use the replica whose status label is a reconstruction label as the first target replica.

[0140] As an optional implementation of the embodiment of the present application, the acquisition unit 502 is also used to identify the states corresponding to multiple replicas in the distributed system, and to identify the replicas in the online incremental reconstruction state with a reconstruction label.

[0141] As an optional implementation of the embodiment of the present application, the deletion unit 503 is also used to obtain a second target copy in the distributed system; the second target copy is a copy in a normal state in the distributed system; the target object where the data block to be deleted is located is determined in the second target copy according to the target index key, and the deletion operation is performed on the target index key in the target object; after deleting the target index key, the target object is checked, and if there are no other valid index keys in the target object, a normal deletion timestamp is configured for the target object; the normal timestamp is the timestamp corresponding to the current time of the distributed system when it is checked that there are no other valid data blocks in the target object; when the distributed system background traverses each object in the second target copy, the latest operation timestamp corresponding to the target object is obtained; if the latest operation timestamp corresponding to the target object is the normal deletion timestamp, the target object is deleted and the space is reclaimed.

[0142] As an optional implementation of the embodiment of the present application, the processing unit 506 is also used to record the write operation timestamp corresponding to when the write operation is performed on the target object, and configure the write operation timestamp for the target object.

[0143] Based on the same inventive concept, an embodiment of the present disclosure also provides an electronic device. Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as Figure 6 As shown, the electronic device provided in this embodiment includes: a memory 601 and a processor 602, wherein the memory 601 is used to store a computer program; and the processor 602 is used to execute the audio data processing method provided in the above embodiment when executing the computer program.

[0144] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computing device implements the virus sample protection method for the test case provided in the above embodiment.

[0145] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0146] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0147] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0148] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A space recovery method for an online incremental reconstruction process, characterized in that: Applied to distributed systems, including: Receive a deletion instruction of a data block to be deleted issued by a client, and obtain a target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted; Obtaining a first target replica in the distributed system; the first target replica is a replica in the distributed system that is in an online incremental reconstruction state; Determine the target object where the to-be-deleted data block is located in the first target copy according to the target index key, and perform a deletion operation on the target index key in the target object; the target object includes multiple index keys including the target index key; After deleting the target index key, the target object is checked. If no other valid index key exists in the target object, a minimum deletion timestamp is configured for the target object; the minimum deletion timestamp is the first timestamp starting from Greenwich Mean Time; When the distributed system background traverses each object in the first target copy, the latest operation timestamp corresponding to the target object is obtained; If the minimum deletion timestamp exists as the latest operation timestamp of the target object, the target object is deleted and the space is reclaimed; If there is a write operation timestamp greater than the minimum deletion timestamp, the target object is retained.

2. The method according to claim 1, characterized in that The step of determining the target object where the to-be-deleted data block is located in the first target replica according to the target index key, and performing a deletion operation on the target index key in the target object includes: Determine the target object where the to-be-deleted data block is located in the first target replica according to the target index key, and mark a deletion tag for the target index key; the deletion tag is used to instruct the distributed system to delete the to-be-deleted data block corresponding to the target index key in the background; When the distributed system background traverses the target object in the first target copy, the to-be-deleted data block corresponding to the target index key in the target object is deleted according to the deletion label on the target index key.

3. The method according to claim 1, characterized in that: The obtaining of the first target replica in the distributed system includes: The state labels corresponding to the multiple replicas in the distributed system are obtained, and the replica whose state label is a reconstruction label is used as the first target replica.

4. The method according to claim 3, characterized in that Before obtaining the status tags corresponding to the multiple replicas in the distributed system and taking the replica whose status tag is a reconstruction tag as the first target replica, the method further includes: The states corresponding to the multiple replicas in the distributed system are identified, and the replicas in the online incremental reconstruction state are marked with reconstruction tags.

5. The method according to claim 1, characterized in that After receiving the deletion instruction of the data block to be deleted sent by the client and obtaining the target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted, the method further includes: Acquire a second target replica in the distributed system; the second target replica is a replica in a normal state in the distributed system; Determine the target object where the to-be-deleted data block is located in the second target replica according to the target index key, and perform a deletion operation on the target index key in the target object; After deleting the target index key, the target object is checked. If no other valid index key exists in the target object, a normal deletion timestamp is configured for the target object; the normal timestamp is a timestamp corresponding to the current time of the distributed system when it is checked that no other valid data blocks exist in the target object; When the distributed system background traverses each object in the second target copy, the latest operation timestamp corresponding to the target object is obtained; If the latest operation timestamp corresponding to the target object is the normal deletion timestamp, the target object is deleted and the space is reclaimed.

6. The method according to claim 1, characterized in that The method further comprises: When a write operation is performed on the target object, a write operation timestamp corresponding to when the write operation is issued is recorded, and the write operation timestamp is configured for the target object.

7. A space recovery device for an online incremental reconstruction process, characterized in that: include: A receiving unit, configured to receive a deletion instruction of a data block to be deleted issued by a client, and obtain a target index key corresponding to the data block to be deleted according to the deletion instruction of the data block to be deleted; An acquisition unit, configured to acquire a first target replica in the distributed system; The first target replica is a replica in the distributed system that is in an online incremental reconstruction state; a deleting unit, configured to determine, in the first target replica, a target object where the data block to be deleted is located according to the target index key, and perform a deletion operation on the target index key in the target object; A checking unit, configured to check the target object after deleting the target index key, and if no other valid index key exists in the target object, configure a minimum deletion timestamp for the target object; the minimum deletion timestamp is the first timestamp starting from Greenwich Mean Time; A traversal unit, configured to obtain a latest operation timestamp corresponding to the target object when the distributed system background traverses each object in the first target copy; A processing unit, configured to delete the target object and reclaim space if the minimum deletion timestamp is the latest operation timestamp of the target object; The processing unit is further configured to retain the target object if there is a write operation timestamp greater than the minimum deletion timestamp.

8. The method according to claim 7, characterized in that The deleting unit is specifically used for: Determine the target object where the to-be-deleted data block is located in the first target replica according to the target index key, and mark a deletion tag for the target index key; the deletion tag is used to instruct the distributed system to delete the to-be-deleted data block corresponding to the target index key in the background; When the distributed system background traverses the target object in the first target copy, the to-be-deleted data block corresponding to the target index key in the target object is deleted according to the deletion label on the target index key.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the virus sample protection method for a test case as described in any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the virus sample protection method for a test case according to any one of claims 1 to 6.

Images