Junk data recovery method and device and storage medium

Abstract

The embodiment of the invention provides a junk data recovery method and device and a computer storage medium, and the method comprises the steps: in response to a junk data recovery request for a to-be-processed object, determining a storage area of the to-be-processed object in each disk; partial data of the to-be-processed object is stored in each storage area; determining a target storage area comprising junk data from the storage areas of the disks, and obtaining position information of the junk data in the target storage area; determining whether the position information meets a preset condition or not, and if yes, performing truncation operation on the junk data in the target storage area; the preset condition is that all the junk data are located at the head or the tail of the target storage area; therefore, the recovery efficiency of the junk data can be improved while the network bandwidth is saved.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a garbage data recovery method, device and computer storage medium. Background Art

[0002] Currently, in a distributed storage system based on the append-only mode, when a client overwrites an object, it does not directly overwrite the original data, but instead writes the data to the end in an append-only manner. Since this method will generate a lot of garbage data, it is necessary to introduce a garbage collection (GC) operation.

[0003] In the related art, when performing a garbage data recovery operation, it is necessary to first read out the valid data and then write the valid data down. This process not only wastes unnecessary network bandwidth, but also affects the garbage data recovery efficiency. Summary of the invention

[0004] Embodiments of the present application provide a garbage data recovery method, device, and computer storage medium.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a garbage data recovery method, the method comprising:

[0007] In response to a garbage data collection request for an object to be processed, determining storage areas of the object to be processed on each disk; each storage area stores part of the data of the object to be processed;

[0008] Determine a target storage area including junk data from the storage areas of each disk, and obtain location information of the junk data in the target storage area;

[0009] Determine whether the location information satisfies a preset condition, and if so, perform a truncation operation on the junk data in the target storage area; the preset condition is that all the junk data is located at the head or tail of the target storage area.

[0010] The present application provides a garbage data recovery device, the device comprising:

[0011] A first determination module is used to determine the storage areas of the object to be processed on each disk in response to a garbage data collection request for the object to be processed; each storage area stores part of the data of the object to be processed;

[0012] A second determining module is used to determine a target storage area including junk data from the storage areas of each disk, and obtain location information of the junk data in the target storage area;

[0013] The truncation module is used to determine whether the location information meets a preset condition, and if so, to perform a truncation operation on the junk data in the target storage area; the preset condition is that all the junk data is located at the head or tail of the target storage area.

[0014] The present application provides an electronic device, the device comprising a memory, a processor and a communication bus;

[0015] The communication bus is used to realize the communication connection between the processor and the memory;

[0016] The processor is used to execute the computer program stored in the memory to implement the garbage data recovery method provided by one or more of the above-mentioned technical solutions.

[0017] The present application provides a computer storage medium, which stores a computer program; after the computer program is executed, it can implement the garbage data recovery method provided by one or more of the above-mentioned technical solutions.

[0018] An embodiment of the present application provides a garbage data recovery method, device and computer storage medium, the method comprising: in response to a garbage data recovery request for an object to be processed, determining the storage area of ​​the object to be processed on each disk; each storage area stores part of the data of the object to be processed; determining a target storage area including garbage data from the storage areas of each disk, and obtaining location information of the garbage data in the target storage area; determining whether the location information meets a preset condition, and if so, truncating the garbage data in the target storage area; the preset condition is that all the garbage data is located at the head or tail of the target storage area.

[0019] It can be seen that in the embodiment of the present application, in the process of recycling garbage data, once it is determined that the garbage data in the disk is located at the head or tail of the target storage area, the garbage data is directly truncated to release storage space; that is, the entire process does not require first reading the valid data in the storage area and then writing the valid data. In this way, the efficiency of garbage data recycling can be improved while saving network bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an architectural diagram of a distributed storage system based on the append-only mode;

[0021] Figure 2AIt is a schematic diagram corresponding to the first step in the garbage data recovery method;

[0022] Figure 2B It is a schematic diagram corresponding to the second step in the garbage data recovery method;

[0023] Figure 2C It is a schematic diagram corresponding to the third step in the garbage data recovery method;

[0024] Figure 2D It is a schematic diagram corresponding to the fourth step in the garbage data recovery method;

[0025] Figure 3 A flowchart of a garbage data recovery method according to an embodiment of the present application;

[0026] Figure 4A A schematic diagram of a truncation operation on junk data according to an embodiment of the present application;

[0027] Figure 4B A schematic diagram of another method of performing a truncation operation on junk data according to an embodiment of the present application;

[0028] Figure 5 A schematic diagram of a migration operation on valid data in an embodiment of the present application

[0029] Figure 6 A schematic diagram of the structure of a garbage data recovery device according to an embodiment of the present application;

[0030] Figure 7 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0031] The present application will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the embodiments provided herein are only used to explain the present application and are not intended to limit the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions recorded in the embodiments of the present application can be implemented in any combination.

[0032] It should be noted that, in the embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or device including a series of elements includes not only the elements explicitly recorded, but also includes other elements not explicitly listed, or also includes elements inherent to the implementation of the method or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other related elements (such as steps in a method or units in a device, for example, a unit may be a part of a processor, a part of a program or software, etc.) in the method or device including the element.

[0033] For example, the garbage data recovery method provided in the embodiment of the present application includes a series of steps, but the garbage data recovery method provided in the embodiment of the present application is not limited to the recorded steps. Similarly, the garbage data recovery device provided in the embodiment of the present application includes a series of modules, but the garbage data recovery device provided in the embodiment of the present application is not limited to including the modules explicitly recorded, and may also include modules required to obtain relevant data or perform processing based on data.

[0034] In the related art, when performing a garbage data recovery operation, it is necessary to first read out the valid data and then write the valid data down. This process not only wastes unnecessary network bandwidth, but also affects the garbage data recovery efficiency.

[0035] Figure 1 This is an architecture diagram of a distributed storage system based on the append-only mode. Figure 1 As shown in the figure, the client object data is stored in the storage area of ​​different disks in the mode of erasure code (EC), corresponding to Figure 1 Shard1, shard2 and shard3 in the disk storage area, the correspondence between the data in the disk storage area and the client object data, that is, metadata, is recorded in the distributed key-value (KV) storage database. Based on this architecture, the process of client reading data is as follows: 1) The client first obtains the metadata of the client object data from the distributed KV storage database; 2) After obtaining the metadata, the distribution of the client object data to be read on the disk can be known, and the starting position (offset) and byte length (len) of the client object data to be read can be converted into the offset and len of the relevant data in each disk storage area, and the read request is sent to the corresponding disk; 3) After receiving the read request, the disk reads the valid data at the specified location and returns the valid data at the specified location to the client.

[0036] Among them, the storage area may include one or more sectors, each sector may be 512 bytes, 4096 bytes (4K sector), or any number of bytes, and the specific size of the storage area can be set according to the actual needs of the distributed storage system; a disk can be divided into multiple storage areas according to the disk capacity, and the size of each storage area can be equal or different. The embodiment of the present application does not impose any restrictions on the partition size of the storage area, the number of partitions in the disk, and the specific size of each partition in the disk.

[0037] In a distributed storage system based on the append-only mode, when a client overwrites an object, the persistence layer (the collection of all disk management processes) does not directly overwrite the original data, but instead writes the data to the end in an append-only manner. After the append-only write is completed, the metadata is modified to point to the data just appended to the end. At this time, the original overwritten data will be marked as garbage data, and garbage collection (GC) operations need to be introduced.

[0038] In the related art, see FIG. 2A to FIG. 2D , the process of recycling garbage data in the disk storage area is as follows: 1) read valid data from the storage area; 2) create a new storage area on the disk and write valid data to the new storage area; 3) update the metadata in the distributed KV storage database and point all metadata to the valid data in the new storage area; 4) delete the old storage area on the disk and recycle the disk space. Combining the above process, it can be seen that when the relevant technology recycles garbage data, it first reads out the valid data and then writes the valid data. This process not only wastes unnecessary network bandwidth, but also affects the efficiency of garbage data recycling.

[0039] In view of the above technical problems, the following embodiments are proposed.

[0040] In some embodiments of the present application, the garbage data recovery method can be implemented using a processor in a garbage data recovery device, and the above-mentioned processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0041] Figure 3 Flow chart of a garbage data recovery method according to an embodiment of the present application. Figure 3 As shown, the method includes:

[0042] Step 100: In response to a garbage collection request for an object to be processed, determine a storage area of ​​the object to be processed on each disk.

[0043] In the embodiment of the present application, the garbage data recovery method can be applied to Figure 1 In the distributed storage system based on the append-only mode shown, the executor of the garbage data collection method is an electronic device that can manage disk processes. Here, there is no limitation on the type of electronic device. For example, it can be a client in the distributed storage system or other types of devices. The client will be used as an example for explanation below.

[0044] Exemplarily, the Ceph distributed storage system is taken as an example for explanation; the Ceph storage system may include a client, a cluster monitor (Cluster Monitor, MON), a disk management unit (Object Storage Device, OSD) and a disk set; multiple disks in the disk set may correspond one to one with multiple OSDs.

[0045] Among them, the working principle of Ceph storage system to realize data migration is: OSD monitors the disk's storage capacity utilization, whether it is running, and other disk operation status; multiple OSDs send the disk operation status to the set MON for summary management, and MON sends the summary result to the client, which can specifically summarize the storage capacity utilization of all disks to obtain the storage capacity utilization of all disks, the storage capacity utilization of the host, and the storage capacity utilization of the cluster; the client determines the location of the valid data in the destination storage area based on the above three utilizations, and sends instructions to the disk to perform the migration operation; the disk executes the migration operation and returns the migration result to the client.

[0046] Exemplarily, the object to be processed refers to an object that requires garbage collection; it should be noted that the object to be processed is pre-stored in storage areas of different disks in units of coding blocks; it can be understood that when the object to be processed is stored, it will be encoded based on the erasure code. Since the object to be processed is generally large and cannot be encoded in one go, the object to be processed will be encoded in multiple times, and each encoding will generate several coding blocks of the same size, which are stored in storage areas of different disks respectively; wherein each coding block may include a data block and a check block; it should be noted that the multiple coding blocks obtained by encoding the object to be processed are ordered, so that the storage area storing the various coding blocks is also ordered, otherwise accurate decoding cannot be achieved subsequently.

[0047] Among them, the garbage data recovery request can be understood as a request instructing the client to perform garbage data recovery for the object to be processed. It should be noted that the source of the garbage data recovery request can be manually initiated or automatically uploaded by the distributed storage system, which can be determined based on the actual usage scenario, and the embodiments of the present application do not limit this.

[0048] Exemplarily, a disk collection in a distributed storage system is also called a disk layer or a persistence layer, and the disk collection is a collection of all disks in the distributed storage system; the garbage collection request includes relevant information about the storage areas of each disk corresponding to the object to be processed; in one embodiment, when the client receives a garbage collection request for a processing object, the storage areas of each disk of the object to be processed can be determined in the disk collection; wherein each storage area stores part of the data of the object to be processed; here, the disks corresponding to the object to be processed may be part of the disks or all of the disks in the disk collection.

[0049] Step 101: Determine a target storage area including junk data from the storage areas of each disk, and obtain location information of the junk data in the target storage area.

[0050] Exemplarily, according to the above content, each disk's storage area stores various coded blocks obtained by pre-encoding the object to be processed, and each storage area stores partial data of the object to be processed; it should be noted that the partial data may include at least one of valid data and garbage data; here, garbage data refers to data in the storage area that needs to be recycled; wherein, the data that needs to be recycled refers to the original data that has been overwritten, and the valid data is the data in the storage area other than the garbage data.

[0051] In an embodiment of the present application, after determining the storage areas of the objects to be processed in each disk according to the above steps, a target storage area including garbage data can be determined from the storage areas of each disk; here, when the number of storage areas is multiple, the number of target storage areas can be one or more.

[0052] Furthermore, after determining the target storage area that includes junk data, the location information of the junk data in the target storage area can be obtained; here, the location information may include the starting position and ending position of the junk data, and may also include the starting position and byte length of the junk data; if it is determined that the storage area of ​​each disk does not include junk data, the current process is terminated.

[0053] It should be noted that, when there are multiple target storage areas, it means that the junk data is distributed in multiple target storage areas. At this time, the location information of the junk data in each target storage area can be obtained.

[0054] Step 102: Determine whether the location information meets a preset condition. If yes, perform a truncation operation on the garbage data in the target storage area.

[0055] In an embodiment of the present application, after obtaining the location information of the junk data in the target storage area according to the above steps, it is determined whether the location information meets the preset conditions; here, the preset condition may be that all the junk data is located at the head or tail of the target storage area; that is, the preset condition may include the following two situations: all the junk data is located at the head of the target storage area or all the junk data is located at the tail of the target storage area.

[0056] Exemplarily, the storage areas of the objects to be processed are ordered in each disk. If the storage area of ​​the disk at the front is one of the target storage areas and the target storage area only includes garbage data, it can be said that the garbage data is all located at the head of the target storage area; correspondingly, if the storage area of ​​the disk at the end is one of the target storage areas and the target storage area only includes garbage data, it can be said that the garbage data is all located at the tail of the target storage area.

[0057] Exemplarily, if the client determines that the location information of the garbage data satisfies any of the above conditions, a truncation instruction including the location information of the garbage data can be generated, and the truncation instruction can be sent to the disk where the target storage area is located. When the disk receives the truncation instruction, the corresponding garbage data can be identified according to the location information of the garbage data in the truncation instruction, and the garbage data can be truncated. It should be noted that after the truncation operation, the garbage data in the target storage area will be recycled, and at this time, the original storage space occupied by the garbage data in the target storage area will be released.

[0058] For ease of understanding, the following Figure 4A and Figure 4B The process of truncating junk data is illustrated by way of example.

[0059] For example, see Figure 4A , the storage areas of the objects to be processed in each disk include the storage area of ​​disk 1, the storage area of ​​disk 2 and the storage area of ​​disk 3; the storage areas of these three disks are in order, among which the storage area of ​​disk 1 is in the front and the storage area of ​​disk 3 is in the end; the storage area of ​​disk 1 includes only valid data, the storage area of ​​disk 2 includes valid data and garbage data, and the storage area of ​​disk 3 includes only garbage data; that is, the target storage area includes the storage areas of disk 2 and disk 3; it can be seen that the garbage data in the storage area of ​​disk 2 is all located at the end of the storage area, and the garbage data in the storage area of ​​disk 3 is also all located at the end of the storage area. At this time, the location information of the garbage data in the target storage area meets the preset conditions, so the garbage data can be truncated; after the truncation operation, the garbage data in the storage areas of disk 2 and disk 3 will be recycled, and at this time, the original storage space occupied by the garbage data will be released.

[0060] For example, see Figure 4B , the storage areas of the objects to be processed in each disk include the storage area of ​​disk 1, the storage area of ​​disk 2 and the storage area of ​​disk 3; the storage areas of these three disks are in order, among which the storage area of ​​disk 1 is in the front and the storage area of ​​disk 3 is in the back; the storage area of ​​disk 1 includes only garbage data, the storage area of ​​disk 2 includes valid data and garbage data, and the storage area of ​​disk 3 includes only valid data; that is, the target storage area includes the storage areas of disk 1 and disk 2; it can be seen that the garbage data in the storage area of ​​disk 1 are all located at the head of the storage area, and the garbage data in the storage area of ​​disk 2 are also all located at the head of the storage area. At this time, the location information of the garbage data in the target storage area meets the preset conditions, so the garbage data can be truncated; after the truncation operation, the garbage data in the storage areas of disk 1 and disk 2 will be recycled, and at this time, the original storage space occupied by the garbage data will be released.

[0061] It can be seen that in the embodiment of the present application, in the process of recycling garbage data, once it is determined that the garbage data in the disk is located at the head or tail of the target storage block, the garbage data is directly truncated to release storage space; the entire process does not require first reading the valid data in the disk storage block and then writing the valid data, so that the amount of data read can be effectively reduced, thereby accelerating the recovery of garbage data, while saving network bandwidth and improving the efficiency of garbage data recovery.

[0062] In some embodiments, the above method may further include: when it is determined that the location information does not meet a preset condition, migrating valid data in the target storage area to the destination storage area based on a cluster balance state.

[0063] The destination storage area is the storage area that is topologically closest to the target storage area. The storage area may be located in the first disk where the target storage area is located, or in the storage area of ​​another disk that belongs to the same host as the first disk.

[0064] Here, the cluster balance state can characterize the load balance state of each disk in the distributed storage system; it can be seen that in the embodiment of the present application, the optimal garbage collection strategy can be selected according to the location information of the garbage data. Specifically, when the garbage data is located at the head or tail of the target storage area, the client directly initiates the truncation operation, which is completed locally on the disk. There is no need to adjust the location of valid data, and the read and write operations and relocation operations are omitted, thus saving network bandwidth and achieving high recovery efficiency. When the garbage data is not located at the head or tail of the target storage area, the storage area with the shortest topological distance and balanced storage capacity is selected for relocation, which saves network bandwidth while optimizing the balance state of the cluster, thereby optimizing the storage performance of the cluster.

[0065] The following is an exemplary description of the migration process of valid data.

[0066] In some embodiments, migrating valid data in the target storage area to the destination storage area based on the cluster balance state may include: determining whether the storage capacity utilization of the first disk meets a first preset utilization, and if so, obtaining valid data in the target storage area and migrating the valid data to the destination storage area; wherein, when the storage capacity utilization of the first disk meets the first preset utilization, determining the destination storage area as the storage area in the first disk that is topologically closest to the target storage area; and the first disk is the disk where the target storage area is located.

[0067] Exemplarily, if the client determines that the location information of the garbage data in the target storage area does not meet the preset conditions, it means that the garbage data is not entirely located at the head or tail of the target storage area, that is, the garbage data may be distributed in the target storage area in a dispersed manner, or may be entirely located in the middle of the target storage area; at this time, the valid data in the target storage area can be migrated to achieve the recovery of the garbage data; accordingly, it can be determined first whether the storage capacity utilization of the first disk meets the first preset utilization.

[0068] It should be noted that the number of the first disks may be one or more. When the number of the first disks is more than one, the processing process for each first disk is similar. For ease of description, one first disk will be used as an example for description.

[0069] Exemplarily, the storage capacity utilization may be the ratio of the used storage capacity to the total storage capacity of the first disk; it can be understood that if the storage capacity utilization of the first disk is greater, the load on the first disk is greater; conversely, if the storage capacity utilization of the first disk is smaller, the load on the disk is smaller.

[0070] In some embodiments, determining whether the storage capacity utilization of the first disk meets a first preset utilization may include: comparing the storage capacity utilization of the first disk with the first preset utilization to obtain a comparison result; when the comparison result indicates that the storage capacity utilization is greater than the first preset utilization, determining that the storage capacity utilization of the first disk does not meet the first preset utilization; when the comparison result indicates that the storage capacity utilization is less than or equal to the first preset utilization, determining that the storage capacity utilization of the first disk meets the first preset utilization.

[0071] In an embodiment of the present application, a first preset utilization rate can be set according to actual conditions, and the storage capacity utilization rate of the first disk can be compared with the first preset utilization rate to obtain a comparison result, and then, based on the comparison result, it is determined whether the storage capacity utilization rate of the first disk meets the first preset utilization rate.

[0072] Here, the comparison result can characterize the load balancing state of the first disk; exemplarily, if the comparison result shows that the storage capacity utilization is greater than the first preset utilization, it means that the load of the first disk is large, that is, the storage capacity utilization of the first disk has not reached balance; if the comparison result shows that the storage capacity utilization is less than or equal to the first preset utilization, it means that the load of the first disk is small, that is, the storage capacity utilization of the first disk can reach balance.

[0073] Furthermore, when it is determined that the storage capacity utilization of the first disk meets the first preset utilization, valid data in the target storage area can be obtained and migrated to the destination storage area; here, the destination storage area is the storage area in the first disk that is topologically closest to the target storage area, which is different from the target storage area; that is, the migration process of valid data occurs inside the first disk.

[0074] Exemplarily, if the client determines that the storage capacity utilization of the first disk satisfies the first preset utilization, it can obtain the valid data in the target storage area, generate a migration instruction including the valid data, and send the migration instruction to the first disk where the target storage area is located. When the first disk receives the migration instruction, it can migrate the valid data in the migration instruction to the target storage area.

[0075] It should be noted that after the valid data in the target storage area is migrated, a data erasure operation will be performed on the target storage area to realize space recovery of the target storage area storing junk data.

[0076] For example, see Figure 5 , assuming that the first disk where one of the target storage areas is located is disk 1, and the storage capacity utilization of disk 1 can be balanced, that is, it meets the first preset utilization, then the valid data of the target storage area of ​​disk 1 can be migrated to the destination storage area, wherein the destination storage area is located inside disk 1; that is, the migration of valid data is performed inside disk 1, and the migration process does not occupy network resources.

[0077] It can be seen that in the embodiment of the present application, for a target storage area whose garbage data location information does not meet the preset conditions, if it is determined that the storage capacity utilization of the disk where the target storage area is located can be balanced, the valid data of the target storage area can be migrated preferentially within the same disk. Since the migration process does not occupy network resources, network resources can be effectively saved.

[0078] In some embodiments, migrating valid data in the target storage area to the destination storage area based on the cluster balance state may also include: determining whether the storage capacity utilization of the first disk meets the first preset utilization; if it is determined not, determining each candidate disk that belongs to the same host as the first disk; screening each candidate disk to obtain a second disk; obtaining valid data in the target storage area, and migrating the valid data to the destination storage area; wherein, when the storage capacity utilization of the first disk does not meet the first preset utilization, determining the destination storage area to be the storage area in the second disk that is topologically closest to the first disk.

[0079] In an embodiment of the present application, if it is determined that the storage capacity utilization of the first disk does not meet the first preset utilization, it means that the storage capacity utilization of the first disk has not reached balance. At this time, each candidate disk belonging to the same host as the first disk can be determined from the disk set; and each candidate disk is screened to obtain the second disk that is closest to the first disk in topological distance.

[0080] Here, each candidate disk and the first disk belong to the same host, that is, each candidate disk and the first disk are different disks under the same host.

[0081] In some embodiments, after determining the candidate disks that belong to the same host as the first disk, the method may further include: obtaining the storage capacity utilization of each candidate disk; sorting the storage capacity utilization of each candidate disk in order from low to high to obtain a sorting result; and determining a preset number of candidate disks ranked at the front of the sorting result as a list of disks that meet a second preset utilization.

[0082] Furthermore, screening each candidate disk to obtain the second disk may include: screening each candidate disk in the disk list to obtain the second disk.

[0083] In the embodiment of the present application, after obtaining the storage capacity utilization of each candidate disk, the storage capacity utilization of each candidate disk can be sorted in order from low to high to obtain a sorting result; and a preset number of candidate disks ranked in the front of the sorting result are determined as a list of disks that meet the second preset utilization. In the embodiment of the present application, the value of the preset number is not specifically limited, for example, the value of the preset number can be 1 or 2, etc.

[0084] Here, the second disk can be a candidate disk in the disk list that is topologically closest to the first disk; exemplarily, after the client filters out the second disk from the disk list, it can obtain valid data in the target storage area, generate a migration instruction including the valid data, and send the migration instruction to the second disk. When the second disk receives the migration instruction, it can migrate the valid data in the migration instruction to its own storage area.

[0085] It should be noted that after the valid data in the target storage area is migrated, a data erasure operation will be performed on the target storage area to realize space recovery of the target storage area storing junk data.

[0086] For example, see Figure 5, assuming that the first disk where one of the target storage areas is located is disk 2, and the storage capacity utilization of disk 2 is not balanced, that is, it does not meet the first preset utilization. If the second disk that meets the second preset utilization is disk 4 after screening, then the valid data of the target storage area of ​​disk 1 can be migrated to the storage area of ​​disk 4. Here, disk 2 and disk 4 belong to the host 2, that is, valid data can be migrated within host 2; the same is true for disk 3 and disk 5, which will not be repeated here.

[0087] It can be understood that, compared with migrating valid data between different hosts, the embodiment of the present application can save certain network resources by migrating valid data within the same host.

[0088] In some embodiments, after migrating the valid data in the target storage area, the above method may further include: obtaining location information of the migrated valid data; and updating metadata of the valid data in the distributed key-value storage database according to the location information of the migrated valid data.

[0089] Among them, the distributed key-value storage database stores metadata of valid data in the target storage area, and the metadata includes the original location information of the valid data before migration; here, there is no limitation on the content included in the original location information, for example, the original location information may include the starting position and the ending position, or may include the starting position and the byte length.

[0090] It is understandable that after the valid data in the target storage area is migrated, the location information of the valid data will change. To ensure that the valid data after migration can be accurately obtained later, the client can use the location information of the valid data after obtaining the location information of the migrated valid data to update the metadata in the distributed key-value storage database. The purpose of the update is to replace the original location information in the metadata with the location information of the valid data after migration, so that the replaced location information can accurately point to the valid data after migration, thereby ensuring the accuracy of subsequent operations.

[0091] Figure 6 Schematic diagram of the structure of a garbage data recovery device according to an embodiment of the present application. Figure 6 As shown, the device includes: a first determination module 300, a second determination module 301 and a truncation module 302, wherein:

[0092] The first determination module 300 is used to determine the storage areas of the object to be processed on each disk in response to a garbage data collection request for the object to be processed; each storage area stores part of the data of the object to be processed;

[0093] The second determining module 301 is used to determine a target storage area including junk data from the storage areas of the disks, and obtain location information of the junk data in the target storage area;

[0094] The truncation module 302 is used to determine whether the location information meets a preset condition, and if so, to perform a truncation operation on the junk data in the target storage area; the preset condition is that all the junk data is located at the head or tail of the target storage area.

[0095] In some embodiments, the apparatus further comprises a migration module, wherein the migration module is configured to:

[0096] When it is determined that the location information does not satisfy the preset condition, the valid data in the target storage area is migrated to a destination storage area based on a cluster balance state; the destination storage area is a storage area that is topologically closest to the target storage area.

[0097] In some embodiments, the migration module is used to:

[0098] Determine whether the storage capacity utilization of the first disk meets the first preset utilization. If it is determined to be yes, obtain the valid data in the target storage area and migrate the valid data to the destination storage area; wherein, when the storage capacity utilization of the first disk meets the first preset utilization, determine that the destination storage area is the storage area in the first disk that is topologically closest to the target storage area; the first disk is the disk where the target storage area is located.

[0099] In some embodiments, the migration module is further configured to:

[0100] Determining whether the storage capacity utilization of the first disk meets a first preset utilization;

[0101] If the determination is no, determine each candidate disk that belongs to the same host as the first disk;

[0102] Screening the candidate disks to obtain the second disk;

[0103] Acquire valid data in the target storage area and migrate the valid data to the destination storage area; wherein, when the storage capacity utilization of the first disk does not meet the first preset utilization, determine that the destination storage area is a storage area in the second disk that is topologically closest to the first disk; the first disk is the disk where the target storage area is located.

[0104] In some embodiments, the migration module is further configured to:

[0105] Comparing the storage capacity utilization of the first disk with a first preset utilization to obtain a comparison result;

[0106] If the comparison result indicates that the storage capacity utilization is greater than the first preset utilization, determining that the storage capacity utilization of the first disk does not meet the first preset condition;

[0107] When the comparison result indicates that the storage capacity utilization is less than or equal to the first preset utilization, it is determined that the storage capacity utilization of the first disk meets the first preset condition.

[0108] In some embodiments, after determining each candidate disk that belongs to the same host as the first disk, the migration module is further configured to:

[0109] Obtaining storage capacity utilization of each candidate disk;

[0110] Sorting the storage capacity utilization rates of the candidate disks in descending order to obtain a sorting result;

[0111] Determine a preset number of candidate disks that are ranked in front of the sorting result as a disk list that meets a second preset utilization rate;

[0112] The candidate disks in the disk list are screened to obtain the second disk.

[0113] In some embodiments, the distributed key-value storage database stores metadata of the valid data. After migrating the valid data in the target storage area, the device further includes an update module, the update module being used to:

[0114] Obtain the location information of the valid data after migration;

[0115] According to the location information of the migrated valid data, metadata of the valid data in the distributed key-value storage database is updated.

[0116] In some embodiments, the update module is further used to:

[0117] A data erasing operation is performed on the target storage area.

[0118] In actual applications, the above-mentioned first determination module 300, second determination module 301, truncation module 302, migration module and update module can all be implemented by a processor located in an electronic device, and the processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0119] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0120] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the relevant technology or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0121] Specifically, a computer program instruction corresponding to a garbage data recovery method in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the computer program instruction corresponding to a garbage data recovery method in the storage medium is read or executed by an electronic device, any one of the garbage data recovery methods in the aforementioned embodiments is implemented.

[0122] Based on the same technical concept as the above embodiments, see Figure 7 , which shows an electronic device 400 provided in an embodiment of the present application, which may include: a memory 401, a processor 402 and a communication bus 403; wherein,

[0123] A communication bus 403, used to realize a communication connection between the processor 402 and the memory 401;

[0124] Memory 401, used for storing computer programs and data;

[0125] The processor 402 is used to execute the computer program stored in the memory to implement any one of the garbage data recovery methods in the aforementioned embodiments.

[0126] In practical applications, the memory 401 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 402.

[0127] The processor 402 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understandable that for different augmented reality cloud platforms, the electronic device used to implement the processor function may also be other, which is not specifically limited in the embodiments of the present application.

[0128] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation thereof can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here. The above description of each embodiment tends to emphasize the differences between the embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, it will not be repeated here.

[0129] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments. The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0131] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] The above are only preferred embodiments of the present application and are not intended to limit the protection scope of the present application.

Claims

1. A garbage data recovery method, characterized in that: The method comprises: In response to a garbage data collection request for an object to be processed, determining storage areas of the object to be processed on each disk; each storage area stores part of the data of the object to be processed; Determine a target storage area including junk data from the storage areas of each disk, and obtain location information of the junk data in the target storage area; Determine whether the location information satisfies a preset condition, and if so, perform a truncation operation on the junk data in the target storage area; the preset condition is that all the junk data is located at the head or tail of the target storage area.

2. The method according to claim 1, characterized in that: The method further comprises: When it is determined that the location information does not satisfy the preset condition, the valid data in the target storage area is migrated to a destination storage area based on a cluster balance state; the destination storage area is a storage area that is topologically closest to the target storage area.

3. The method according to claim 2, characterized in that The migrating valid data in the target storage area to the destination storage area based on the cluster balance state includes: Determining whether the storage capacity utilization of the first disk meets a first preset utilization; If the answer is yes, obtain the valid data in the target storage area and migrate the valid data to the destination storage area; wherein, when the storage capacity utilization of the first disk meets the first preset utilization, determine that the destination storage area is the storage area in the first disk that is topologically closest to the target storage area; the first disk is the disk where the target storage area is located.

4. The method according to claim 2, characterized in that: The migrating valid data in the target storage area to the destination storage area based on the cluster balance state includes: Determining whether the storage capacity utilization of the first disk meets a first preset utilization; If the determination is no, determine each candidate disk that belongs to the same host as the first disk; Screening the candidate disks to obtain the second disk; Acquire valid data in the target storage area and migrate the valid data to the destination storage area; wherein, when the storage capacity utilization of the first disk does not meet the first preset utilization, determine that the destination storage area is a storage area in the second disk that is topologically closest to the first disk; the first disk is the disk where the target storage area is located.

5. The method according to claim 3 or 4, characterized in that: The determining whether the storage capacity utilization of the first disk meets a first preset utilization includes: Comparing the storage capacity utilization of the first disk with the first preset utilization to obtain a comparison result; If the comparison result indicates that the storage capacity utilization is greater than the first preset utilization, determining that the storage capacity utilization of the first disk does not meet the first preset utilization; When the comparison result indicates that the storage capacity utilization is less than or equal to the first preset utilization, it is determined that the storage capacity utilization of the first disk satisfies the first preset utilization.

6. The method according to claim 4, characterized in that After determining each candidate disk that belongs to the same host as the first disk, the method further includes: Obtaining storage capacity utilization of each candidate disk; Sorting the storage capacity utilization rates of the candidate disks in descending order to obtain a sorting result; Determine a preset number of candidate disks that are ranked in front of the sorting result as a disk list that meets a second preset utilization rate; Screening the candidate disks to obtain the second disk includes: The candidate disks in the disk list are screened to obtain the second disk.

7. The method according to any one of claims 2 to 4, characterized in that: The distributed key-value storage database stores metadata of the valid data. After migrating the valid data in the target storage area, the method further includes: Obtain the location information of the valid data after migration; According to the location information of the migrated valid data, metadata of the valid data in the distributed key-value storage database is updated.

8. The method according to any one of claims 2 to 4, characterized in that: After migrating the valid data in the target storage area, the method further includes: A data erasing operation is performed on the target storage area.

9. A garbage data recovery device, characterized in that: The device comprises: A first determination module is used to determine the storage areas of the object to be processed on each disk in response to a garbage data collection request for the object to be processed; each storage area stores part of the data of the object to be processed; A second determining module is used to determine a target storage area including junk data from the storage areas of each disk, and obtain location information of the junk data in the target storage area; The truncation module is used to determine whether the location information meets a preset condition, and if so, to perform a truncation operation on the junk data in the target storage area; the preset condition is that all the junk data is located at the head or tail of the target storage area.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images