Method and device for deleting invalid data

Abstract

The invention discloses an invalid data deletion method and device, and belongs to the technical field of data storage. And deleting invalid data. The method comprises the steps of obtaining an invalid data proportion of each data fragment in a data disk in a storage node, wherein the invalid data proportion is a ratio of the total amount of invalid data in the data fragments to the total data capacity of the data fragments; based on the invalid data proportions, multiple target data fragments needing to execute invalid data deletion operation are determined in multiple data fragments in the data disk, and the invalid data proportions of the multiple target data fragments are all larger than the invalid data proportions of other data fragments; the other data fragments are any data fragment except the multiple target data fragments in the data disk; and executing deletion operation on the plurality of data blocks included in the plurality of target data fragments. According to the method and the device, the real-time balance of the invalid data deletion operation executed by the plurality of data disks is ensured, and the concurrency of the invalid data deletion operation executed by the plurality of data disks is favorably balanced.

Description

[0001] This application claims priority to Chinese patent application No. 202311171306.9 filed on September 12, 2023, with invention name “A data recovery method, device and computing device cluster”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the technical field of data storage, and in particular to a method and device for deleting invalid data. Background Art

[0003] In the field of data storage, in order to improve the utilization of storage space, invalid data deletion operations are usually performed on storage units in storage nodes. Performing invalid data deletion operations on storage units means first migrating valid data recorded in the storage unit to other storage units, and then deleting the storage unit after completing the migration of valid data in the storage unit.

[0004] At present, a storage node includes multiple data disks, each of which includes multiple data shards, and each of which includes multiple data blocks. When deleting invalid data, the valid data recorded in the data block is usually migrated, and after the migration of the valid data in the data block is completed, the data block is deleted. Among them, the data block that needs to perform the invalid data deletion operation is the data block with a high idle ratio (also called invalid data ratio). The idle ratio of a data block is the ratio of the total amount of invalid data recorded in the data block to the total data capacity of the data block.

[0005] However, this invalid data deletion method will cause large differences in the concurrency of invalid data deletion operations performed by different data disks, affecting the read and write performance of the storage node. Summary of the invention

[0006] The present application provides a method and device for deleting invalid data. The present application ensures the real-time balance of multiple data disks in performing invalid data deletion operations, helps to balance the concurrency of multiple data disks in performing invalid data deletion operations, avoids performance problems caused by hot disks, and ensures the read and write performance of storage nodes. The technical solutions provided by the present application are as follows:

[0007] In a first aspect, the present application provides a method for deleting invalid data. The method is applied to a storage node. The storage node includes multiple data disks. The data disk includes multiple data shards. The data shards include multiple data blocks. The method includes: obtaining the invalid data ratio of each data shard in the data disk, the invalid data ratio being the ratio of the total amount of invalid data in the data shard to the total data capacity of the data shard; based on the invalid data ratio, determining multiple target data shards on which invalid data deletion operations need to be performed in the multiple data shards in the data disk, the invalid data ratios of the multiple target data shards are all greater than the invalid data ratios of other data shards, and the other data shards are any data shards in the data disk except the multiple target data shards; and performing a deletion operation on the multiple data blocks included in the multiple target data shards.

[0008] In the invalid data deletion method, when selecting a data shard on which an invalid data deletion operation needs to be performed from multiple data shards of a data disk, the storage unit granularity targeted is the data shard. The storage unit granularity targeted when performing an invalid data deletion operation is the data shard. It can be seen that the storage unit granularity targeted when determining the storage unit on which an invalid data deletion operation needs to be performed is the same as the storage unit granularity targeted when performing the invalid data deletion operation. In this way, the storage unit on which the invalid data deletion operation needs to be performed can be selected from a global perspective, and the invalid data deletion tasks of multiple data disks in the storage node can be more evenly scheduled, ensuring the real-time balance of multiple data disks performing invalid data deletion operations, which helps to balance the concurrency of multiple data disks performing invalid data deletion operations, avoid performance problems caused by hot disks, and ensure the read and write performance of the storage node.

[0009] In a possible implementation manner of the first aspect, invalid data proportions of all data blocks in a first data slice are obtained, and then based on the invalid data proportions of all data blocks, the invalid data proportion of the first data slice is determined.

[0010] Since each data shard includes multiple data blocks, and a data block is the most basic storage unit, based on the above method, the invalid data ratio of the first data shard can be determined more accurately, and the invalid data can be deleted more accurately.

[0011] In another possible implementation of the first aspect, representative data blocks are selected from all data blocks in the first data shard, and then the invalid data proportion of the first data shard is determined based on the invalid data proportion of these representative data blocks. The representative data blocks may be selected as data blocks whose invalid data proportion is greater than a specified proportion threshold. Alternatively, the representative data blocks are data blocks whose invalid data proportion is a specified number in the first data shard. The specified number of invalid data proportions are all greater than the invalid data proportions of other data blocks, and the other data blocks are any data blocks in the first data shard except the specified number of data blocks.

[0012] Since the data storage situation of the representative data block can better represent the storage situation of the data in the data shard, based on the above method, the invalid data proportion of the first data shard can be determined more accurately, and the invalid data can be deleted more accurately.

[0013] In the present application, a deletion operation is performed on multiple data blocks included in multiple target data shards, including: based on the access popularity of data recorded in a first data block in the target data shard, valid data in the data recorded in the first data block is migrated, and the first data block is any one of the multiple data blocks; all valid data in the data recorded in the first data block is migrated to obtain a second data block, and the data recorded in the second data block are all invalid data, and the second data block is deleted.

[0014] In a possible implementation of the first aspect, a data block includes multiple data pages. Based on the access heat of data recorded in a first data block in a target data shard, valid data in the data recorded in the first data block is migrated, including: obtaining the access heat of data recorded in multiple data pages in the first data block, where the access heat includes multiple heat levels; migrating data recorded in multiple data pages whose access heat of data recorded in multiple data pages is at the same heat level to the same data block.

[0015] By migrating data with different access popularity to different data blocks, the cold data and hot data are separated. In this way, during the business operation, the recycling frequency of the data blocks where the cold data is located will be reduced, which can reduce the ineffective migration of cold data and save computing resources and IO resources.

[0016] In a possible implementation of the first aspect, access popularity includes data access frequency and / or access timeliness, where the access frequency reflects the number of times the data is accessed within a specified time period, and the access timeliness reflects how far the data is accessed from the current time.

[0017] The access popularity of data is positively correlated with the access frequency. The higher the access frequency of data, the higher the access popularity of the data. The closer the access time of data is to the current time, the greater the access time of the data. Conversely, the smaller the access time is. The access popularity of data is positively correlated with the access time. The greater the access time of data, the higher the access popularity of the data.

[0018] In a possible implementation manner of the first aspect, the method further includes: when a business load of a data disk to which the target data shard belongs is greater than a first threshold, stopping the invalid data deletion operation on the target data shard.

[0019] When the proportion of resources occupied by the data disk is greater than the first proportion threshold, it means that the read and write business of the data disk is relatively busy. If the invalid data deletion operation is stopped on the target data shard in the data disk, the computing resources and IO resources occupied by the invalid data deletion operation can be allocated to the read and write operations to ensure the performance of the read and write operations.

[0020] In a possible implementation of the first aspect, the method further includes: when the business load of the data disk to which the target data shard belongs is less than or equal to a second threshold, continuing to perform the invalid data deletion operation on the target data shard, and the second threshold is less than or equal to the first threshold.

[0021] When the business load of the data disk is less than or equal to the second threshold, it means that the read and write business of the data disk is no longer busy, and invalid data deletion operations can be performed while performing read and write operations. Then, invalid data deletion operations can continue to be performed on the target data shards in the data disk.

[0022] In a possible implementation of the first aspect, the method also includes: obtaining a first priority level of the computing resources of the storage node used for the invalid data deletion operation, the first priority level being lower than a second priority level of the computing resources used by the storage node for executing read and write operations, the read and write operations being used to respond to the read and write operations of the user; based on the first priority level, allocating the computing resources of the storage node used for executing the invalid data deletion operation.

[0023] The priority of the invalid data deletion operation in using computing resources is lower than the priority of the read-write operation in using computing resources, which can be manifested as when the invalid data deletion operation and the read-write operation need to use computing resources at the same time, the computing resources are preferentially allocated to the read-write operation. Also, after the computing resources are allocated to the read-write operation, if there are no remaining computing resources, the invalid data deletion operation is queued for allocation, and after the waiting time of the invalid data deletion operation reaches a specified time, when there are idle computing resources, the computing resources are allocated to the invalid data deletion resource. In another implementation, the priority of the invalid data deletion operation in using computing resources is lower than the priority of the read-write operation in using computing resources, which can be manifested as allocating a larger proportion of computing resources to the read-write operation and a smaller proportion of computing resources to the invalid data deletion operation among the computing resources owned by the storage node. By allocating computing resources to the invalid data deletion operation according to the priority, and the first priority is lower than the second priority of the storage node in using computing resources for executing the read-write operation, it can be ensured that the read-write operation has the priority to use computing resources, and the invalid data deletion operation is prevented from preempting the computing resources of the read-write operation, thereby ensuring the performance of the read-write operation.

[0024] In a possible implementation of the first aspect, before determining, based on the proportion of invalid data, among multiple data shards in the data disk, multiple target data shards on which invalid data deletion operations need to be performed, the method also includes: determining the total number of first tasks on which the storage node can perform invalid data deletion tasks on the data disk; and determining the total number of target data shards based on the total number of first tasks.

[0025] In a possible implementation of the first aspect, before determining the target data shards on which the invalid data deletion operation needs to be performed, the total number of target data shards on which the invalid data deletion operation needs to be performed is first determined, so as to determine the total number of data shards among the multiple data shards of the data disk.

[0026] In a possible implementation of the first aspect, the total number of target data shards is determined based on the total number of first tasks that the storage node can perform on the data disk to delete invalid data. The process of determining the total number of target data shards includes: determining the total number of first tasks, and then determining the total number of target data shards based on the total number of first tasks. When the storage node includes multiple data disks, if the storage node allocates the same capabilities for performing invalid data deletion operations to multiple data disks, the total number of target data shards of the multiple data disks obtained according to this method is substantially equal to the sum of the total number of invalid data deletion tasks that have not been completed for the corresponding data disks. In this way, the concurrent number of invalid data deletion operations performed by multiple data disks can be substantially equal, which can ensure the real-time balance of invalid data deletion operation tasks performed by multiple data disks, thereby ensuring the read and write performance of the storage node.

[0027] In a possible implementation of the first aspect, the first total number of tasks is obtained based on a first upper limit value of the total number of invalid data deletion tasks that the storage node can perform on the data disk and a first total number of invalid data deletion tasks that the storage node has not completed on the data disk.

[0028] In another possible implementation of the first aspect, the total number of target data shards is determined based on the total number of second tasks that the storage node can perform to delete invalid data on all data disks in the storage node. The process of determining the total number of target data shards includes: obtaining the total number of second tasks, and then determining the total number of target data shards based on the total number of second tasks.

[0029] In a possible implementation of the first aspect, the second total number of tasks is obtained based on a second upper limit value of the total number of invalid data deletion tasks that the storage node can perform on multiple data disks and a second total number of invalid data deletion tasks that the storage node has not completed on multiple data disks.

[0030] In a possible implementation of the first aspect, a deletion operation is performed on multiple data blocks included in multiple target data shards, including: distributing invalid data deletion tasks to the multiple target data shards by executing a distribution process until a stop condition is reached, the stop condition including: the total number of distributed tasks reaches a specified total number, the distribution process including: updating the third task total number of invalid data deletion tasks that are currently unfinished for each target data shard in the multiple target data shards, distributing invalid data deletion tasks to the target data shard with the smallest third task total number among the multiple target data shards; and performing invalid data deletion operations on the target data shards to which the invalid data deletion tasks are distributed. In this way, the invalid data deletion tasks can be distributed more evenly, ensuring the real-time balance of invalid data deletion operations performed by multiple data disks.

[0031] In a second aspect, the present application provides an invalid data deletion device. The device is applied to a storage node. The storage node includes multiple data disks. The data disk includes multiple data shards. The data shards include multiple data blocks. The device includes: an acquisition module, which is used to obtain the invalid data ratio of each data shard in the data disk, and the invalid data ratio is the ratio of the total amount of invalid data in the data shard to the total data capacity of the data shard; a determination module, which is used to determine, based on the invalid data ratio, multiple target data shards in the data disk that need to perform invalid data deletion operations, the invalid data ratio of the multiple target data shards is greater than the invalid data ratio of other data shards, and the other data shards are any data shards in the data disk except the multiple target data shards; a processing module, which is used to perform a deletion operation on the multiple data blocks included in the multiple target data shards.

[0032] The second aspect or any implementation method of the second aspect is a step implementation of the device corresponding to the first aspect or any implementation method of the first aspect. The description in the second aspect or any implementation method of the second aspect is applicable to the first aspect or any implementation method of the first aspect and will not be repeated here.

[0033] In a third aspect, the present application provides a computing device including a memory and a processor, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method provided in the first aspect of the present application and any possible implementation thereof.

[0034] In a fourth aspect, the present application provides a computing device cluster, including multiple computing devices, the multiple computing devices including multiple processors and multiple memories, the multiple memories storing program instructions, and the multiple processors running the program instructions, so that the computing device cluster executes the method provided in the first aspect of the present application and any possible implementation thereof.

[0035] In a fifth aspect, the present application provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium, and the computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device executes the method provided in the first aspect of the present application and any possible implementation thereof.

[0036] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method provided in the first aspect of the present application and any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of a storage structure of a storage node provided in an embodiment of the present application;

[0038] Figure 2 It is a schematic diagram of a storage node function provided by an embodiment of the present application that can be implemented through processes and threads;

[0039] Figure 3 It is a structural diagram of an implementation scenario involved in an invalid data deletion method provided in an embodiment of the present application;

[0040] Figure 4 is a flow chart of an invalid data deletion method provided by an embodiment of the present application;

[0041] Figure 5 This is a flowchart of performing a deletion operation provided by an embodiment of the present application;

[0042] Figure 6 This is a schematic diagram of migrating cold and hot data separately provided in an embodiment of the present application;

[0043] Figure 7 is a schematic diagram of an invalid data deletion device provided in an embodiment of the present application;

[0044] Figure 8 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0046] In the field of data storage technology, storage nodes are usually used to store data. Figure 1 As shown, the storage node includes multiple data disks, each data disk includes multiple data slices, and each data slice includes multiple data blocks. A data block is the most basic storage unit. A data block has a fixed size. For example, the size of all data blocks in a storage node is 64 megabytes (MB).

[0047] In one implementation, the functions of the storage node can be implemented through processes and threads. Figure 2 As shown in FIG. 1 , in a distributed storage system, each storage node includes a read-write process and an invalid data deletion (also known as data recycling or garbage collection) process. The read-write process is used to respond to the user's read-write operations and manage the data stored in the storage node. The invalid data deletion process is used to perform invalid data deletion operations, etc. At the same time, Figure 2 As shown, each data disk also has its own input output (IO) processing thread. The IO processing thread is used to respond to the IO request of the data disk. Among them, the IO processing thread of any data disk can be regarded as the exclusive resource of the data disk.

[0048] In the present application, the data disk can support appending writing, but does not support overwriting. When the data disk does not support overwriting, the service life of the data disk can be guaranteed. Similarly, the data block supports appending writing and random reading, but does not support overwriting. In this way, for a data block, during the operation of the storage node, when the value of a key is already stored in the data block, if the value of the key is subsequently written to the data block, the value written first will become invalid data, and the value written later will be valid data. Similarly, if the value corresponding to the key is not refreshed, the value is also valid data. Since the data block is the most basic storage unit, the data block can only be retained or deleted as a whole, and deleting part of it is not supported. Correspondingly, the data in the data block can only be retained or deleted as a whole in units of data blocks, and deleting part of the data in the data block is not supported. It can be seen from this that although the invalid data is deleted logically, it still exists physically and still occupies storage space. Only after the data on the entire data block is invalid, the storage space occupied by the invalid data can be released by physically deleting the entire data block. If the data blocks occupied by the invalid data are not physically deleted, these invalid data will cause a waste of storage space. Therefore, in order to reduce the waste of storage space, it is necessary to perform garbage collection on the storage space and delete invalid data. Performing an invalid data deletion operation on a storage node means first migrating the valid data recorded in the storage unit to other storage units, and deleting the storage unit after completing the migration of all valid data in the storage unit. Among them, when the data block is the most basic storage unit, the storage unit here is a data block. It should be noted that the most basic storage unit in the present application can also be a storage unit of other granularities such as a data page or a data partition. At this time, the storage unit targeted by the invalid data deletion operation is the storage unit of the other granularity, and the embodiments of the present application do not specifically limit it. The following takes the storage unit targeted by the invalid data deletion operation as a data block as an example to illustrate the implementation process of the invalid data deletion method of the present application.

[0049] At present, when deleting invalid data, the valid data recorded in the data block is usually migrated, and after completing the migration of the valid data in the data block, the data block is deleted. In addition, the data block that needs to perform the invalid data deletion operation is a data block with a high proportion of invalid data. The invalid data proportion of a data block is the ratio of the total amount of invalid data recorded in the data block to the total data capacity of the data block.

[0050] However, this invalid data deletion method will cause different data disks to perform invalid data deletion operations with different concurrency, affecting the read and write performance of the storage node. For example, when performing invalid data deletion operations on data shards, some data shards may have more data blocks that need to perform invalid data deletion operations, while other data shards may have only a small number of data blocks that need to perform invalid data deletion operations. The concurrency of invalid data deletion operations on these two parts of data shards will be very different. For example, assuming that each data disk performs invalid data deletion operations on 20 data blocks in each round, a data shard of a data disk has 20 data blocks that need to perform invalid data deletion operations, and the 20 data blocks that need to perform invalid data deletion operations on another data disk are located on 20 data shards. In this way, when the data disk generates invalid data deletion tasks in units of data shards, the first data disk needs to generate 1 invalid data deletion task, and the second data disk needs to generate 20 invalid data deletion tasks. The two data disks have an unbalanced scheduling of invalid data deletion tasks, resulting in a large difference in the concurrency of the two data disks executing invalid data deletion operations. The data disk that generates more invalid data deletion tasks will become a hot disk, which will affect the business latency of the storage node.

[0051] Based on this, an embodiment of the present application provides an invalid data deletion method. The method is applied to a storage node. The storage node includes multiple data disks. The data disk includes multiple data shards, and the data shards include multiple data blocks. The method includes: obtaining the invalid data ratio of each data shard in the data disk, and then, based on the invalid data ratio of each data shard in the data disk, determining multiple target data shards that need to perform invalid data deletion operations among the multiple data shards of the data disk, and then performing invalid data deletion operations on multiple data blocks included in the multiple target data shards. Among them, the invalid data ratio reflects the total amount of invalid data in the data shards. For example, the invalid data ratio is the ratio of the total amount of invalid data in the data shards to the total data capacity of the data shards.

[0052] In this method, when a storage node selects a data shard on which an invalid data deletion operation needs to be performed from multiple data shards of a data disk, the storage unit granularity targeted is the data shard. The storage unit granularity targeted when a storage node performs an invalid data deletion operation is the data shard. A data shard includes multiple data blocks, and it can be seen that the storage unit granularity targeted when the storage node determines the storage unit on which an invalid data deletion operation needs to be performed is the same as the storage unit granularity targeted when the invalid data deletion operation is performed. In this way, the storage node can select the storage unit on which an invalid data deletion operation needs to be performed from a global perspective, and can more evenly schedule the invalid data deletion tasks of multiple data disks in the storage node, ensuring the real-time balance of multiple data disks performing invalid data deletion operations, helping to balance the concurrency of multiple data disks performing invalid data deletion operations, avoiding performance problems caused by hot disks, and ensuring the read and write performance of the storage node.

[0053] It should be noted that, although this article uses the storage unit granularity targeted when performing the invalid data deletion operation and the storage unit granularity targeted when performing the invalid data deletion operation as data sharding as an example for explanation, it does not constitute a limitation on the storage unit granularity targeted by the two. A person of ordinary skill in the art will know that as business needs change, the implementation scenario of this application can be adjusted according to application needs, and the storage unit granularity targeted by the two can also change accordingly. For example, in some implementation scenarios, a storage node has storage units of N granularities, and the storage node includes multiple first-level storage units, each first-level storage unit includes multiple second-level storage units, and so on, each N-1-th level storage unit includes multiple N-th level storage units. In order to more evenly schedule the invalid data deletion tasks in multiple first-level storage units in the storage node, the storage unit granularity targeted when performing the invalid data deletion operation and the storage unit granularity targeted when performing the invalid data deletion operation can be selected as any level of storage unit from the second to the Nth level storage units, as long as the storage unit granularity targeted when performing the invalid data deletion operation and the storage unit granularity targeted when performing the invalid data deletion operation remain consistent.

[0054] The following article provides a detailed introduction to the technical solution of the present application from multiple perspectives, including implementation scenarios, method flow, hardware devices, and software devices.

[0055] The following is an example of an application scenario of the embodiment of the present application.

[0056] Figure 3 Schematic diagram of an implementation scenario of an invalid data deletion method provided in an embodiment of the present application. Figure 3As shown, the implementation scenario includes: a computing device 300. The computing device 300 can be used as a storage node. The storage node can store data and respond to user read and write requests.

[0057] The computing device 300 may be a server, and each server may be used as a storage node. In terms of implementation, the computing device 300 may be a storage node in a storage resource pool, or the computing device 300 may be a storage node in other storage systems. For example, the computing device is a storage node in a storage resource pool of a cloud computing service center. In this case, the cloud computing service center can use the storage function of the storage node to provide storage cloud services to users, or provide the storage function as an additional service for other cloud services. For another example, the computing device is a storage node in a distributed storage system used by the storage layer of a cloud database. In this case, the cloud database can use the storage function of the storage node to store the data of the cloud database.

[0058] It should be noted that the above description of the implementation form of the storage node is only an example of the implementation form of the storage node, and is not used to limit the implementation form of the storage node. The storage node in this application can also have other implementation forms. For example, the storage node is a storage node in other storage systems that have multiple data disks and support additional writing. This embodiment of the application does not specifically limit it. In addition, the computing device 300 can also be implemented by other resources besides cloud resources, which is also not specifically limited in this embodiment of the application.

[0059] In one implementation, the invalid data deletion method provided in the embodiment of the present application can be implemented by running an executable program on the computing device 300. For example, the executable program of the invalid data deletion method can be presented in the form of an application installation package, and after the application installation package is installed in the computing device 300, the invalid data deletion method can be implemented by running the executable program.

[0060] Figure 3 Schematic diagram of a computing device provided in an embodiment of the present application. Figure 3 As shown, the computing device 300 includes a processor 301, a memory 302, a communication interface 303 and a bus 304. The processor 301, the memory 302 and the communication interface 303 are connected to each other through the bus 304.

[0061] Processor 301 may include a general processor and / or a dedicated hardware chip. A general processor may include: a central processing unit (CPU), a microprocessor or a graphics processing unit (GPU). The CPU is, for example, a single-core processor (single-CPU), or a multi-core processor (multi-CPU). A dedicated hardware chip is a hardware module for high-performance processing. A dedicated hardware chip includes at least one of a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or a network processor (NP). Processor 301 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, some or all of the functions of the invalid data deletion method of the present application may be completed by an integrated logic circuit of hardware in processor 301 or instructions in software form.

[0062] The memory 302 is used to store computer programs, and the computer programs include an operating system 302a and executable codes (i.e., program instructions) 302b. The memory 302 is, for example, a read-only memory or other types of static storage devices that can store static information and instructions, or a random access memory or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory, a read-only optical disc or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a blue-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired executable code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. For example, the memory 302 is used to store an outbound port queue, etc. The memory 302 is, for example, independent and connected to the processor 301 via a bus 304. Or the memory 302 and the processor 301 are integrated together. The memory 302 can store executable code. When the executable code stored in the memory 302 is executed by the processor 301, the processor 301 is used to perform some or all functions of the invalid data deletion method provided in the embodiment of the present application. For the implementation of the process executed by the processor 301, please refer to the relevant description in the aforementioned embodiment. The memory 302 may also include software modules and data required for other running processes such as an operating system.

[0063] The communication interface 303 uses a transceiver module such as, but not limited to, a transceiver to achieve communication with other devices or communication networks. For example, the communication interface 303 can be any one or any combination of the following devices: a network interface (such as an Ethernet interface), a wireless network card, and other devices with network access functions.

[0064] The bus 304 is any type of communication bus for interconnecting the internal devices of the computing device (e.g., the memory 302, the processor 301, and the communication interface 303). For example, a system bus. The embodiment of the present application takes the interconnection of the above-mentioned devices inside the computing device through the bus 304 as an example. Optionally, the above-mentioned devices inside the computing device 300 can also be connected to each other in communication with each other using other connection methods other than the bus 304. For example, the above-mentioned devices inside the computing device 300 are interconnected through an internal logical interface.

[0065] It should be noted that the above-mentioned multiple devices can be respectively arranged on independent chips, or at least partially or completely arranged on the same chip. Whether to independently arrange each device on different chips or to integrate and arrange it on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices. The descriptions of the processes corresponding to the above-mentioned figures have different focuses. For the parts not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0066] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product providing a program development platform includes one or more computer instructions, and when these computer program instructions are loaded and executed on a computing device, the functions of the invalid data deletion method provided in the embodiment of the present application are implemented in whole or in part.

[0067] Furthermore, computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0068] It should be understood that the above content is an illustrative description of the implementation scenario of the invalid data deletion method provided in the embodiment of the present application, and does not constitute a limitation on the implementation scenario of the invalid data deletion method. A person of ordinary skill in the art can know that as business needs change, the implementation scenario can be adjusted according to application requirements, and the embodiment of the present application does not list them one by one.

[0069] The following takes the invalid data deletion method executed by a storage node as an example to illustrate the invalid data deletion method provided by the embodiment of the present application. Figure 4 As shown, the invalid data deletion method includes the following steps:

[0070] Step 401: The storage node obtains the invalid data ratio of each data shard in the data disk, where the invalid data ratio is the ratio of the total amount of invalid data in the data shard to the total data capacity of the data shard.

[0071] In the present application, the storage node needs to select the target data shard on which the invalid data deletion operation needs to be performed from the multiple data shards of the data disk. The original intention of performing the invalid data deletion operation is to reuse the storage space occupied by invalid data. Therefore, before selecting the target data shard, the storage node needs to first determine the invalid data ratio of each data shard in the data disk, so as to determine the target data shard based on the invalid data ratio. Among them, the invalid data ratio reflects the total amount of invalid data in the data shard. For example, the invalid data ratio is the ratio of the total amount of invalid data in the data shard to the total data capacity of the data shard. When there are more invalid data in the data shard, the invalid data ratio of the data shard is large, which means that the lower the utilization rate of the storage space of the data shard, the more necessary it is to perform the invalid data deletion operation on the data shard.

[0072] According to the previous description, each data shard includes multiple data blocks, and the data block is the most basic storage unit. For the first data shard, the storage node may optionally first obtain the invalid data ratio of the data blocks in the first data shard, and then determine the invalid data ratio of the first data shard based on the invalid data ratio of the data blocks. The first data shard is any one of the multiple data shards in the data disk. The storage node is configured with a counter for each data block, and the counter will count the total amount of invalid data in the data block based on the writing result of the data in the data block. In addition, the storage node can obtain the size of each data block, and the storage node can determine the invalid data ratio of any data block based on the total amount of invalid data in the data block and the size of the data block.

[0073] In one implementation, the storage node obtains the invalid data ratio of all data blocks in the first data shard, and then determines the invalid data ratio of the first data shard based on the invalid data ratio of all data blocks. For example, the storage node determines the invalid data ratio of the first data shard as the average value of the invalid data ratio of all data blocks in the first data shard. The average value of the invalid data ratio of all data blocks is equal to the quotient of the sum of the invalid data ratios of all data blocks and the total number of all data blocks.

[0074] In another implementation, the storage node selects representative data blocks from all data blocks in the first data shard, and then determines the invalid data ratio of the first data shard based on the invalid data ratio of these representative data blocks. For example, the storage node determines the invalid data ratio of the first data shard as the average value of the invalid data ratio of the representative data blocks. The average value of the invalid data ratio of the representative data blocks is equal to the quotient of the sum of the invalid data ratios of all representative data blocks and the total number of all representative data blocks.

[0075] Among them, the representative data block can be selected as a data block whose invalid data ratio is greater than a specified ratio threshold. Alternatively, the representative data block is a data block whose invalid data ratio is a specified number in the first data shard. The specified number of invalid data ratios are all data blocks that are greater than the invalid data ratios of other data blocks, and the other data blocks are any data blocks other than the specified number of data blocks in the first data shard. For example, when determining the representative data block, the storage node can first obtain the invalid data ratio of all data blocks in the first data shard, and then sort the data blocks in order of invalid data ratio from large to small, and determine the data blocks with the first specified number sorted as representative data blocks. The value of the specified number can be optionally set according to application requirements. For example, the specified number is determined based on factors such as the capabilities of the processor used to perform the invalid data deletion operation and the IO capabilities of the data shard.

[0076] On this basis, the representative data block may also be a data block that satisfies the specified recycling condition. The specified recycling condition may be set according to the application requirements. For example, in some businesses, some data blocks record data indicating a special state. In the process of implementing the business, these data blocks are not allowed to be deleted, and the specified recycling condition is that the data block is allowed to be deleted. For another example, if the specified recycling condition is that the invalid data ratio of the data block is greater than the specified ratio threshold, then when determining the representative data block, both the specified number restriction and the specified ratio threshold restriction should be considered. For example, if the total number of data blocks whose invalid data ratio in the first data shard is greater than or equal to the specified ratio threshold is greater than the specified number, then the representative data block is the first specified number of data blocks after the invalid data ratio is sorted from large to small. If the total number of data blocks whose invalid data ratio in the first data shard is greater than the specified ratio threshold is less than the specified number, then the representative data block is all data blocks whose invalid data ratio is greater than the specified ratio threshold.

[0077] Step 402: Determine the total number of target data shards that need to perform invalid data deletion operations.

[0078] Before determining the target data shards on which the invalid data deletion operation needs to be performed, the total number of target data shards on which the invalid data deletion operation needs to be performed can be optionally determined first, so as to determine the total number of data shards in the multiple data shards of the data disk. In one implementation, the total number of target data shards is determined based on the total number of first tasks that the storage node can perform invalid data deletion tasks on the data disk. The process of determining the total number of target data shards includes: the storage node determines the total number of first tasks, and then determines the total number of target data shards based on the total number of first tasks. The total number of first tasks can be understood as the total number of tasks that the storage node can perform invalid data deletion tasks on the data disk within its capabilities. The total number of first tasks is obtained based at least on the processing capacity that the storage node can allocate to the data disk and the tasks that the storage node has not completed for the data disk. For example, the total number of first tasks is obtained based on the first upper limit value of the total number of tasks that the storage node can perform invalid data deletion tasks on the data disk and the first total number of invalid data deletion tasks that the storage node has not completed for the data disk. For example, the total number of first tasks is equal to the difference between the first upper limit value and the first total number. The first upper limit value is determined by the ability of the data disk to perform invalid data deletion operations, and the first upper limit value can be regarded as the maximum concurrency of the data disk. For example, assuming that the storage node allocates a processor core to each data disk, and the core allocated to the data disk is used to perform invalid data deletion operations on the data disk, the first upper limit value corresponding to the data disk is determined by factors such as the processing power of the core allocated to the data disk and the IO capacity of the data disk. The first total number is the total number of invalid data deletion tasks that the storage node has not completed for the data disk. The invalid data deletion tasks that the storage node has not completed for the data disk include: tasks that the storage node is executing on the data disk, and tasks that the storage node has allocated to the data disk but has not yet started to execute. The storage node allocates a task counter to the data disk, and the task counter can count the tasks that have been allocated to the data disk and the tasks that have been completed therein. Then the storage node can obtain the first total number based on the counting result of the task counter. For example, the first total number is equal to the difference between the number of tasks that have been allocated to the data disk and the number of tasks that have been completed therein. At this time, the first total number can be regarded as the occupied capacity of the capacity owned by the data disk. When the total number of target data shards is obtained based on the total number of first tasks, it can be considered that the total number of target data shards is obtained based on the remaining capacity after deducting the occupied capacity from the capacity of the data disk for performing the invalid data deletion operation. When the storage node includes multiple data disks, if the storage node allocates the same capacity for performing the invalid data deletion operation to the multiple data disks, the total number of target data shards of the multiple data disks obtained in this way is substantially equal to the sum of the total number of invalid data deletion tasks that have not been completed for the corresponding data disks.In this way, the concurrent number of invalid data deletion operations performed by multiple data disks can be basically equal, which can ensure the real-time balance of invalid data deletion operation tasks performed by multiple data disks, thereby ensuring the read and write performance of the storage node.

[0079] In another implementation, the total number of target data shards is determined based on the total number of second tasks that the storage node can perform invalid data deletion tasks on all data disks in the storage node. The process of determining the total number of target data shards includes: the storage node obtains the total number of second tasks, and then determines the total number of target data shards based on the total number of second tasks. The second total number of tasks can be understood as the total number of tasks that the storage node can perform invalid data deletion tasks on all data disks in the storage node within its capabilities. The second total number of tasks is obtained based at least on the processing capacity of the storage node and the unfinished tasks of the storage node. For example, the second total number of tasks is obtained based on the second upper limit value of the total number of tasks that the storage node can perform invalid data deletion tasks on multiple data disks, and the second total number of invalid data deletion tasks that the storage node has not completed on multiple data disks. For example, the second total number of tasks is equal to the difference between the second upper limit value and the second total number. The second upper limit value is determined by the ability of the storage node to perform invalid data deletion operations, and the second upper limit value can be regarded as the maximum concurrency of the storage node. For example, assuming that the storage node is allocated with a core of a processor for performing an invalid data deletion operation, the second upper limit value is determined by factors such as the processing power of the core and the IO capacity of the storage node. The second total is the total number of invalid data deletion tasks that the storage node has not completed for all data disks in the storage node. The invalid data deletion tasks that the storage node has not completed for all data disks in the storage node include: the total number of tasks that the storage node is executing for all data disks, and the total number of tasks that the storage node has allocated to all data disks but has not yet started to execute. The storage node is allocated with a task counter for the storage node, and the task counter can count all the tasks assigned to the storage node and the completed tasks therein respectively. Then the storage node can obtain the second total based on the counting result of the task counter. For example, the second total is equal to the difference between the number of all the tasks assigned to the storage node and the number of the completed tasks therein. At this time, the second total can be regarded as the occupied capacity of the storage node for performing invalid data deletion operations. When the total number of target data shards is obtained based on the second total number of tasks, it can be regarded as the total number of target data shards based on the remaining capacity after the storage node's capacity for performing invalid data deletion operations is subtracted from the occupied capacity.

[0080] Further, the total number of target data shards can be obtained based on the total number of the first tasks and the total number of the second tasks. At this time, the total number of target data shards is limited by the total number of the first tasks and the total number of the second tasks. For example, the total number of target data shards can be selected as the minimum value of the total number of the first tasks and the total number of the second tasks. At this time, the resource consumption of the invalid data deletion operation to be performed needs to meet the capacity corresponding to the minimum value of the total number of the first tasks and the total number of the second tasks, so as to determine the real-time balancing of the load of the data disk performing the invalid data deletion operation.

[0081] Step 403: The storage node determines, based on the invalid data ratio of each data shard in the data disk and the total number of target data shards, a plurality of target data shards on which invalid data deletion operations need to be performed among the plurality of data shards in the data disk.

[0082] After obtaining the invalid data ratio of each data shard in the data disk and the total number of target data shards that need to perform invalid data deletion operations, multiple target data shards that need to perform invalid data deletion operations can be determined from multiple data shards in the data disk based on the invalid data ratio of each data shard and the total number. In one implementation, the storage node can optionally sort the multiple data shards of the data disk in order of the invalid data ratio from large to small, and select the data shards that are ranked first in the sorting result, and the selected data shards are the target data shards. In this way, it can be ensured that the invalid data ratio of multiple target data shards is greater than the invalid data ratio of other data shards. Among them, other data shards are any data shards in the data disk except multiple target data shards. Since the larger the invalid data ratio of a data shard is, the lower the utilization rate of the storage space of the data shard is, and the more necessary it is to perform invalid data deletion operations on the data shard, so by determining the target data shard in this way, it can be ensured that limited resources are used as much as possible on more urgent tasks, and the effective use of invalid data deletion resources is achieved.

[0083] Step 404: The storage node allocates computing resources of the storage node to perform the invalid data deletion operation.

[0084] Before executing the invalid data deletion operation, the storage node may optionally allocate the computing resources of the storage node for executing the invalid data deletion operation. Optionally, the storage node may optionally obtain the first priority of the computing resources of the storage node for the invalid data deletion operation, and then allocate the computing resources of the storage node for executing the invalid data deletion operation based on the first priority. For example, considering that if the invalid data deletion operation occupies the computing resources used by the read and write operations, it will affect the performance of the read and write operations, then the first priority may be lower than the second priority of the storage node for executing the read and write operations. That is, the priority of the computing resources used by the invalid data deletion operation is lower than the priority of the computing resources used by the read and write operations. The read and write operations are used to respond to the user's read and write operations. The first priority may be configured by the maintenance personnel of the storage node, or the first priority may be determined by the storage node itself, and the embodiments of the present application do not specifically limit this. In one implementation, the priority of the computing resources used by the invalid data deletion operation is lower than the priority of the computing resources used by the read and write operations, which may be manifested as when the invalid data deletion operation and the read and write operations need to use the computing resources at the same time, the computing resources are preferentially allocated to the read and write operations. And, after allocating computing resources to read and write operations, if there are no remaining computing resources, the invalid data deletion operation will be queued for allocation, and after the waiting time of the invalid data deletion operation reaches a specified time, when there are idle computing resources, the computing resources are allocated to the invalid data deletion resource. In another implementation, the priority of the invalid data deletion operation in using computing resources is lower than the priority of the read and write operations in using computing resources, which can be manifested as allocating a larger proportion of computing resources to read and write operations and a smaller proportion of computing resources to invalid data deletion operations among the computing resources owned by the storage node. By allocating computing resources to invalid data deletion operations according to priority, and the first priority is lower than the second priority of the storage node in using computing resources for executing read and write operations, the storage node can ensure that the read and write operations have priority in using computing resources, avoid invalid data deletion operations from preempting the computing resources of read and write operations, and thus ensure the performance of read and write operations.

[0085] Step 405: The storage node performs a deletion operation on multiple data blocks included in multiple target data shards.

[0086] The process of a storage node performing an invalid data deletion operation on multiple data blocks included in multiple target data shards is actually a process in which the storage node distributes invalid data deletion tasks to multiple target data shards, and performs invalid data deletion operations on target data shards to which invalid data deletion tasks are distributed. In the present application, the storage node may optionally distribute invalid data deletion tasks according to a load balancing strategy. Its implementation process includes: the storage node distributes invalid data deletion tasks to multiple target data shards by executing a distribution process until a stop condition is reached. The stop condition includes: the total number of distributed tasks reaches a specified total number. The specified total number may be equal to the total number of first tasks or the total number of second tasks. The distribution process includes: the storage node updates the total number of third tasks of invalid data deletion tasks that are currently unfinished for each target data shard in multiple target data shards, and distributes invalid data deletion tasks to the target data shard with the smallest total number of third tasks in multiple target data shards. For example, the storage node obtains the total number of third tasks of invalid data deletion tasks that are currently unfinished for each target data shard in the data disk, and sorts the multiple target data shards in the order of the total number of third tasks from small to large, and distributes invalid data deletion tasks to the target data shard ranked first. After the distribution is completed, the total number of third tasks of the invalid data deletion tasks that are currently unfinished for each target data shard in the data disk is re-acquired, and the multiple target data shards are sorted in order from small to large according to the total number of third tasks, and then the invalid data deletion task is distributed to the target data shard ranked first. According to the distribution logic loop, the distribution is stopped until the stop condition is reached, and the resources reserved for the undistributed tasks are released. Among them, distributing invalid data deletion tasks to target data shards means encapsulating the target data shard into a task so that the storage node performs invalid data deletion operations on the target data shard. In this way, invalid data deletion tasks are distributed to multiple target data shards, not only can the number of tasks distributed to all data disks in the storage node be basically the same, but it can also ensure that the load of all data disks is in a relatively balanced state, which helps to ensure the performance of the storage node.

[0087] In a storage system, different data may have different access frequencies. For example, some data is updated frequently, while other data is rarely updated. These frequently updated data are called hot data, and these rarely updated data are called cold data. In the process of invalid data deletion, if cold data and hot data are migrated to the same data block, the frequent updates of the hot data will cause the cold data on the data block to be frequently migrated during the subsequent invalid data deletion process, resulting in more resource consumption in the invalid data deletion process. In view of this, the storage node in this application migrates cold data and hot data differently according to the access popularity of the data to solve the above problem. In one implementation method, if Figure 5 As shown, the implementation process includes:

[0088] Step 4051: The storage node obtains the access popularity of the data recorded in the first data block in the target data shard, where the first data block is any one of the multiple data blocks.

[0089] According to the previous description, the data shard includes multiple data blocks. When performing an invalid data deletion operation on the target data shard, the invalid data deletion operation can be performed in units of data shards. In step 4051, the storage node can first obtain the access heat of each data recorded in the first data block in the target data shard, and then perform differentiated migration of cold data and hot data based on the access heat. Optionally, the storage node can also obtain the access heat of each valid data recorded in the first data block. The access heat of data can be reflected by the access frequency of the data. The access heat of data is positively correlated with the access frequency. When the access frequency of data is higher, the access heat of the data is higher. The access frequency indicates the number of times the data is accessed within a specified time period. Among them, the storage node records the access time of the data, and the access frequency of the data can be determined based on the access time.

[0090] Optionally, the present application can not only reflect the access popularity through the access frequency of the data, but also the access popularity can be optionally reflected through the access timeliness of the data. The access timeliness of the data reflects how close the time when the data is accessed is to the current time. When the time when the data is accessed is closer to the current time, the access timeliness of the data is greater, and vice versa. The access popularity of the data is positively correlated with the access timeliness. When the access timeliness of the data is greater, the access popularity of the data is higher. Among them, the storage node records the access time of the data, and the access timeliness of the data can be determined based on the access time.

[0091] According to the previous description, a data block includes multiple data pages. When the storage node migrates valid data, it can choose to migrate the valid data in units of data pages. Correspondingly, when the storage node obtains the access heat of data, it can choose to obtain the access heat of valid data in units of data pages. That is, the storage node obtains the access heat of valid data recorded in the data block in the target data shard, which actually means that the storage node obtains the access heat of each data page in the target data shard. Its implementation process includes: the storage node obtains the time when the data page is accessed, and then obtains the access frequency and / or access time of the data page based on the access time, and obtains the access heat of the data page according to the access frequency and / or access time. Among them, when any data recorded in the data page is accessed, it is considered that the data page is accessed. For the relationship between the access frequency and / or access time of the data page and the access heat of the data page, please refer to the relationship between the access frequency and / or access time of the data and the access heat of the data. When the access popularity of data is determined based on the access frequency and access timeliness of the data, the accuracy and timeliness of identifying hot and cold data can be guaranteed.

[0092] For example, when the access frequency is represented by the number of accesses, the access heat hd[k], the number of accesses access[k], and the access decayFactor of the data page in the kth time period satisfy the following formula:

[0093] hd[k]=hd[k-1]×decayFactor+access[k]×(1-decayFactor).

[0094] Where hd[0]=0, k is a positive integer. hd[k] represents the access popularity of the data page in the kth time period. decayFactor represents the access age of the data page, also known as the time decay parameter, and its value range is (0, 1). It can be seen from the formula that the access age affects the calculation result of the access popularity as a weight. Therefore, when determining the value of the access age, if you are more concerned about the impact of the access age on the access popularity, you can choose to set the value of the access age to a smaller value, such as setting the value of the access age to 0.03. access[k] represents the number of accesses to the data page in the kth time period. When a storage node needs to perform multiple rounds of invalid data deletion operations on a storage shard, each round of execution is called a time period.

[0095] Step 4052: The storage node migrates valid data in the data recorded in the first data block based on the access popularity of the data recorded in the first data block in the target data shard.

[0096] After obtaining the access heat of the data, the storage node can migrate the valid data based on the access heat. In one implementation, in order to migrate cold data and hot data differently, the storage node can migrate the data recorded in multiple data pages whose access heat is at the same heat level to the same data block. For example, the cold data and hot data are migrated to different data blocks respectively. When migrating valid data in units of data pages, a data page whose access heat is greater than a heat threshold is a hot data page, and all data recorded therein is hot data; a data page whose access heat is less than or equal to the heat threshold is a cold data page, and all data recorded therein is cold data, and the access heat of all data recorded in the same data page is equal to the access heat of the data page. Figure 6As shown, the squares filled with circles represent hot data, the squares filled with black represent cold data, and the squares filled with slashes represent invalid data. According to the migration principle of migrating the data recorded in multiple data pages whose access heat is at the same heat level to the same data block, after the data in data block 1, data block 2 and data block 3 are migrated, the hot data in data block 1 and data block 3 are migrated to data block 4, and the cold data in data block 1 and data block 2 are migrated to data block 5. Among them, the division of heat levels can be set based on application requirements. For example, in a scenario with strict cost control, each heat level is set to have a smaller span to reduce the invalid migration of cold data to a greater extent, thereby reducing the computing resources and IO resource consumption caused by cold data. By migrating the data recorded in multiple data pages whose access heat is at the same heat level to the same data block, the separation of cold data and hot data is achieved. In this way, during the business operation, the frequency of recycling the data blocks where cold data is located will be reduced, which can reduce the invalid migration of cold data and save computing resources and IO resources.

[0097] It should be noted that, although the above text uses data and data pages as examples to explain the process of migrating valid data, it does not exclude that the storage node can also migrate valid data at other granularities, and the other granularity can be larger than the granularity of the data page, or smaller than the granularity of the data page. When migrating valid data at other granularities, it is necessary to obtain the access heat of the storage unit of the other granularity. When migrating valid data, the data recorded in the storage unit of the other granularity is migrated to the storage unit of the larger granularity.

[0098] Step 4053: The storage node migrates all valid data in the data recorded in the first data block to obtain a second data block. The data recorded in the second data block is all invalid data, and the second data block is deleted.

[0099] After the storage node completes the migration of valid data on the data block, the first data block of the migrated data becomes the second data block. The data recorded in the second data block is all invalid data. Then the second data block can be deleted to complete the invalid data deletion operation on the data block. Figure 6 As shown in FIG. 1 , after the valid data in data blocks 1 to 3 are migrated to data blocks 4 and 5, the migrated data blocks 1 to 3 are recorded as the second data blocks, and the migrated data blocks 1 to 3 can be deleted. It should be noted that when a data block contains all invalid data, the data block can be directly deleted without migrating the data recorded in it. Figure 6 Middle data block 6.

[0100] Optionally, after the storage node starts to perform the invalid data deletion operation on the target data shard, it can also choose to stop or continue to perform the invalid data deletion operation according to the execution situation. For example, in the process of performing the invalid data deletion operation on the target data shard, the storage node can optionally obtain the business load of the data disk to which the target data shard belongs. When the business load of the data disk to which the target data shard belongs is greater than a first threshold, the storage node stops performing the invalid data deletion operation on the target data shard. On this basis, the storage node can also choose to continue to perform the invalid data deletion operation on the target data shard when the business load of the data disk to which the target data shard belongs is less than or equal to a second threshold.

[0101] The business load may be selected as the business delay of the data disk. At this time, the business load is greater than the first threshold, which means that the business delay of the data disk is greater than the corresponding first delay threshold. When the business delay is greater than the first delay threshold, it means that the read and write business of the data disk is relatively busy. If the invalid data deletion operation is stopped for the target data slice in the data disk, the computing resources and IO resources occupied by the invalid data deletion operation can be allocated to the read and write operations to ensure the performance of the read and write operations. Alternatively, the business load is the resource ratio occupied by the data disk. For example, when the resources occupied by the data disk are processor resources, the resource ratio occupied by the data disk may be the ratio of the processor resource time slice occupied by the data disk. When the ratio is large, it means that the data disk has occupied more resources. At this time, the business load is greater than the first threshold, which means that the resource ratio occupied by the data disk is greater than the corresponding first ratio threshold. When the resource ratio occupied by the data disk is greater than the first ratio threshold, it means that the read and write business of the data disk is relatively busy. If the invalid data deletion operation is stopped for the target data slice in the data disk, the computing resources and IO resources occupied by the invalid data deletion operation can be allocated to the read and write operations to ensure the performance of the read and write operations. When the business load of the data disk is less than or equal to the second threshold, it means that the read and write business of the data disk is no longer busy, and invalid data deletion operations can be performed while performing read and write operations. The storage node can continue to perform invalid data deletion operations on the target data shards in the data disk.

[0102] Among them, the first threshold value and the second threshold value can be equal or unequal. The values ​​of the first threshold value and the second threshold value can be set according to application requirements, for example, the first threshold value and the second threshold value can both be equal to 2 milliseconds. When the first threshold value and the second threshold value are not equal, the second threshold value is less than the first threshold value. For example, in order to avoid switching back and forth between task states, the first threshold value and the second threshold value can be set to be unequal, and the first threshold value is a specified multiple of the second threshold value (such as 1.2 times). When the business load is between the first threshold value and the second threshold value, the task state can be maintained in the current state. For example, assuming that the business load is the business delay, the strategy for stopping and continuing to perform invalid data deletion operations based on the business load is shown in Table 1. When the current state is to perform an invalid data deletion operation, if the business delay is less than the second delay threshold, the current state is maintained unchanged, that is, the invalid data deletion operation is continued. When the current state is to perform an invalid data deletion operation, if the business delay is less than the first delay threshold, and the business delay is greater than or equal to the second delay threshold, the current state is maintained unchanged, that is, the invalid data deletion operation is continued. When the current state is to execute the invalid data deletion operation, if the service delay is greater than or equal to the first delay threshold, the current state is switched, that is, the invalid data deletion operation is stopped. When the current state is to stop executing the invalid data deletion operation, if the service delay is less than the second delay threshold, the current state is switched, that is, the invalid data deletion operation is continued. When the current state is to stop executing the invalid data deletion operation, if the service delay is less than the first delay threshold, and the service delay is greater than or equal to the second delay threshold, the current state is kept unchanged, that is, the invalid data deletion operation is continued to be stopped. When the current state is to stop executing the invalid data deletion operation, if the service delay is greater than or equal to the first delay threshold, the current state is kept unchanged, that is, the invalid data deletion operation is continued to be stopped.

[0103] Table 1

[0104]

[0105] Optionally, after starting to execute the invalid data deletion operation, the storage node can continuously obtain the business load of the data disk to which the target data shard belongs, or periodically obtain the business load of the data disk to which the target data shard belongs, so as to decide whether to continue or stop executing the invalid data deletion operation based on the business load.

[0106] It should be noted that the above description of the invalid data deletion method is a description of executing a round of invalid data deletion process. And the total number of target data shards that need to perform invalid data deletion operations above is the total number of target data shards that need to perform invalid data deletion operations in one round. The above total number of first tasks is the total number of tasks that the storage node can perform invalid data deletion tasks on the data disk in one round. The above total number of second tasks is the total number of second tasks that the storage node can perform invalid data deletion tasks on multiple data disks in one round. When the storage node needs to perform multiple rounds of invalid data deletion operations on the storage shards, the implementation process of executing each round of invalid data deletion process can refer to the above description accordingly, and this application will not repeat it.

[0107] In summary, in the invalid data deletion method provided in the embodiment of the present application, when the storage node selects the data shard on which the invalid data deletion operation needs to be performed from the multiple data shards of the data disk, the storage unit granularity targeted is the data shard. The storage unit granularity targeted when the storage node performs the invalid data deletion operation is the data shard. It can be seen that the storage unit granularity targeted when the storage node determines the storage unit on which the invalid data deletion operation needs to be performed is the same as the storage unit granularity targeted when the invalid data deletion operation is performed. In this way, the storage node can select the storage unit on which the invalid data deletion operation needs to be performed from a global perspective, and can more evenly schedule the invalid data deletion tasks of multiple data disks in the storage node, ensuring the real-time balance of multiple data disks performing invalid data deletion operations, helping to balance the concurrency of multiple data disks performing invalid data deletion operations, avoiding performance problems caused by the emergence of hot disks, and ensuring the read and write performance of the storage node.

[0108] It should be noted that the sequence of steps of the invalid data deletion method provided in the embodiment of the present application can be adjusted appropriately, and the steps can be increased or decreased accordingly according to the situation. Any technician familiar with the technical field can easily think of a method of change within the technical scope disclosed in this application, which should be included in the protection scope of this application, so it will not be repeated.

[0109] The following is an example of a virtual device in an embodiment of the present application.

[0110] The above describes the invalid data deletion method of the embodiment of the present application. Corresponding to the above method, the embodiment of the present application also provides an invalid data deletion device. The device is applied to a storage node. The storage node includes multiple data disks. The data disk includes multiple data shards. The data shards include multiple data blocks. Figure 7 Schematic diagram of the structure of an invalid data deletion device provided by an embodiment of the present application. Figure 7 The following multiple components are shown: Figure 7 The invalid data deletion device shown can perform the above Figure 4It should be understood that the device may include more additional components than the components shown or omit some of the components shown, and the embodiments of the present application are not limited to this. Optionally, the invalid data deletion device may be configured on a cloud platform. Figure 7 As shown, the invalid data deleting device 70 may include:

[0111] The acquisition module 701 is used to obtain the invalid data ratio of each data shard in the data disk, where the invalid data ratio is the ratio of the total amount of invalid data in the data shard to the total data capacity of the data shard.

[0112] Determination module 702 is used to determine, based on the invalid data ratio, multiple target data shards in the data disk that need to perform invalid data deletion operations, the invalid data ratio of the multiple target data shards is greater than the invalid data ratio of other data shards, and the other data shards are any data shards in the data disk except the multiple target data shards.

[0113] The processing module 703 is used to perform a deletion operation on multiple data blocks included in the multiple target data slices.

[0114] Optionally, the acquisition module 701 is specifically used to: obtain the invalid data ratio of the data blocks in the data shard, and obtain the invalid data ratio in the data shard based on the invalid data ratio of a specified number of data blocks in the data shard, wherein the invalid data ratio of the specified number of data blocks is greater than the invalid data ratio of other data blocks, and the other data blocks are any data blocks in the data shard except the specified number of data blocks.

[0115] Optionally, the processing module 703 is specifically used to: migrate valid data recorded in the first data block based on the access popularity of the data recorded in the first data block in the target data shard, where the first data block is any one of the multiple data blocks; complete the migration of all valid data recorded in the first data block to obtain a second data block, and the data recorded in the second data block are all invalid data, and delete the second data block.

[0116] Optionally, the data block includes multiple data pages, and the processing module 703 is specifically used to: obtain the access heat of the data recorded in the multiple data pages in the first data block, the access heat including multiple heat levels; migrate the data recorded in the multiple data pages whose access heat is at the same heat level to the same data block.

[0117] Optionally, the access popularity includes the access frequency and / or access timeliness of the data. The access frequency reflects the number of times the data is accessed within a specified time period, and the access timeliness reflects how close the data is to the current time.

[0118] Optionally, the processing module 703 is further configured to: when the business load of the data disk to which the target data slice belongs is greater than a first threshold, stop performing the invalid data deletion operation on the target data slice.

[0119] Optionally, the processing module 703 is further used to: when the business load of the data disk to which the target data slice belongs is less than or equal to a second threshold, continue to perform the invalid data deletion operation on the target data slice, and the second threshold is less than or equal to the first threshold.

[0120] Optionally, the processing module 703 is also used to: obtain a first priority level of the computing resources of the storage node used for the invalid data deletion operation, the first priority level being lower than a second priority level of the computing resources used by the storage node for executing read and write operations, the read and write operations being used to respond to the user's read and write operations; based on the first priority level, allocate computing resources of the storage node used for executing the invalid data deletion operation.

[0121] Optionally, the determination module 702 is further used to: determine the total number of first tasks that the storage node can perform invalid data deletion tasks on the data disk; and determine the total number of target data shards based on the total number of first tasks.

[0122] Optionally, the first total number of tasks is obtained based on a first upper limit value of the total number of invalid data deletion tasks that the storage node can perform on the data disk and a first total number of invalid data deletion tasks that the storage node has not completed on the data disk.

[0123] Optionally, the determination module 702 is further used to obtain the total number of second tasks that the storage node can perform on multiple data disks to delete invalid data; based on the total number of second tasks, determine the total number of target data shards.

[0124] Optionally, the second total number of tasks is obtained based on a second upper limit value of the total number of invalid data deletion tasks that the storage node can perform on multiple data disks and a second total number of invalid data deletion tasks that the storage node has not completed on the multiple data disks.

[0125] Optionally, the processing module 703 is used to: distribute invalid data deletion tasks to multiple target data shards by executing a distribution process until a stop condition is reached, the stop condition including: the total number of distributed tasks reaches a specified total number, the distribution process including: updating the total number of third tasks of the currently unfinished invalid data deletion tasks for each target data shard in the multiple target data shards, distributing invalid data deletion tasks to the target data shard with the smallest total number of third tasks among the multiple target data shards; and performing invalid data deletion operations on the target data shard to which the invalid data deletion tasks are distributed.

[0126] Here, please refer to the description in the previous method embodiment for the detailed working process of the acquisition module 701, the determination module 702 and the processing module 703. For example, the acquisition module 701 adopts the implementation method of step 401 to obtain the invalid data ratio of each data slice in the data disk. The determination module 702 adopts the implementation method of step 403 to determine the multiple target data slices that need to perform invalid data deletion operations. The processing module 703 adopts the implementation method of step 405 to perform a deletion operation on the multiple data blocks included in the multiple target data slices. The embodiments of this application will not be repeated here.

[0127] In summary, in the invalid data deletion device provided in the embodiment of the present application, when the determination module selects a data shard on which an invalid data deletion operation needs to be performed from multiple data shards of a data disk, the granularity of the storage unit targeted is the data shard. The granularity of the storage unit targeted when the processing module performs the invalid data deletion operation is the data shard. It can be seen that the granularity of the storage unit targeted when determining the storage unit on which an invalid data deletion operation needs to be performed is the same as the granularity of the storage unit targeted when the invalid data deletion operation is performed. In this way, the storage node can select the storage unit on which the invalid data deletion operation needs to be performed from a global perspective, and can more evenly schedule the invalid data deletion tasks of multiple data disks in the storage node, ensuring the real-time balance of multiple data disks performing invalid data deletion operations, helping to balance the concurrency of multiple data disks performing invalid data deletion operations, avoiding performance problems caused by hot disks, and ensuring the read and write performance of the storage node.

[0128] Among them, the acquisition module 701, the determination module 702 and the processing module 703 can all be implemented by software, or can be implemented by hardware. Exemplarily, the following takes the acquisition module 701 as an example to introduce the implementation of the acquisition module 701. Similarly, the implementation of the determination module 702 and the processing module 703 can refer to the implementation of the acquisition module 701.

[0129] As an example of a software functional unit, the acquisition module 701 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module 701 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0130] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0131] As an example of a hardware functional unit, the acquisition module 701 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 701 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0132] The multiple computing devices included in the acquisition module 701 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 701 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 701 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0133] It should be noted that, in other embodiments, any one of the acquisition module 701, the determination module 702 and the processing module 703 can be used to execute any step in the invalid data deletion method. The steps that the acquisition module 701, the determination module 702 and the processing module 703 are responsible for implementing can be specified as needed, and the acquisition module 701, the determination module 702 and the processing module 703 respectively implement different steps in the invalid data deletion method to achieve the full functions of the invalid data deletion device.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each component described above can refer to the corresponding content in the aforementioned method embodiment, and will not be repeated here.

[0135] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0136] Optionally, the structure of at least one computing device included in the computing device cluster can be found in Figure 3 The computing device 300 shown. The memory 302 in one or more computing devices 300 in the computing device cluster may store the same instructions for executing the invalid data deletion method.

[0137] In some possible implementations, the memory 302 of one or more computing devices 300 in the computing device cluster may also store partial instructions for executing the invalid data deletion method. In other words, the combination of one or more computing devices 300 may jointly execute instructions for executing the invalid data deletion method.

[0138] It should be noted that the memory 302 in different computing devices 300 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the invalid data deletion device. That is, the instructions stored in the memory 302 in different computing devices 300 can implement the functions of one or more modules in the acquisition module 701, the determination module 702 and the processing module 703.

[0139] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 8 A possible implementation is shown. Figure 8 As shown, two computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device. In this type of possible implementation, computing devices 800A and 800B include a bus 802, a processor 804, a memory 806, and a communication interface 808.

[0140] It should be understood that Figure 8 The functions of the computing device 800A shown in the figure can also be completed by multiple computing devices 800. Similarly, the functions of the computing device 800B can also be completed by multiple computing devices 800. And the deployment mode of the module for implementing the invalid data deletion method in the computing device can also be adjusted according to the application requirements.

[0141] An embodiment of the present application also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium. The computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device implements the invalid data deletion method provided in the embodiment of the present application.

[0142] The embodiment of the present application also provides a computer program product including instructions. When the computer program product is executed on a computer, the computer implements the invalid data deletion method provided by the embodiment of the present application.

[0143] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0144] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the original data and executable code involved in this application are all obtained with full authorization.

[0145] In the embodiments of the present application, the terms "first", "second" and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. The term "at least one" means one or more, and the term "plurality" means two or more, unless otherwise expressly defined.

[0146] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0147] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concept and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for deleting invalid data, characterized in that: The method is applied to a storage node, the storage node includes a plurality of data disks, the data disks include a plurality of data shards, the data shards include a plurality of data blocks, and the method includes: Obtaining the invalid data ratio of each data shard in the data disk, where the invalid data ratio is the ratio of the total amount of invalid data in the data shard to the total data capacity of the data shard; Based on the invalid data proportion, multiple target data shards on which invalid data deletion operations need to be performed are determined among the multiple data shards in the data disk, the invalid data proportions of the multiple target data shards are all greater than the invalid data proportions of other data shards, and the other data shards are any data shards in the data disk except the multiple target data shards; A deletion operation is performed on the multiple data blocks included in the multiple target data slices.

2. The method according to claim 1, characterized in that The obtaining the invalid data ratio of each data shard in the data disk includes: Obtain the invalid data ratio of the data blocks in the data shard, and obtain the invalid data ratio in the data shard based on the invalid data ratio of a specified number of data blocks in the data shard, wherein the invalid data ratio of the specified number of data blocks is greater than the invalid data ratio of other data blocks, and the other data blocks are any data blocks in the data shard except the specified number of data blocks.

3. The method according to claim 1 or 2, characterized in that The performing a deletion operation on the multiple data blocks included in the multiple target data slices includes: Migrating valid data in the data recorded in a first data block in the target data shard based on access popularity of the data recorded in the first data block, where the first data block is any one of the multiple data blocks; All valid data in the data recorded in the first data block is migrated to obtain a second data block, and all data recorded in the second data block are invalid data, so the second data block is deleted.

4. The method according to claim 3, characterized in that The data block includes a plurality of data pages, and the migrating valid data in the data recorded in the first data block based on the access popularity of the data recorded in the first data block in the target data shard includes: Acquire access heat of data recorded in a plurality of data pages in the first data block, wherein the access heat includes a plurality of heat levels; The data recorded in the multiple data pages whose access heats of the data recorded in the multiple data pages are at the same heat level are migrated to the same data block.

5. The method according to claim 3 or 4, characterized in that The access popularity includes the access frequency and / or access timeliness of the data. The access frequency reflects the number of times the data is accessed within a specified time period, and the access timeliness reflects how close the data is to the current time.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: When the business load of the data disk to which the target data slice belongs is greater than a first threshold, the invalid data deletion operation on the target data slice is stopped.

7. The method according to claim 6, characterized in that The method further comprises: When the business load of the data disk to which the target data shard belongs is less than or equal to a second threshold, the invalid data deletion operation continues to be performed on the target data shard, and the second threshold is less than or equal to the first threshold.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Obtaining a first priority of the computing resources of the storage node used by the invalid data deletion operation, the first priority being lower than a second priority of the computing resources used by the storage node for executing a read / write operation, the read / write operation being used to respond to a read / write operation of a user; Based on the first priority, computing resources of the storage node are allocated to perform the invalid data deletion operation.

9. An invalid data deletion device, characterized in that: The device is applied to a storage node, the storage node includes a plurality of data disks, the data disks include a plurality of data slices, the data slices include a plurality of data blocks, and the device includes: An acquisition module, used to acquire the invalid data ratio of each data shard in the data disk, where the invalid data ratio is the ratio of the total amount of invalid data in the data shard to the total data capacity of the data shard; A determination module, configured to determine, based on the invalid data proportion, a plurality of target data slices in the data disk on which invalid data deletion operations need to be performed, wherein the invalid data proportions of the plurality of target data slices are all greater than the invalid data proportions of other data slices, and the other data slices are any data slices in the data disk except the plurality of target data slices; The processing module is used to perform a deletion operation on the multiple data blocks included in the multiple target data slices.

10. The device according to claim 9, characterized in that The acquisition module is specifically used for: Obtain the invalid data ratio of the data blocks in the data shard, and obtain the invalid data ratio in the data shard based on the invalid data ratio of a specified number of data blocks in the data shard, wherein the invalid data ratio of the specified number of data blocks is greater than the invalid data ratio of other data blocks, and the other data blocks are any data blocks in the data shard except the specified number of data blocks.

11. The device according to claim 9 or 10, characterized in that The processing module is specifically used for: Migrating valid data in the data recorded in a first data block in the target data shard based on access popularity of the data recorded in the first data block, where the first data block is any one of the multiple data blocks; All valid data in the data recorded in the first data block is migrated to obtain a second data block, and all data recorded in the second data block are invalid data, so the second data block is deleted.

12. The device according to claim 11, characterized in that The data block includes a plurality of data pages, and the processing module is specifically used for: Acquire access heat of data recorded in a plurality of data pages in the first data block, wherein the access heat includes a plurality of heat levels; The data recorded in the multiple data pages whose access heats of the data recorded in the multiple data pages are at the same heat level are migrated to the same data block.

13. The device according to claim 11 or 12, characterized in that The access popularity includes the access frequency and / or access timeliness of the data. The access frequency reflects the number of times the data is accessed within a specified time period, and the access timeliness reflects how close the data is to the current time.

14. The device according to any one of claims 9 to 13, characterized in that: The processing module is also used for: When the business load of the data disk to which the target data slice belongs is greater than a first threshold, the invalid data deletion operation on the target data slice is stopped.

15. The device according to claim 14, characterized in that The processing module is also used for: When the business load of the data disk to which the target data shard belongs is less than or equal to a second threshold, the invalid data deletion operation continues to be performed on the target data shard, and the second threshold is less than or equal to the first threshold.

16. The device according to any one of claims 9 to 15, characterized in that: The processing module is also used for: Obtaining a first priority of the computing resources of the storage node used by the invalid data deletion operation, the first priority being lower than a second priority of the computing resources used by the storage node for executing a read / write operation, the read / write operation being used to respond to a read / write operation of a user; Based on the first priority, computing resources of the storage node are allocated to perform the invalid data deletion operation.

17. A computing device, characterized in that The method comprises a computing device, wherein the computing device comprises a processor and a memory, wherein program instructions are stored in the memory, and the processor executes the program instructions so that the computing device executes any one of the methods of claims 1 to 8.

18. A computer-readable storage medium, characterized in that: The method comprises program instructions, and when the program instructions are executed on a computing device, the computing device is caused to execute the method according to any one of claims 1 to 8.

19. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 8.

Images