Resource management method, interface, and distributed file system

Abstract

The present application provides a resource management method, an interface, and a distributed file system. The resource management method comprises in response to a first task of processing first data in a target file, acquiring a data location of the first data in a corresponding target storage node, the first data being data of a specified segment in the target file; marking the data location as in a deleted state; when a second task of performing resource management on the target storage node is triggered, releasing resource space occupied by the data corresponding to the data location in the deleted state. By means of the present application, the technical problem with resource management means in the related art of increased network and storage costs is resolved, achieving the technical effect of reducing network and storage costs.

Description

[0001] This application relates to the field of data storage, and more particularly to a resource management method, interface, and distributed file system. Background Technology: Distributed storage systems refer to data stored on multiple independent storage servers and managed through a network. The multi-replica mechanism provides high reliability, high availability, and high scalability. Distributed storage systems connect multiple storage servers into a storage cluster; a typical storage cluster contains thousands of storage servers. However, the upper-layer business of distributed storage systems is basically based on the Log-Structured Merge-Tree (LSM-Tree) model. (LSM-Tree is a hierarchical, ordered data storage structure designed for storage media that only support append-only writes, or storage media where sequential write performance is far superior to random write performance (such as conventional magnetic recording hard disk drives, CMR HDDs). Its core idea is to transform random writes into sequential writes. First, a large number of writes are cached in memory. When they accumulate to a certain level, they are written to files in batches. This allows multiple data entries to be written in a single I / O, making full use of each I / O operation to achieve efficient sequential data writing. Sequential writing is much faster than random writing; it appends content, not updating in place, similar to the append mode of ordinary log writing, which does not overwrite old entries.) This architecture is fundamentally based on achieving high performance and a simple architecture. A necessary function of this architecture is background garbage collection. To promptly release storage space occupied by junk data, current resource management methods often increase network and storage costs. No effective solution has yet been proposed in related technologies to address this technical problem. This application provides a resource management method, interface, and distributed file system to solve one or more of the above-mentioned technical problems. In a first aspect, this application provides a resource management method applied to a distributed file system, comprising: responding to a first task of processing first data in a target file, obtaining the data location of the first data in a corresponding target storage node, wherein the first data is data in a specified segment of the target file; marking the data location as deleted; and releasing the resource space occupied by the data corresponding to the deleted data location when a second task of resource management of the target storage node is triggered.Secondly, embodiments of this application provide a resource management method applied to a distributed file system, comprising: responding to a file trimming task sent by a service layer, obtaining the data location of the first data to be trimmed by the file trimming task in the corresponding target storage node, wherein the first data is data of a specified segment in a target file; marking the data location as deleted; and releasing the resource space occupied by the data corresponding to the deleted data location when a garbage collection task of the target storage node is triggered. Thirdly, embodiments of this application provide a resource management interface, set in a distributed file system, configured to manage resources in the target storage node using the above-described resource management method. Fourthly, embodiments of this application provide a distributed file system, comprising: a positioning module, configured to, upon receiving the first task, locate the computing node corresponding to the chunk containing the first data; and a computing node, configured to manage resources in the target storage node using the above-described resource management method. Fifthly, embodiments of this application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method described in any of the above-described embodiments. Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in any of the preceding claims. Seventhly, embodiments of this application provide a computer program product including computer instructions, which, when executed by a processor, implement the method described in any of the preceding claims. Compared with related technologies, this application has the following advantages: In embodiments of this application, in response to a first task of processing first data in a target file, the data location of the first data in a corresponding target storage node is obtained, wherein the first data is data in a specified segment of the target file; the data location is marked as deleted; and when a second task of resource management for the target storage node is triggered, the resource space occupied by the data corresponding to the deleted data location is released.In other words, the embodiments of this application can specify a segment of data (Range) of a file for deletion without deleting the entire file, thus freeing up the target space. This differs from related technologies, which treat a file as the smallest unit of resource management, requiring the entire file to be deleted before space is released (e.g., reading valid data from file A, rewriting it to a new file B, leaving unused data unmoved, then deleting file A to release the overall space occupied by file A). This solves the technical problem of increased network and storage costs associated with resource management methods in related technologies, achieving the effect of reducing network and storage costs. The above description is merely an overview of the technical solution of this application. To better understand the technical means of this application, it can be implemented according to the contents of the specification. Furthermore, to make the above and other objects, features, and advantages of this application more apparent, specific embodiments of this application are described below. Brief Description of the Drawings: In the drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these accompanying drawings only depict some embodiments according to this application and should not be considered as limiting the scope of this application. Figure 1 shows a schematic diagram of a resource management scheme provided in an embodiment of this application; Figure 2 shows a flowchart of a resource management method provided in an embodiment of this application; Figure 3 shows a schematic diagram (I) of a resource management method provided in an embodiment of this application; Figure 4 shows a schematic diagram (II) of a resource management method provided in an embodiment of this application; Figure 5 shows a schematic diagram (III) of a resource management method provided in an embodiment of this application; Figure 6 shows a schematic diagram (IV) of a resource management method provided in an embodiment of this application; Figure 7 shows a schematic diagram (V) of a resource management method provided in an embodiment of this application; Figure 8 shows a schematic diagram (VI) of a resource management method provided in an embodiment of this application; Figure 9 shows a schematic diagram (VII) of a resource management method provided in an embodiment of this application; Figure 10 shows a schematic diagram (VIII) of a resource management method provided in an embodiment of this application; Figure 11 shows a structural block diagram of a resource management device provided in an embodiment of this application; Figure 12 shows a structural block diagram of a resource management interface provided in an embodiment of this application; Figure 13 shows a flowchart of another resource management method provided in an embodiment of this application; Figure 14 shows a structural block diagram of a distributed file system provided in an embodiment of this application; Figure 15 shows a block diagram of an electronic device used to implement an embodiment of this application. The following detailed description only includes some exemplary embodiments.As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and descriptions are considered exemplary in nature and not restrictive. To facilitate understanding of the technical solutions of the embodiments of this application, the related technologies of the embodiments of this application are described below. The following related technologies are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this application, all of which fall within the protection scope of the embodiments of this application. The application scenarios of the embodiments of this application include, but are not limited to, block storage (Elastic Block Storage, EBS), object storage, etc. Basically, the upper-layer services of distributed file systems are designed based on the LSM-Tree model, and the methods provided in the embodiments of this application can be used. These service scenarios all have service-layer garbage collection mechanisms. Figure 1 is a schematic diagram of an application environment provided by an embodiment of this application. As shown in Figure 1, the upper-layer service of the distributed file system is block storage (EBS), whose architecture is based on the LSM-Tree model. Files in the distributed file system contain valid data (1) and garbage data (2). This includes EBS front-end read / write (EBS reads / writes data from the distributed file system through the block server) and EBS garbage collection read / write (after garbage collection starts, it needs to read valid data from the files and delete invalid data). Typically, the server-side write amplification (write amplification refers to the multiple disk write operations required by the LSM-Tree during write operations. Each write operation requires writing data to the log and flushing dirty pages. Even if only one byte of data is modified, the entire page needs to be written to disk. These multiple disk write operations lead to write amplification, meaning the actual amount of data written is greater than the amount of data modified) is calculated using the following formula: the greater the garbage collection write volume, the greater the EBS-side write amplification, which reflects the increase in network (throughput) and storage costs (SSD lifetime consumption). Therefore, the need to reduce server-side garbage collection and write amplification, thereby reducing network and storage costs, is increasing. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.The technical solution of this application and how the technical solution of this application solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings. The server side can achieve the effect of releasing garbage data space through a garbage collection process. Currently, there are two main implementation methods: Method 1: Read the valid data (marked by number 1) in file A and rewrite it to a new file B, while not moving the garbage data that needs to be released (marked by number 2). Delete file A, releasing the overall space occupied by file A (as shown in Figure 2), thereby achieving a release of data space. Clearly, moving valid data in Figure 2 consumes network bandwidth and the storage life of the SSD disk, increasing the storage cost on the server side. Method 2: Distributed file systems provide overwrite operations. When a user no longer needs a certain piece of data (e.g., data 2), they can overwrite the location of the garbage data (using data 1 to overwrite data 2), thereby reusing the space occupied by the garbage data from the service level (as shown in Figure 3). However, this method is complex to implement. Currently, distributed file systems only support sequential write file semantics. Providing overwrite file semantics would be very difficult in terms of overall architecture and implementation. From the engineering practices of previous versions supporting overwrite capabilities, it is not a sufficiently simple and excellent architecture design. In order to effectively solve the above technical problems, this application provides a resource management method applied to a distributed file system. Figure 4 shows a flowchart of a resource management method according to an embodiment of this application. The method may include: Step S402, in response to a first task of processing first data in a target file, obtaining the data location of the first data in the corresponding target storage node, wherein the first data is data of a specified segment in the target file. It should be noted that the above first data may be a piece of garbage data or multiple pieces of garbage data in the target file. Because the service layer is based on an LSM Tree architecture, new data is written to the distributed storage system files in an append-only manner (i.e., data modifications are not replaced in place, but are made by continuously adding logs). When users use the cloud disk, if there are overwrite write operations, garbage space that needs to be reclaimed will be generated, and the corresponding actual garbage data is part of the data in the file.Therefore, the first task mentioned above could be to scan the index information after the garbage collection process is triggered at the service layer, generate garbage information in the file based on the scan results, and then generate a corresponding file trimming task based on this garbage information. The target storage node mentioned above could be a partitioned storage device, where the entire device is divided into multiple fixed-size, contiguous partitions (zones). Each partition only supports strict sequential writes (as shown in Figure 5, new data is written starting from the position pointed to by the write pointer, and the direction in which the write pointer moves to the next position is shown by the horizontal arrow in Figure 5), and erases are performed on a partition-space basis. There are two typical types of partitioned storage devices: Zone Namespace Solid State Disk (ZNS SSD) and Shingled Magnetic Recording Disk.

[0002] Magnetic Recording Hard Disk Drive (SMR HDD) OStep S404: Mark the data location as deleted. It should be noted that in this embodiment, if there are multiple data locations, all of them need to be marked as deleted. Step S406: When the second task of resource management for the target storage node is triggered, release the resource space occupied by the data corresponding to the deleted data location. It should be noted that the second task can be a garbage collection task. The resource space can be a part of the ZNS SSD partition space. Optionally, in this embodiment, the method of releasing the resource space occupied by the data corresponding to the deleted data location includes, but is not limited to: directly deleting the data corresponding to the deleted data location (for example, assuming there are 3 data locations, data location 1, data location 2, and data location 3, where data location 2 is marked as deleted, then the actual data corresponding to data location 2 is directly deleted from the storage node; or data location 1 and data location 2 are marked as deleted, then the actual data corresponding to data location 1 and data location 2 are directly deleted from the storage node). Through the above steps S402-S406, in response to the first task of processing the first data in the target file, the data location of the first data in the corresponding target storage node is obtained, wherein the first data is data of a specified segment in the target file; the data location is marked as deleted; when the second task of resource management of the target storage node is triggered, the resource space occupied by the data corresponding to the deleted data location is released. That is to say, the embodiments of this application can specify a segment of data (Range) of a file for deletion without deleting the entire file, thereby releasing the target space. This is different from the related technologies, which treat the file as the smallest unit of resource management, requiring the entire file to be deleted before the space is released (for example, reading the valid data in file A, rewriting it to a new file B, not moving the garbage data that needs to be released, and then deleting file A to release the overall space occupied by file A). This solves the technical problem that resource management methods in related technologies often increase network and storage costs, and achieves the technical effect of reducing network and storage costs.In one possible implementation, before obtaining the data location of the first data in the corresponding target storage node, the method further includes: S11, after the target file is divided into multiple chunks, dividing the chunk containing the first data into multiple data blocks; S12, storing the data locations of the multiple data blocks in the target storage node in the memory of the computing node corresponding to the target storage node. Optionally, the computing node can be a storage engine (chunk server), such as a partitioned storage engine. That is, in this embodiment, the storage engine is a storage engine that provides chunk semantics and supports writing and deleting. The partitioned storage engine is a single-machine storage engine corresponding to a partitioned device, a service process running on each storage server, configured to provide storage services on a single storage server, including high-performance read / write, user input / output (I / O) request scheduling, data correctness scanning, and abnormal data reporting functions. In addition, in the embodiments of this application, a distributed storage system file is usually divided into chunks of variable length (e.g., chunk 1, chunk 2, chunk 3, chunk 4), and each chunk is saved to different chunk servers in multiple copies (e.g., copy 1, copy 2, copy 3) or in an encoded manner. Data block server 2 and data block server 3 are shown in Figure 6. In this embodiment, the computing node (i.e., the storage engine) provides the ability to fine-grained control over the data within a file, and each part of the data can have an independent lifecycle. As shown in Figure 7, block 3 is a 1M block, composed of three 1M data blocks. The location information of each 1M data block is maintained in the memory of the single-machine storage engine. These three 1M data blocks correspond to three actual locations on the ZNS SSD disk (location 1, location 2, and location 3 in Figure 7), and these locations store the actual data of block 3. The single-machine storage engine has the ability to independently control the lifecycle of these three 1M data blocks, that is, it can delete one data block without affecting the reading of other data blocks. As shown in Figure 8, the storage engine memory stores the location information of the three data blocks (data block 1, data block 2, and data block 3) of block 3. Based on this location information, data block 2 is deleted, and only the data corresponding to data block 1 and data block 3 are valid data in the corresponding ZNS SSD partition space. Optionally, in this embodiment, the block containing the first data can be one block or multiple blocks. When there are multiple blocks containing the first data, it is necessary to store the data locations of the multiple data blocks included in each of the multiple blocks in the target storage node in the memory of the computing node corresponding to the target storage node. For example, assuming the first data corresponds to block 2 and block 3 respectively, block 2 is divided into data block 1, data block 2, and data block 3, and block 3 is divided into data block 4, data block 5, and data block 6, then it is necessary to store the data locations of data blocks 1 to 6 in the target storage node in the memory of the computing node corresponding to the target storage node. This allows for independent control of data blocks 1 to 6, such as deleting one or more of them. In one possible implementation, the aforementioned first data (i.e., garbage data) may exist partly in data block 1 and partly in data block 2. In this case, the user can choose not to delete data block 1 and data block 2 initially, but wait until both data block 1 and data block 2 have become garbage data before deleting them. Through the above S11-S12, the ability to achieve fine-grained control over the internal data of a file is provided, that is, the ability to finely control the data lifecycle is opened up, unlike related technologies that maintain file-level location information on computing nodes, which can only achieve overall maintenance.In one possible implementation, obtaining the data location of the first data in the corresponding target storage node includes: S21, obtaining the data block corresponding to the first data to obtain the target data block; S22, obtaining the data location of the target data block from the memory of the computing node, and setting the data location of the target data block as the data location of the first data in the corresponding target storage node. For example, if the first data corresponds to data block B, the location of data block B in the ZNS SSD can be obtained from the storage engine memory, and then the location of data block B is set as the data location of the first data. As another example, if the first data corresponds to data blocks B and C, the data locations of data blocks B and C can be obtained from the storage engine memory, and then the data locations of data blocks B and C are set as the data location of the first data. Through the above steps S21-S22, the data location of a segment of data in the target file in the storage node can be determined. In one possible implementation, obtaining the data block corresponding to the first data and obtaining the target data block includes: S31, obtaining the metadata of the multiple data blocks stored in the memory of the computing node; S32, determining the data block corresponding to the first data through the metadata. It should be noted that the aforementioned metadata can be data describing the data, mainly information describing data attributes, used to support functions such as indicating storage location, historical data, resource lookup, and file records. For example, if block 3 corresponds to data block 1, data block 2, and data block 3 respectively, then data block 2 can be determined to be the data block corresponding to the first data, or data blocks 2 and 3 can be determined to be the data blocks corresponding to the first data, based on their corresponding metadata. Therefore, in this embodiment, the data block corresponding to the first data (garbage data) can be determined based on the metadata of multiple data blocks. In one possible implementation, after releasing the resource space occupied by the data corresponding to the deleted data location, the process further includes: S41, obtaining the second data other than the first data in the chunk containing the first data; S42, writing the second data into the target partition space of the target storage node in a sequential write manner. It should be noted that the first data can be garbage data in a file, and the second data can be valid data in a file.For example, assuming there are three data locations: data location 1, data location 2, and data location 3, where data location 2 is marked as deleted, the data corresponding to data locations 1 and 3 are written sequentially to the target partition space of the target storage node. Alternatively, if data locations 1 and 2 are marked as deleted, the data corresponding to data location 3 is written sequentially to the target partition space of the target storage node. That is, after deleting the garbage data in the target storage node, this application embodiment also proposes a method for garbage space reclamation. Specifically, the second data (valid data) is written sequentially to another partition space (target partition space) of the target storage node, and the partition space containing the first data (garbage data) is released entirely (wherein, the partition space containing the first data includes the space occupied by the first data and the space occupied by the second data), thereby achieving the ability to release storage space. Optionally, the above sequential writing method can be a process of writing data sequentially to the disk, that is, transferring data blocks directly from memory to adjacent physical locations on the disk. Through the aforementioned steps S41-S42, only valid data is moved during garbage collection, saving a significant amount of data movement and conserving network bandwidth and disk throughput for the server side. In one possible implementation, after writing the second data sequentially into the partition space of the target storage node, the method further includes: S51, determining whether the first data exists in the partition space of the target storage node; S52, if the first data does not exist, returning the result that the first data has been deleted to the upper-layer service of the distributed file system. That is, after the single-machine storage engine triggers garbage collection, it performs a pruning operation on the first data. After the first data is pruned, this embodiment checks whether the read first data has been pruned in the single-machine storage engine. If it has been pruned, it will explicitly return that the data has been deleted, without any undefined behavior. Optionally, the target storage node is determined as follows: After the target file is divided into multiple chunks, the chunk containing the first data is determined; the computing node containing the first data is determined after multiple copies of the chunk are stored; the target storage node is determined through the computing node, wherein each copy corresponds one-to-one with a computing node. As shown in Figure 9, the file is divided into 4 chunks, each chunk corresponding to 3 copies: copy 1, copy 2, and copy 3. These 3 copies are stored in data block server 1, data block server 2, and data block server 3, respectively.A segment of data to be deleted in the file is divided into block 2, and then this segment of data is stored in replicas 1, 2, and 3 respectively. Therefore, when deleting this segment of data, data block server 1, data block server 2, and data block server 3 need to delete the segment of data stored in their respective replicas. The following uses an EBS scenario as an example to describe an embodiment of this application. It should be noted that in this example, the storage engine corresponds to the compute node, the ZNS SSD corresponds to the storage node, the file pruning task corresponds to the first task, and the garbage collection corresponds to the second task. As shown in Figure 10, the main steps include:

[0003] 551. When EBS triggers the garbage collection process, it scans the index information. Based on the scan results, it can generate garbage information in the file, and further generate corresponding file trimming tasks based on the garbage information.

[0004] 552. The EBS consumes file pruning tasks, which involve passing garbage information to the distributed file system. If this transmission is successful, EBS assumes the distributed file system will handle the deletion and release of this space and provides feedback to the EBS layer via space statistics. 553. After receiving the pruning task, the distributed file system, based on the garbage information and the corresponding chunk information, locates the chunk server responsible for this data and passes the garbage information to the chunk server (i.e., the storage engine).

[0005] 554. After receiving the pruning information, the storage engine marks the corresponding data location as deleted. When the garbage collection mechanism is triggered, only valid data is moved, and data in the deleted state is not moved, thus releasing the garbage data space. See the two areas in Figure 10 for details. In summary, this application proposes a new distributed file operation, namely file pruning, based on a partitioned storage engine, which realizes fine-grained release capability of file space. Through testing in a real service scenario, it is verified that file pruning can effectively reduce the write amplification of the server side by more than 16%, effectively reducing the network and storage costs of the server side. For example, in one scenario, the EBS layer creates a cloud disk, and a testing tool is used on the cloud disk to trigger a 1M I / O write with completely random access. When the same logical block address (LBA) on the cloud disk is written repeatedly, the corresponding distributed file system will generate pruning garbage data. Each piece of garbage data is deleted using the file pruning method described above, verifying that service layer write amplification can be reduced to 72%. In another scenario, full I / O records for 4 days were collected from 10 cloud disks in the online cluster. The same 10 cloud disks were created in the test cluster, and the online I / O records were replayed in sequence using the EBS replay tool, verifying that service-side write amplification can be effectively reduced by 4%-35%. Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a resource management device. Figure 11 shows a resource management structure block diagram according to an embodiment of this application. The device may include: a first acquisition module 1102, configured to acquire the data location of the first data in the corresponding target storage node in response to a first task of processing first data in a target file, wherein the first data is data of a specified segment in the target file. The first data may be a segment of garbage data or multiple segments of garbage data in the target file. Since the service layer is based on an LSM Tree architecture, new data is written to the distributed storage system file in the form of append write (i.e., data modification does not replace the data in place, but is modified by continuously adding logs). When a user uses the cloud disk, if there is an overwrite write, garbage space that needs to be reclaimed will be generated, and the corresponding real garbage data is part of the data in the file. The first task may be to scan the index information after the service layer triggers the garbage collection process, generate garbage information in the file based on the scan results, and generate a corresponding file trimming task based on the garbage information.The target storage node mentioned above can be a partitioned storage device, which divides the entire device into multiple fixed-size, contiguous partitions (zones). Each partition only supports strict sequential writes (as shown in Figure 5, data is written starting from the position pointed to by the write pointer, and the direction in which the write pointer moves to the next position is shown by the horizontal arrow in Figure 5), and erases are performed on a partition-space basis, as shown in Figure 5. There are two typical types of partitioned storage devices: Zone Namespace Solid State Disk (ZNS SSD) and Shingled Magnetic Recording Hard Disk Drive (SMR HDD). OThe marking module 1104 marks the data location as deleted. It should be noted that in this embodiment, when there are multiple data locations, the marking module 1104 is also configured to mark all multiple data locations as deleted. The management module 1106 is configured to release the resource space occupied by the data corresponding to the deleted data location when the second task of resource management for the target storage node is triggered. It should be noted that the second task can be a garbage collection task. The resource space can be a portion of the ZNS SSD partition space. Optionally, in this embodiment, the method for releasing the resource space occupied by the data corresponding to the deleted data location includes, but is not limited to: directly deleting the data corresponding to the deleted data location (for example, assuming there are 3 data locations, data location 1, data location 2, and data location 3, where data location 2 is marked as deleted, then the actual data corresponding to data location 2 is directly deleted in the storage node; or data location 1 and data location 2 are marked as deleted, then the actual data corresponding to data location 1 and data location 2 respectively are directly deleted in the storage node). Using the apparatus shown in FIG11, in response to a first task of processing first data in a target file, the data location of the first data in the corresponding target storage node is obtained, where the first data is data in a specified segment of the target file; the data location is marked as deleted; and when a second task of resource management for the target storage node is triggered, the resource space occupied by the data corresponding to the deleted data location is released. In other words, the embodiments of this application can specify a certain segment of data (Range) of a file for deletion without deleting the entire file, thereby freeing up the target space. This is different from related technologies, which treat a file as the smallest unit of resource management and require the entire file to be deleted before the space is freed up (for example, reading the valid data in file A, rewriting it into a new file B, not moving the garbage data that needs to be released, and then deleting file A to release the overall space occupied by file A). This solves the technical problem that resource management methods in related technologies often increase network and storage costs, and achieves the technical effect of reducing network and storage costs.In one possible implementation, the above apparatus further includes: a partitioning module, configured to partition the chunk containing the first data into multiple data blocks after the target file is partitioned into multiple chunks; and a storage module, configured to store the data locations of the multiple data blocks in the target storage node in the memory of the computing node corresponding to the target storage node. Optionally, the computing node can be a storage engine (chunk server), such as a partitioned storage engine. That is, in this embodiment, the storage engine is a storage engine that provides chunk semantics and supports writing and deletion. The partitioned storage engine is a single-machine storage engine corresponding to the partitioned device, a service process running on each storage server, configured to provide storage services on a single storage server, including high-performance read / write, user input / output (I / O) request scheduling, data correctness scanning, and abnormal data reporting functions. In addition, in the embodiments of this application, a distributed storage system file is usually divided into chunks of variable length (e.g., chunk 1, chunk 2, chunk 3, chunk 4), and each chunk is saved to different chunk servers in multiple copies (e.g., copy 1, copy 2, copy 3) or in an encoded manner. As shown in Figure 6, there are three data block servers (e.g., data block server 1, data block server 2, and data block server 3). In this embodiment, the compute node (i.e., the storage engine) provides the ability to fine-grained control over the data within a file, allowing each part of the data to have an independent lifecycle. As shown in Figure 7, block 3 is a 1MB block composed of three 1MB data blocks. The location information of each 1MB data block is maintained in the memory of the single-machine storage engine. These three 1MB data blocks correspond to three actual locations on the ZNS SSD disk (location 1, location 2, and location 3 in Figure 7), where the actual data of block 3 is stored. The single-machine storage engine has the ability to independently control the lifecycle of these three 1MB data blocks; that is, one data block can be deleted without affecting the reading of the other data blocks. Optionally, in this embodiment, the block containing the first data can be one block or multiple blocks. When there are multiple blocks containing the first data, it is necessary to store the data locations of the multiple data blocks included in each of the multiple blocks in the target storage node in the memory of the computing node corresponding to the target storage node. For example, assuming the first data corresponds to block 2 and block 3 respectively, block 2 is divided into data block 1, data block 2, and data block 3, and block 3 is divided into data block 4, data block 5, and data block 6, then it is necessary to store the data locations of data blocks 1 to 6 in the target storage node in the memory of the computing node corresponding to the target storage node. This allows for independent control of data blocks 1 to 6, such as deleting one or more of them. In one possible implementation, the aforementioned first data (i.e., garbage data) may exist partly in data block 1 and partly in data block 2. In this case, the user can choose not to delete data block 1 and data block 2 initially, but wait until both data block 1 and data block 2 become garbage data before deleting them. This module provides the capability for fine-grained control over the internal data of a file, enabling precise control over the data lifecycle, unlike related technologies that maintain file-level location information on the computing node, which only allows for overall maintenance. Optionally, the aforementioned first acquisition module 1102 may further include: a first acquisition unit configured to acquire the data block corresponding to the first data to obtain a target data block; and a setting unit configured to acquire the data location of the target data block from the memory of the computing node and set the data location of the target data block as the data location of the first data in the corresponding target storage node.For example, if the first data corresponds to data block B, the location of data block B in ZNS SSD can be obtained from the storage engine memory, and then the location of data block B is set as the data location of the first data. As another example, if the first data corresponds to data blocks B and C, the data locations of data blocks B and C can be obtained from the storage engine memory, and then the data locations of data blocks B and C are set as the data location of the first data. Through the above unit, the data location of a segment of data in the target file within the storage node can be determined. Optionally, the first acquisition unit may further include: an acquisition subunit, configured to acquire the metadata of the multiple data blocks stored in the memory of the computing node; and a determination subunit, configured to determine the data block corresponding to the first data using the metadata. It should be noted that the metadata can be data describing the data, mainly information describing data attributes, used to support functions such as indicating storage location, historical data, resource lookup, and file records. For example, if block 3 corresponds to data block 1, data block 2, and data block 3 respectively, then data block 2 can be identified as the data block corresponding to the first data, or data blocks 2 and 3 can be identified as the data blocks corresponding to the first data, based on their corresponding metadata. Therefore, in this embodiment, the data block corresponding to the first data (garbage data) can be determined based on the metadata of multiple data blocks. The resource management also includes a second acquisition module, configured to acquire, after releasing the resource space occupied by the data corresponding to the deleted data location, the second data other than the first data in the block chunk where the first data is located; and a writing module, configured to write the second data into the target partition space of the target storage node in a sequential manner. For example, suppose there are three data locations: data location 1, data location 2, and data location 3. Data location 2 is marked as deleted. Then, the data corresponding to data location 2 is deleted, and the data corresponding to data locations 1 and 3 are written to the target partition space of the target storage node in a sequential manner. Alternatively, if data locations 1 and 2 are marked as deleted, the data corresponding to data locations 1 and 2 is deleted, and the data corresponding to data location 3 is written to the target partition space of the target storage node in a sequential manner. It should be noted that the first data mentioned above could be garbage data in the file, while the second data mentioned above could be valid data in the file.For example, assuming there are three data locations: data location 1, data location 2, and data location 3, where data location 2 is marked as deleted, the data corresponding to data locations 1 and 3 are written sequentially to the target partition space of the target storage node. Alternatively, if data locations 1 and 2 are marked as deleted, the data corresponding to data location 3 is written sequentially to the target partition space of the target storage node. That is, after deleting the garbage data in the target storage node, this application embodiment also proposes a method for garbage space reclamation. Specifically, the second data (valid data) is written sequentially to another partition space of the target storage node.

[0006] In the target partition space, the entire partition space containing the first data (garbage data) is released (wherein, the partition space containing the first data includes the space occupied by the first data and the space occupied by the second data), thereby realizing the ability to release storage space. Optionally, the above sequential write method can be a process of writing data to disk sequentially, that is, transferring data blocks directly from memory to adjacent physical locations on the disk. Through the above second acquisition module and writing module, only valid data is moved during garbage collection, saving a lot of data movement, and saving network bandwidth and disk throughput for the server side. In a possible implementation, the above management module 1106 also includes a confirmation unit, which is configured to determine whether the first data exists in the partition space of the target storage node after the second data is written to the partition space of the target storage node in a sequential write manner; if it is determined that the first data does not exist, the result that the first data has been deleted is returned to the upper-layer service of the distributed file system. That is, after the single-machine storage engine triggers garbage collection, it performs a pruning operation on the first data. After the first data is pruned, this embodiment checks in the single-machine storage engine whether the read first data has been pruned. If it has been pruned, it will explicitly return that the data has been deleted, and no indeterminate behavior will occur. Optionally, the target storage node is determined in the following way: after the target file is divided into multiple chunks, the chunk in which the first data is located is determined; the computing node after multiple copies of the chunk in which the first data is located are determined; the target storage node is determined through the computing node, wherein the copy corresponds one-to-one with the computing node. As shown in Figure 9, the file is divided into 4 chunks, each chunk corresponds to 3 copies, copy 1, copy 2, and copy 3, which are stored in data block server 1, data block server 2, and data block server 3, respectively. A segment of data to be deleted from the file is divided into block 2, and then this segment of data is stored in replicas 1, 2, and 3 respectively. Therefore, when deleting this segment of data, data block server 1, data block server 2, and data block server 3 need to delete the segment of data stored in their respective replicas. In summary, the above-mentioned device proposed in this application embodiment implements a new distributed file operation, namely file pruning, based on a partitioned storage engine, and further realizes the ability to release file space with fine granularity. Corresponding to the application scenario and method provided in this application embodiment, this application embodiment also provides a resource management interface.Figure 12 shows a block diagram of a resource interface structure according to an embodiment of this application. This interface is configured to manage resources in the target storage node using the aforementioned resource management method. That is, based on the existing append-write and random access interfaces in the distributed file system, this embodiment also proposes a resource management interface. Through the resource management interface shown in Figure 12, a specific segment (Range) of data in a file can be deleted without deleting the entire file, thus freeing up the target space. This solves the technical problem that resource management methods in related technologies often increase network and storage costs, achieving the effect of reducing network and storage costs. Corresponding to the application scenario and method provided in this embodiment, this embodiment also provides a resource management method applied to a distributed file system, as shown in Figure 13, including:

[0007] S1302, In response to the file trimming task sent by the service layer, obtain the data location of the first data to be trimmed by the file trimming task in the corresponding target storage node, wherein the first data is the data of a specified segment in the target file;

[0008] S1304, Mark the data location as deleted;

[0009] S1306, when the garbage collection task of the target storage node is triggered, the resource space occupied by the data corresponding to the deleted data position is released. Through the above steps S1302-S1306, a certain segment of data (Range) of a file can be deleted without deleting the entire file, thus releasing the target space. This is different from related technologies, which treat the file as the smallest unit of resource management, requiring the entire file to be deleted before the space is released (for example, reading the valid data in file A, rewriting it to a new file B, not moving the garbage data that needs to be released, and then deleting file A to release the overall space occupied by file A). This solves the technical problem that resource management methods in related technologies often increase network and storage costs, achieving the technical effect of reducing network and storage costs. Corresponding to the application scenario and method provided in the embodiments of this application, the embodiments of this application also provide a distributed file system. Figure 14 shows a block diagram of a distributed file system structure according to an embodiment of this application, including: a positioning module 1402, configured to locate the computing node corresponding to the chunk containing the first data after receiving the first task; and a computing node 1404, configured to manage the resources in the target storage node using the resource management method described above. Optionally, in response to the first task of processing the first data in the target file, the computing node 1404 obtains the data location of the first data in the corresponding target storage node, wherein the first data is data in a specified segment of the target file; marks the data location as deleted; and releases the resource space occupied by the data corresponding to the deleted data location when the second task of resource management of the target storage node is triggered. Optionally, the positioning module 1402 is further configured to determine the computing node after multiple copies of the chunk containing the first data are saved, wherein each copy corresponds one-to-one with a computing node. The system shown in Figure 14 allows for the deletion of a specific segment (Range) of a file without deleting the entire file, thus freeing up target space. This solves the problem that resource management methods in related technologies often increase network and storage costs, achieving the goal of reducing network and storage costs. The functions of each module in the devices and systems of this application embodiment can be found in the corresponding descriptions of the methods described above, and they possess corresponding beneficial effects, which will not be repeated here. Figure 15 is a block diagram of an electronic device used to implement the embodiments of this application.As shown in Figure 15, the electronic device includes a memory 1501 and a processor 1502. The memory 1501 stores a computer program that can run on the processor 1502. When the processor 1502 executes the computer program, it implements the method described in the above embodiments. The number of memories 1501 and processors 1502 can be one or more. The electronic device also includes a communication interface 1503, configured to communicate with external devices and perform data exchange and transmission. If the memory 1501, processor 1502, and communication interface 1503 are implemented independently, they can be interconnected via a bus to complete mutual communication. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 15, but this does not indicate that there is only one bus or one type of bus. Optionally, in specific implementations, if the memory 1501, processor 1502, and communication interface 1503 are integrated on a single chip, then the memory 1501, processor 1502, and communication interface 1503 can communicate with each other through an internal interface. This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in this application embodiment. This application also provides a chip including a processor configured to call and execute instructions stored in the memory, causing a communication device with the chip installed to execute the method provided in this application embodiment. This application also provides a chip including an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is configured to execute code in the memory, and when the code is executed, the processor is configured to execute the method provided in the application embodiment.It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture. Further, optionally, the aforementioned memory can include read-only memory and random access memory. This memory can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available.For example, Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). In the above embodiments, it can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the described specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise expressly specified.Any process or method described in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved. The logic and / or steps described in the flowcharts or otherwise herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, instruction execution systems, apparatuses, or devices (such as computer-based systems, processor-included systems, or other systems that can fetch and execute instructions from instruction execution systems, apparatuses, or devices). It should be understood that various parts of this application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments. Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc. The above descriptions are merely exemplary embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

Claims 1. A resource management method applied to a distributed file system, comprising: in response to a first task of processing first data in a target file, obtaining the data location of the first data in a corresponding target storage node, wherein, The first data is the data of a specified segment in the target file; the data position is marked as deleted. When the second task of resource management for the target storage node is triggered, the resource space occupied by the data corresponding to the data location in the deleted state is released.

2. The method according to claim 1, wherein, Before obtaining the data location of the first data in the corresponding target storage node, the method further includes: after the target file is divided into multiple chunks, dividing the chunk containing the first data into multiple data blocks; and storing the data locations of the multiple data blocks in the target storage node in the memory of the computing node corresponding to the target storage node.

3. The method according to claim 2, wherein, Obtaining the data location of the first data in the corresponding target storage node includes: obtaining the data block corresponding to the first data to obtain the target data block; obtaining the data location of the target data block from the memory of the computing node, and setting the data location of the target data block as the data location of the first data in the corresponding target storage node.

4. The method according to claim 3, wherein, Obtaining the data block corresponding to the first data, and obtaining the target data block, includes: obtaining the metadata of the plurality of data blocks stored in the memory of the computing node; and determining the data block corresponding to the first data through the metadata.

5. The method according to claim 1, wherein, After releasing the resource space occupied by the data corresponding to the data location in the deleted state, the method further includes: obtaining the second data other than the first data in the chunk where the first data is located; and writing the second data into the target partition space of the target storage node in a sequential writing manner.

6. The method according to claim 5, wherein, After writing the second data into the partition space of the target storage node in a sequential manner, the method further includes: determining whether the first data exists in the partition space of the target storage node; and if the first data does not exist, returning the result that the first data has been deleted to the upper-layer service of the distributed file system.

7. The method according to claim 1, wherein, The target storage node is determined as follows: After the target file is divided into multiple chunks, the chunk containing the first data is determined; the computing node after multiple copies of the chunk containing the first data are saved is determined; the target storage node is determined through the computing node, wherein each copy corresponds to a computing node.

8. The method according to any one of claims 1-7, wherein, The data in the specified segment of the target file includes: a segment of data in the target file or multiple segments of data in the target file.

9. A resource management interface, set in a distributed file system, configured to manage resources in the target storage node using the resource management method according to any one of claims 1 to 8.

10. A resource management method applied to a distributed file system, comprising: In response to a file trimming task sent by the service layer, the data location of the first data to be trimmed in the target storage node is obtained, wherein the first data is a specified segment of data in the target file; the data location is marked as deleted; when the garbage collection task of the target storage node is triggered, the resource space occupied by the data corresponding to the deleted data location is released.

11. A distributed file system, comprising: The positioning module is configured to, upon receiving the first task as described in claim 1, locate the computing node corresponding to the chunk in which the first data is located; the computing node is configured to manage the resources in the target storage node using the resource management method as described in any one of claims 1 to 8 or claim 10.

12. The distributed file system according to claim 11, wherein, The positioning module is also configured to determine the computing node after multiple copies of the chunk containing the first data are saved, wherein each copy corresponds to a computing node.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-8 or 10.

14. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1-8 or 10.

15. A computer program product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method of any one of claims 1-8 or 10.

Images