Space recovery method and device for file storage system

Abstract

The invention provides a space recovery method and device for a file storage system, and can be applied to the technical field of computers. The method comprises the following steps: receiving deleted file information from each node, and adding a virtual space corresponding to the deleted file information into a to-be-released space queue as a to-be-released virtual space; regularly querying the to-be-released space queue, if a to-be-released virtual space exists in the to-be-released space queue, determining idle nodes in the plurality of nodes, obtaining state information of the idle nodes, determining the number of released tasks to be executed by each idle node according to the state information, and releasing the to-be-released virtual space in the to-be-released space queue. A corresponding number of release tasks are allocated to each idle node, so that each node executes the allocated release tasks, and the release tasks comprise releasing the mapping relation between the to-be-released virtual space and the physical space; and marking the physical space without the mapping relation as idle.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly to a space recovery method and device for a file storage system. Background Art

[0002] File storage systems can optimize resource utilization by dynamically allocating and releasing storage space. Space reclamation is a very important function in file storage systems, which can return unused space to the pool space so that it can be occupied by other file systems.

[0003] In related technologies, space recovery tasks are all performed by the master node of the file storage system. If a large number of files are continuously deleted, resulting in too many objects to be recovered on the master node, the master node will use a large amount of resources to perform space recovery tasks, resulting in other task resources being occupied, making the system's task processing efficiency low. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a space recovery method and device for a file storage system.

[0005] According to the first aspect of the present disclosure, a space recovery method for a file storage system is provided, comprising: receiving deleted file information from each node, and adding the virtual space corresponding to the deleted file information as the virtual space to be released to a queue of space to be released; regularly querying the queue of space to be released, and if there is virtual space to be released in the queue of space to be released, determining an idle node among multiple nodes, obtaining status information of the idle node, determining the number of release tasks to be performed by each idle node based on the status information, and assigning a corresponding number of release tasks to each idle node, so that each node executes the assigned release task, wherein the release task corresponds one-to-one to the virtual space to be released in the queue of space to be released, and the release task includes releasing the mapping relationship between the virtual space to be released and the physical space; and marking the physical space whose mapping relationship is released as idle.

[0006] According to an embodiment of the present disclosure, the status information of a node includes at least one of the CPU occupancy rate and task duration of the node, wherein the task duration is the maximum waiting time of a release task currently to be executed by the node.

[0007] According to an embodiment of the present disclosure, the number of release tasks to be executed by each idle node is determined based on status information, including: when the task duration of the idle node is greater than the preset duration, stopping sending the release tasks to be executed to the idle node; when the task duration of the idle node is less than the preset duration, determining the number of release tasks to be executed by the idle node based on the CPU occupancy of the idle node.

[0008] According to an embodiment of the present disclosure, the above method further includes: when there are adjacent virtual spaces to be released in the space queue to be released, merging the adjacent virtual spaces to be released based on a preset merging threshold so that the merged virtual spaces to be released are less than or equal to the merging threshold.

[0009] According to an embodiment of the present disclosure, the above method also includes: when any node goes offline, in response to the offline node not completing the assigned release task, the virtual space to be released corresponding to the uncompleted release task is marked as to be allocated, and the virtual space to be released corresponding to the uncompleted release task is added to the head of the queue of space to be released.

[0010] According to an embodiment of the present disclosure, the above method also includes: when the task duration of the node is less than the preset duration, determining the number of deleted file information sent by the node based on the central processing unit occupancy rate of the node; when the task duration of the node is greater than the preset duration, setting the number of deleted file information sent by the node to a preset maximum number.

[0011] According to an embodiment of the present disclosure, the above method also includes: when any node fails, determining the service tasks of the failed node as service tasks to be transferred, wherein the service tasks include release tasks; and deploying the service tasks to be transferred on the non-faulty nodes.

[0012] According to an embodiment of the present disclosure, the physical file space is stored in designated hardware, and multiple nodes interact with the designated hardware through a designated network.

[0013] According to an embodiment of the present disclosure, the multiple nodes include a master node and slave nodes, and the above method is executed by the master node.

[0014] The second aspect of the present disclosure provides a space recovery device for a file storage system, comprising: a receiving module for receiving deleted file information from each node, and adding the virtual space corresponding to the deleted file information as the virtual space to be released to a queue of space to be released; an allocation module for periodically querying the queue of space to be released, and if there is virtual space to be released in the queue of space to be released, determining an idle node among multiple nodes, obtaining status information of the idle node, determining the number of release tasks to be performed by each idle node according to the status information, and allocating a corresponding number of release tasks to each idle node, so that each node performs the allocated release tasks, wherein the release tasks correspond one-to-one to the virtual space to be released in the queue of space to be released, and the release tasks include releasing the mapping relationship between the virtual space to be released and the physical space; and a marking module for marking the physical space corresponding to the virtual space whose mapping relationship is released as idle.

[0015] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0016] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0017] The fifth aspect of the present disclosure further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0018] According to an embodiment of the present disclosure, deleted file information is received from each node, and the virtual space corresponding to the deleted file information is added to the queue of spaces to be released as virtual space to be released; the queue of spaces to be released is periodically queried, and if there is virtual space to be released in the queue of spaces to be released, an idle node is determined among multiple nodes, and the status information of the idle nodes is obtained. The number of release tasks to be performed by each idle node is determined based on the status information, and a corresponding number of release tasks are allocated to each idle node, so that each node executes the allocated release tasks, wherein the release tasks correspond one-to-one to the virtual space to be released in the queue of spaces to be released, and the release tasks include unmapping the mapping relationship between the virtual space to be released and the physical space; and marking the physical space whose mapping relationship is unmapped as idle. Since the release tasks can be adaptively allocated according to the status information of the node, the problem of insufficient resources and task congestion caused by only one node thread processing tasks in the related art is avoided, the utilization rate of all nodes is guaranteed, and the overall processing efficiency of the system can be improved. In addition, since the released tasks correspond one-to-one to the virtual spaces to be released in the queue of spaces to be released, the problem of multiple tasks targeting the same virtual space to be released is avoided. There is no need to use locks to limit the processing of the virtual space to be released, which can reduce the complexity of the system and reduce the system's resource overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0020] Figure 1 A diagram schematically illustrates an application scenario of a space recovery method and apparatus for a file storage system according to an embodiment of the present disclosure;

[0021] Figure 2 Schematically shows a flow chart of a space recovery method for a file storage system according to an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram schematically illustrates the relationship between a node and designated hardware according to an embodiment of the present disclosure;

[0023] Figure 4 Schematically illustrates a flow chart for determining the number of released tasks to be executed according to an embodiment of the present disclosure;

[0024] Figure 5 Schematically shows a flow chart of merging virtual spaces to be released according to an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram schematically illustrates a space recovery method for a file storage system according to an embodiment of the present disclosure;

[0026] Figure 7 A block diagram schematically illustrates a structure of a space recovery device for a file storage system according to an embodiment of the present disclosure; and

[0027] Figure 8 A block diagram of an electronic device suitable for implementing a space recovery method for a file storage system according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] In the technical solutions disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] In scenarios where personal information is used for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure all provide users with corresponding operation portals for them to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge, and skills, and have reached a certain level of professionalism.

[0034] An embodiment of the present disclosure provides a space recovery method for a file storage system, comprising: receiving deleted file information from each node, and adding the virtual space corresponding to the deleted file information as a virtual space to be released to a queue of space to be released; regularly querying the queue of space to be released, and if there is virtual space to be released in the queue of space to be released, determining an idle node among multiple nodes, obtaining status information of the idle node, determining the number of release tasks to be performed by each idle node based on the status information, and assigning a corresponding number of release tasks to each idle node, so that each node executes the assigned release task, wherein the release task includes releasing the mapping relationship between the virtual space to be released and the physical space; and marking the physical space corresponding to the virtual space whose mapping relationship is released as idle.

[0035] Figure 1 The following schematically illustrates an application scenario of a space recovery method and device for a file storage system according to an embodiment of the present disclosure.

[0036] like Figure 1 As shown, the application scenario of the space recovery method, device and equipment of the file storage system of the embodiment of the present disclosure includes multiple nodes and designated hardware, wherein each node is provided with a designated network and file system (File System, FS), and each node performs read and write interactions with the designated hardware.

[0037] The designated hardware may be a backend physical storage medium, for example, a hard disk drive, a solid-state drive, a storage tape, or other hardware device.

[0038] The designated network may be a storage area network (SAN). SAN provides an independent storage network to enable access to storage devices and has high availability and high performance.

[0039] A FS is deployed on each node to access and manage data stored on the specified hardware. For example, when deleting a file, the FS interacts with the specified hardware to delete the stored data.

[0040] FS and SAN interact through shared memory based on the paravirtualization device framework (virtio) and accelerated virtualization technology (vhost-scsi) to improve system processing performance.

[0041] Each node has two independent processes, FS and SAN. Each node can read and write specified hardware. FS sends SCSI commands to communicate with SAN through the above-mentioned shared memory, realizing communication between FS and SAN on each node, as well as communication between nodes. This communication is used to receive deleted file information from each node and allocate a corresponding number of release tasks to each idle node.

[0042] It should be understood that Figure 1 The number of nodes and designated hardware in FIG is only illustrative. Any number of nodes and designated hardware may be used depending on the implementation requirements.

[0043] The following will be based on Figure 1 The scene described by Figures 2 to 6 A space recovery method for a file storage system in a disclosed embodiment is described in detail.

[0044] Figure 2 The flowchart of the space recovery method of the file storage system according to the embodiment of the present disclosure is schematically shown.

[0045] like Figure 2 As shown, the space recovery method of the file storage system of this embodiment includes operations S210 to S260.

[0046] In operation S210, deleted file information is received from each node, and the virtual space corresponding to the deleted file information is added to a queue of spaces to be released as virtual spaces to be released.

[0047] A file storage system is a system architecture that organizes multiple computers or servers to work together to store, manage, and process large amounts of data. The file storage system includes multiple nodes that perform data processing tasks within the file storage system.

[0048] Multiple nodes include master nodes and slave nodes. The master node executes the file storage system's space reclamation method. The master node is responsible for managing and coordinating the entire cluster in the file storage system. It plays a core role in the file storage system and assumes cluster control and management functions.

[0049] The deleted file information may be information generated when the file storage system deletes a file and the deletion action is fed back to the node. The deleted file information may be relevant information of the deleted file, such as the file name, file content, and other information. The virtual space of the deleted file can be retrieved through the deleted file information.

[0050] The master node will remove the virtual space node directory entry of the deleted file and mark the virtual space as to be allocated, thereby adding it to the to-be-released space queue as the to-be-released virtual space. The to-be-released space queue can be a linked list on the master node for storing the to-be-released virtual space.

[0051] Figure 3 The diagram schematically shows the relationship between the queue of space to be released and the node according to an embodiment of the present disclosure.

[0052] Each node has a 128k cache area for recording deleted file information. The master node has a 2M cache area to record all virtual spaces and an n×128k queue of to-be-released spaces to record the virtual spaces to be released of n nodes. Each virtual space is recorded by addressing the logical block (lba) and the data length, which takes up 16 bytes in total. Figure 3 As shown, the virtual space to be released is divided into a certain size, and each block is 64k, to record the virtual space to be released corresponding to the released tasks allocated to the four nodes (MDS0, MDS2, MDS4 and MDS5).

[0053] In addition, after the master node adds the virtual space to be released into the space to be released queue, it may send a storage result to the corresponding node to inform the node that the reception of the deleted file information has been completed.

[0054] In operation S220 , it is determined whether there is any virtual space to be released in the queue of space to be released. If not, operation S210 is continued to be executed to obtain a new virtual space to be released. If so, operation S230 is executed.

[0055] In operation S230 , an idle node is determined among the plurality of nodes, and status information of the idle node is acquired.

[0056] The node is queried at regular intervals to determine whether it is in an idle state, so as to identify idle nodes among multiple nodes. The idle state may be determined based on the load performance of the node. For example, when the CPU (Central Processing Unit) occupancy rate of the node reaches 90% or more, the node is in a non-idle state.

[0057] The status information may be the status of the node currently executing related tasks, such as the number of tasks or the execution duration when executing related tasks, where related tasks include released tasks; and the status information may also be the load performance information of the node, such as CPU occupancy, disk occupancy and other information.

[0058] The node status information includes at least one of the node's central processing unit occupancy rate and task duration, wherein the task duration is the maximum waiting time of the node's currently pending release task.

[0059] Specifically, if there are already assigned release tasks on the node, the waiting time for each currently unexecuted release task is determined, and the maximum waiting time is selected as the task duration. For example, at the current 10th second, the node is executing release task A. At this time, release task B received at the 2nd second has been waiting for 8 seconds, and release task C received at the 7th second has been waiting for 3 seconds. Therefore, the maximum waiting time, i.e., the task duration, can be 8 seconds.

[0060] In operation S240, the number of release tasks to be executed by each idle node is determined according to the status information, and a corresponding number of release tasks are allocated to each idle node, so that each node executes the allocated release tasks, where the release tasks correspond one-to-one to the virtual space to be released in the queue of space to be released.

[0061] Each node performs different tasks and occupies different resources accordingly. Therefore, it is necessary to determine the maximum number of tasks that each node can receive and release based on the status information to avoid wasting node resources.

[0062] The release task is used to release the mapping relationship between the virtual space to be released and the physical space. Therefore, the release task corresponds one-to-one to the virtual space to be released in the queue of the space to be released. For example, the release task A can only access the mapping relationship between the virtual space to be released A and the physical space A, and cannot process the mapping relationship between the virtual space to be released B and the physical space B. It can safely release the virtual space that is no longer used, avoid repeated unmapping, and ensure stability and efficiency.

[0063] In operation S250 , it is determined whether the mapping relationship between the virtual space to be released and the physical space is released. If not, operation S240 is continued to be executed until the release task is completed, and then operation S260 is executed.

[0064] In operation S260 , the master node marks the physical space whose mapping relationship is released as free.

[0065] The physical file space is stored on the designated hardware, and multiple nodes interact with the designated hardware through a designated network.

[0066] The designated hardware may be a back-end physical storage medium, such as a hard disk drive, a solid-state drive, a storage tape, or other hardware devices.

[0067] After the mapping relationship is released, the physical space cannot access the virtual space. At this time, both the virtual space and the physical space are released, and the physical space is marked as free so that the virtual space corresponding to other files can be mapped to complete the space recovery of the file storage system.

[0068] According to an embodiment of the present disclosure, deleted file information is received from each node, and the virtual space corresponding to the deleted file information is added to the queue of spaces to be released as virtual space to be released; the queue of spaces to be released is periodically queried, and if there is virtual space to be released in the queue of spaces to be released, an idle node is determined among multiple nodes, and the status information of the idle nodes is obtained. The number of release tasks to be performed by each idle node is determined based on the status information, and a corresponding number of release tasks are allocated to each idle node, so that each node executes the allocated release tasks, wherein the release tasks correspond one-to-one to the virtual space to be released in the queue of spaces to be released, and the release tasks include unmapping the mapping relationship between the virtual space to be released and the physical space; and marking the physical space whose mapping relationship is unmapped as idle. Since the release tasks can be adaptively allocated according to the status information of the node, the problem of insufficient resources and task congestion caused by only one node thread processing tasks in the related art is avoided, the utilization rate of all nodes is guaranteed, and the overall processing efficiency of the system can be improved. In addition, since the released tasks correspond one-to-one to the virtual spaces to be released in the queue of spaces to be released, the problem of multiple tasks targeting the same virtual space to be released is avoided. There is no need to use locks to limit the processing of the virtual space to be released, which can reduce the complexity of the system and reduce the system's resource overhead.

[0069] Figure 4 The flowchart of determining the number of released tasks to be executed according to an embodiment of the present disclosure is schematically shown.

[0070] According to the embodiments of the present disclosure, Figure 4 As shown, determining the number of released tasks to be executed by each idle node according to the state information may include operations S410 to S450.

[0071] In operation S410, it is determined whether the task duration of the idle node is less than a preset duration. If it is greater than or equal to the preset duration, operation S450 is executed to stop sending the release task to be executed to the idle node.

[0072] If it is less than the preset time, the number of release tasks to be executed by the idle node is determined according to the CPU occupancy rate of the idle node. Specifically:

[0073] If the CPU occupancy rate is greater than the first threshold, operation S420 is executed to set the number of release tasks to be executed by the idle node to 0. At this time, the CPU occupancy rate is too high and the current resources of the node are unable to execute the release tasks.

[0074] If the CPU occupancy rate is greater than the second threshold and less than the first threshold, operation S430 is performed to determine the number of released tasks to be executed according to the load of the idle node, for example, the number of released tasks to be executed is determined by the method shown in formula (1).

[0075] (1);

[0076] Wherein, request_count is the number of release tasks to be executed, max_cout is the preset maximum number, and system_load is the load of the node, where the load includes but is not limited to the CPU occupancy of the node.

[0077] If the CPU occupancy rate is less than the second threshold, operation S440 is executed to set the number of release tasks to be executed by the idle node to the preset maximum number. At this time, the CPU occupancy rate is very small and the resources that the node can provide are sufficient to support the preset maximum number of release tasks.

[0078] For example, the master node queries the task duration and CPU occupancy rate of each node through a request, wherein the preset duration can be set to 6 seconds, the first threshold can be set to 90%, and the second threshold can be set to 70%.

[0079] If the maximum waiting time for the current node to execute all assigned release tasks is greater than 6 seconds, the number of release tasks to be executed is determined by the CPU busyness (i.e., central processing unit occupancy). When the CPU occupancy is greater than 70% and less than 90%, the load of the current node is obtained based on the above formula (1) to determine the number of release tasks to be executed; when the CPU occupancy is greater than 90%, request_count = 0; when the CPU occupancy is less than 70%, request_count = max_cout.

[0080] If the maximum waiting time for the current node to execute all the allocated release tasks is less than 6s, then request_count = max_cout.

[0081] Figure 5 The flowchart of merging virtual spaces to be released according to an embodiment of the present disclosure is schematically shown.

[0082] According to the embodiments of the present disclosure, Figure 5 As shown, the merging process of adjacent virtual spaces to be released is as follows Figure 5 As shown, it includes operations S510 to S530.

[0083] In operation S510 , it is determined whether there are adjacent virtual spaces to be released in the queue of spaces to be released. If so, operation S520 is performed to merge the adjacent virtual spaces to be released.

[0084] If the addresses of at least two virtual spaces to be released are adjacent, they can be merged and subsequently processed as a release task to reduce the number of release tasks.

[0085] In operation S530 , it is determined whether the merged virtual space to be released is less than or equal to the merge threshold. If it is greater than the merge threshold, operation S520 is re-executed so that the merged virtual space to be released is less than or equal to the merge threshold.

[0086] Since the size of the virtual space to be released that can be processed by the release task is limited, a merging threshold needs to be set to prevent the virtual space to be released after merging from being too large, which would cause the release task to fail abnormally.

[0087] According to an embodiment of the present disclosure, when any node goes offline, in response to the offline node failing to complete the assigned release task, the to-be-released virtual space corresponding to the uncompleted release task is marked as to be allocated, and the to-be-released virtual space corresponding to the uncompleted release task is added to the head of the to-be-released space queue.

[0088] When a node goes offline, if it has unfinished release tasks, the virtual space to be released corresponding to the unfinished release tasks will be marked as pending allocation and added to the head of the queue of space to be released, so that the virtual space to be released can be processed first and other nodes can complete the corresponding release tasks as soon as possible.

[0089] In addition, when the node comes online, it determines the deleted file information that has not been sent yet, and sends the deleted file information to the master node.

[0090] According to an embodiment of the present disclosure, when the task duration of a node is less than a preset duration, the amount of deleted file information sent by the node to the master node is determined based on the node's central processing unit occupancy rate; when the task duration of the node is greater than the preset duration, the amount of deleted file information sent by the node to the master node is set to a preset maximum amount.

[0091] When the task duration of the node is greater than the preset duration, the number of deleted file information sent by the node is set to a preset maximum number.

[0092] When the task duration of the node is less than the preset duration, the amount of deleted file information sent by the node is determined according to the CPU occupancy rate of the node.

[0093] If the CPU usage is greater than a first threshold, the node is not allowed to send deleted file information in the current state.

[0094] If the CPU occupancy rate is greater than the second threshold and less than the first threshold, the number threshold for sending deleted file information can be determined based on the node load based on the above formula (1). At this time, request_count in formula (1) represents the number threshold for sending deleted file information.

[0095] If the CPU occupancy rate is less than the second threshold, the number of deleted file information sent by the node is set to a preset maximum number.

[0096] According to an embodiment of the present disclosure, when any node fails, the service tasks of the failed node are determined as service tasks to be transferred, wherein the service tasks include release tasks; and the service tasks to be transferred are deployed on the surviving nodes.

[0097] Each node has a metadata service / service task (Metadata Serve, MDS) bound to it. Each MDS service has a recycling (discard) thread for processing release tasks. In addition to the recycling thread, the master node also has a recycling service (discard_server) thread to process related requests / deleted file information sent by each node, as well as distribute release tasks.

[0098] When a node joins the file storage system for the first time, it starts the MDS and obtains a unique number. The node that creates the file storage system is MDS0, the master node. The MDSs of other nodes are numbered sequentially, such as MDS1, MDS2, and so on.

[0099] When a node fails and goes offline, the MDS corresponding to the failed node can be started on another online node. For example, if MDS2 is running on Node 2 and Node 2 goes offline, MDS2 on Node 2, acting as the service task to be transferred, searches for another online node, determines that Node 3 is not faulty and is online, and deploys MDS2 on Node 3. At this point, both MDS2 and MDS3 run on Node 3 simultaneously, ensuring that Node 2 can continue its original work after the failure.

[0100] The space recovery method of the file storage system of the embodiment of the present disclosure is executed by the master node. Specifically, the MDS corresponding to the master node can obtain the node status information and deleted file information, so as to adaptively allocate release tasks to the node and mark the physical space corresponding to the virtual space whose mapping relationship is released as free to complete the space recovery of the file storage system.

[0101] Figure 6 A schematic diagram schematically illustrates a space recovery method for a file storage system according to an embodiment of the present disclosure.

[0102] like Figure 6 As shown, the master node can also perform release tasks. The recycling service thread of the master node MDS0 checks the status of its own release space queue and whether the MDS dicard threads of each node are idle every 50ms to allocate release tasks to the MDS dicard threads of each node.

[0103] In operation S601, the recycling service thread of the master node MDS0 sends a release task to the recycling thread of the master node MDS0; in operation S602, the recycling thread of the master node MDS0 executes the release task, and in operation S603, feeds back the completion result of the release task to the recycling service thread of the master node MDS0. At this time, in operation S604, the recycling service thread of the master node MDS0 deletes the virtual space to be released from the release space queue, and sends a message confirming the completion of the release task to the recycling thread of the master node MDS0 through operation S605; in operation S606, the recycling thread of the master node MDS0 deletes the virtual space to be released of the completed release task from its own recorded virtual space to be released directory entry, and sends the current status information to the recycling service thread of the master node MDS0 through operation S607, so that the recycling service thread of the master node MDS0 can determine whether a new release task can be assigned.

[0104] In operation S608, the recycling service thread of master node MDS0 sends requests to the recycling thread of master node MDS0 and the recycling thread of node MDS1, respectively. Then, in operation S609, the recycling service thread of master node MDS0 obtains status information sent by the recycling thread of master node MDS0 and the recycling thread of node MDS1. In operation S610, the recycling service thread of master node MDS0 analyzes the status information of the recycling thread of master node MDS0 and the recycling thread of node MDS1, respectively, and determines the threshold value of the amount of deleted file information that the recycling thread of master node MDS0 and the recycling thread of node MDS1 can send to the recycling service thread of master node MDS0.

[0105] If the number threshold of the recycling thread of node MDS1 indicates that a certain number of deleted file information can be sent to the recycling service thread of the main node MDS0, operation S611 is executed, the recycling thread of node MDS1 traverses the deleted file information recorded by itself, and sends the deleted file information to the recycling service thread of the main node MDS0 through operation S612; in operation S613, the recycling service thread of the main node MDS0 receives the deleted file information and merges it in the released space queue.

[0106] If the number threshold of the recycling thread of the master node MDS0 is 0, indicating that the deleted file information cannot be sent to the recycling service thread of the master node MDS0, operation S614 is executed, and the recycling thread of the master node MDS0 waits for the next request issued by the recycling service thread of the master node MDS0.

[0107] Based on the above-mentioned space recovery method for a file storage system, the present disclosure further provides a space recovery device for a file storage system. The device will be described in detail below in conjunction with FIG7.

[0108] Figure 7 The structure block diagram of the space recovery device of the file storage system according to the embodiment of the present disclosure is schematically shown.

[0109] like Figure 7 As shown, the space recovery device 700 of the file storage system of this embodiment includes a receiving module 710 , an allocation module 720 and a marking module 730 .

[0110] The receiving module 710 is used to receive deleted file information from each node, and add the virtual space corresponding to the deleted file information as the virtual space to be released to the space queue to be released. In one embodiment, the receiving module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0111] The allocation module 720 is used to periodically query the queue of space to be released. If there is virtual space to be released in the queue of space to be released, an idle node is determined among multiple nodes, and the status information of the idle node is obtained. The number of release tasks to be performed by each idle node is determined based on the status information, and a corresponding number of release tasks are allocated to each idle node, so that each node executes the allocated release tasks, wherein the release tasks correspond one-to-one to the virtual space to be released in the queue of space to be released, and the release tasks include releasing the mapping relationship between the virtual space to be released and the physical space. In one embodiment, the allocation module 720 can be used to perform operations S230 and S240 described above, which will not be repeated here.

[0112] The marking module 730 is used to mark the physical space corresponding to the virtual space whose mapping relationship is released as free. In one embodiment, the marking module 730 can be used to perform the operation S260 described above, which will not be repeated here.

[0113] According to an embodiment of the present disclosure, deleted file information is received from each node, and the virtual space corresponding to the deleted file information is added to the queue of spaces to be released as virtual space to be released; the queue of spaces to be released is periodically queried, and if there is virtual space to be released in the queue of spaces to be released, an idle node is determined among multiple nodes, and the status information of the idle nodes is obtained. The number of release tasks to be performed by each idle node is determined based on the status information, and a corresponding number of release tasks are allocated to each idle node, so that each node executes the allocated release tasks, wherein the release tasks correspond one-to-one to the virtual space to be released in the queue of spaces to be released, and the release tasks include unmapping the mapping relationship between the virtual space to be released and the physical space; and marking the physical space whose mapping relationship is unmapped as idle. Since the release tasks can be adaptively allocated according to the status information of the node, the problem of insufficient resources and task congestion caused by only one node thread processing tasks in the related art is avoided, the utilization rate of all nodes is guaranteed, and the overall processing efficiency of the system can be improved. In addition, since the released tasks correspond one-to-one to the virtual spaces to be released in the queue of spaces to be released, the problem of multiple tasks targeting the same virtual space to be released is avoided. There is no need to use locks to limit the processing of the virtual space to be released, which can reduce the complexity of the system and reduce the system's resource overhead.

[0114] According to embodiments of the present disclosure, any multiple modules among the receiving module 710, the allocation module 720, and the marking module 730 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present disclosure, at least one of the receiving module 710, the allocation module 720, and the marking module 730 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination thereof. Alternatively, at least one of the receiving module 710, the allocation module 720, and the marking module 730 may be at least partially implemented as a computer program module that, when executed, performs the corresponding functionality.

[0115] According to an embodiment of the present disclosure, the status information of a node includes at least one of the CPU occupancy rate and task duration of the node, wherein the task duration is the maximum waiting time length for the current node to execute all allocated release tasks.

[0116] According to an embodiment of the present disclosure, the allocation module 720 includes a determination submodule and a setting submodule.

[0117] The determination submodule is used to determine the number of released tasks to be executed by the idle node according to the CPU occupancy rate of the idle node when the task duration of the idle node is less than the preset duration.

[0118] A submodule is set to stop sending the release task to be executed to the idle node when the task duration of the idle node is greater than the preset duration.

[0119] According to an embodiment of the present disclosure, the space recovery device 700 of the file storage system further includes a merging module.

[0120] The merging module is configured to merge adjacent virtual spaces to be released based on a preset merging threshold when there are adjacent virtual spaces to be released in the queue of spaces to be released, so that the size of the merged virtual spaces to be released is less than or equal to the merging threshold.

[0121] According to an embodiment of the present disclosure, the space recovery device 700 of the file storage system further includes a marking module.

[0122] The marking module is used to mark the to-be-released virtual space corresponding to the uncompleted release task as to-be-allocated when any node goes offline, in response to the node not completing the assigned release task, and add the to-be-released virtual space corresponding to the uncompleted release task to the head of the to-be-released space queue.

[0123] According to an embodiment of the present disclosure, the space recovery device 700 of the file storage system further includes a first determining module and a sending module.

[0124] The first determining module is used to determine the amount of deleted file information to be sent according to the CPU occupancy rate of the node when the task duration of the node is less than a preset duration.

[0125] The setting module is used to set the number of deleted file information sent by the node to a preset maximum number when the task duration of the node is greater than the preset duration.

[0126] According to an embodiment of the present disclosure, the space recovery device 700 of the file storage system further includes a second determination module and a deployment module.

[0127] The second determining module is used to determine, when any node fails, service tasks associated with the failed node as service tasks to be transferred, wherein the service tasks include released tasks.

[0128] The deployment module is used to deploy the service tasks to be transferred on the nodes that are not faulty.

[0129] According to an embodiment of the present disclosure, the physical file space is stored in designated hardware, and multiple nodes interact with the designated hardware through a designated network.

[0130] Figure 8 A block diagram of an electronic device suitable for implementing a space recovery method for a file storage system according to an embodiment of the present disclosure is schematically shown.

[0131] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0132] Various programs and data required for the operation of the electronic device 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0133] According to an embodiment of the present disclosure, electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to bus 804. Electronic device 800 may also include one or more of the following components connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or modem. Communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. Removable media 811, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 810 as needed, so that computer programs read from the removable media can be installed into storage section 808 as needed.

[0134] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0135] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above, and / or one or more memories other than ROM 802 and RAM 803.

[0136] Embodiments of the present disclosure also include a computer program product, comprising a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the space reclamation method for a file storage system provided in an embodiment of the present disclosure.

[0137] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0138] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0139] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the processor 801, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0140] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] Those skilled in the art will appreciate that the features described in the various embodiments of the present disclosure may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of the present disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of the present disclosure. All such combinations and / or couplings fall within the scope of the present disclosure.

[0143] The above describes the embodiments of the present disclosure. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A space recovery method for a file storage system, characterized in that: The file storage system includes multiple nodes, and the method includes: Receive deleted file information from each node, and add the virtual space corresponding to the deleted file information as a virtual space to be released into a queue of spaces to be released; The queue of spaces to be released is periodically queried. If there is virtual space to be released in the queue of spaces to be released, an idle node is determined among the multiple nodes, status information of the idle nodes is obtained, a number of release tasks to be executed by each idle node is determined based on the status information, and a corresponding number of release tasks are assigned to each idle node, so that each idle node executes the assigned release tasks, wherein the release tasks correspond one-to-one to the virtual spaces to be released in the queue of spaces to be released, and the release tasks include releasing a mapping relationship between the virtual spaces to be released and the physical spaces; Mark the unmapped physical space as free.

2. The method according to claim 1, characterized in that The state information of the node includes at least one of a CPU occupancy rate and a task duration of the node, wherein the task duration is a maximum waiting time of a release task to be currently executed by the node.

3. The method according to claim 2, characterized in that The step of determining the number of release tasks to be executed by each idle node according to the state information includes: for each idle node, When the task duration of the idle node is greater than a preset duration, stop sending the release task to be executed to the idle node; In a case where the task duration of the idle node is less than a preset duration, the number of released tasks to be executed by the idle node is determined according to the CPU occupancy rate of the idle node.

4. The method according to claim 1, wherein The method further comprises: When there are adjacent virtual spaces to be released in the to-be-released space queue, the adjacent virtual spaces to be released are merged based on a preset merging threshold, so that the merged virtual spaces to be released are less than or equal to the merging threshold.

5. The method according to claim 1, wherein The method further comprises: When any node goes offline, in response to the offline node not completing the assigned release task, the to-be-released virtual space corresponding to the uncompleted release task is marked as to be allocated, and the to-be-released virtual space corresponding to the uncompleted release task is added to the head of the to-be-released space queue.

6. The method according to claim 2, characterized in that The method further comprises: When the task duration of the node is less than a preset duration, determining the amount of deleted file information to be sent according to the CPU occupancy rate of the node; When the task duration of the node is greater than a preset duration, the number of deleted file information sent by the node is set to a preset maximum number.

7. The method according to claim 1, characterized in that The method further comprises: When any node fails, determining the service tasks of the failed node as service tasks to be transferred, wherein the service tasks include the released tasks; Deploy the service task to be transferred on a non-faulty node.

8. The method according to claim 1, characterized in that The physical file space is stored in designated hardware, and the plurality of nodes interact with the designated hardware via a designated network.

9. The method according to claim 1, characterized in that The multiple nodes include a master node and slave nodes, and the method is executed by the master node.

10. A space recovery device for a file storage system, characterized in that: The file storage system includes multiple nodes, and the device includes: A receiving module, configured to receive deleted file information from each node, and add the virtual space corresponding to the deleted file information as a virtual space to be released to a queue of spaces to be released; an allocation module, configured to periodically query a queue of spaces to be released; if there is virtual space to be released in the queue of spaces to be released, determine an idle node among the plurality of nodes, obtain status information of the idle nodes, determine a number of release tasks to be performed by each idle node based on the status information, and allocate a corresponding number of release tasks to each idle node, so that each idle node performs the allocated release tasks, wherein the release tasks correspond one-to-one to the virtual spaces to be released in the queue of spaces to be released, and the release tasks include undoing the mapping relationship between the virtual spaces to be released and the physical spaces; and The marking module is used to mark the physical space whose mapping relationship is released as idle.

Images