Garbage recycling method and device, electronic equipment and storage medium

Abstract

The invention relates to a garbage collection method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: continuously carrying out heartbeat detection on each node in a node pool, and determining the number of nodes with normal heartbeat; in response to determining that the number is changed, broadcasting a stop instruction to each node in the node pool; for the node which does not reply the specified message within the specified duration, if the heartbeat of the node is normal, a forced termination instruction is sent to the node; in response to determining that the garbage collection process of each node is dormant, determining an updated topological structure according to the node with normal heartbeat; and broadcasting the updated topological structure and a starting instruction to each node with normal heartbeat. By means of the method and device, the situation that double owners appear on a single metadata object can be avoided, and data consistency is guaranteed.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a waste recycling method, apparatus, electronic device, and storage medium. Background Technology

[0002] In distributed object storage systems, when data is deleted or overwritten, the physical space it previously occupied is not immediately released, resulting in "garbage." Garbage collection (GC) is a critical backend service for reclaiming this space, and its efficiency directly impacts the cost and performance of the storage cluster. However, achieving efficient, correct, and seamless GC in ultra-large-scale clusters faces the challenge of split-brain risk. This means that when a network partition or node failure occurs, the cluster may split, causing two GC service instances to simultaneously believe they own the same batch of data blocks and concurrently perform garbage collection operations. This can easily lead to serious data inconsistency incidents, such as accidental deletion of valid data or duplicate garbage collection, and subsequent repair is extremely difficult. Summary of the Invention

[0003] Embodiments of this disclosure provide a waste recycling method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, embodiments of this disclosure provide a garbage collection method, comprising: continuously performing heartbeat detection on each node in a node pool to determine the number of nodes with normal heartbeats; in response to a change in the determined number, broadcasting a stop instruction to each node in the node pool to cause the garbage collection process of each node to hibernate after completing the current garbage scanning task and reply with a specified message; for a node that does not reply with the specified message within a specified time period, if the node's heartbeat is normal, sending a forced termination instruction to the node to terminate the node's garbage collection process; in response to determining that the garbage collection processes of each node are all hibernating, determining an updated topology based on the nodes with normal heartbeats; broadcasting the updated topology and a start instruction to each node with normal heartbeats to cause the garbage collection processes in each node with normal heartbeats to determine the primary node as the metadata object in each node based on the updated topology, and the garbage collection processes in each primary node to perform garbage collection based on each metadata object.

[0005] Secondly, embodiments of this disclosure provide a garbage collection apparatus, comprising: a heartbeat detection unit configured to continuously detect the heartbeat of each node in a node pool to determine the number of nodes with normal heartbeats; an instruction broadcasting unit configured to broadcast a stop instruction to each node in the node pool in response to a change in the determined number, so that the garbage collection process of each node will hibernate after completing the current garbage scanning task and reply with a specified message; a forced termination unit configured to send a forced termination instruction to a node that has not replied with a specified message within a specified time period, if the node has a normal heartbeat, to terminate the garbage collection process of that node; a topology update unit configured to determine an updated topology based on the nodes with normal heartbeats in response to determining that the garbage collection processes of each node are hibernating; and a garbage collection unit configured to broadcast the updated topology and a start instruction to each node with normal heartbeats, so that the garbage collection processes in each node with normal heartbeats will determine the primary node as the metadata object in each node based on the updated topology, and the garbage collection processes in each primary node will perform garbage collection based on each metadata object.

[0006] Thirdly, embodiments of this disclosure provide an electronic device including a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the garbage collection method as described in the first aspect.

[0007] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the garbage collection method as described in the first aspect.

[0008] By applying the technical solution disclosed herein, heartbeat detection can be continuously performed on each node in the node pool to determine the number of nodes with normal heartbeats. If this number changes, a stop command is broadcast to each node in the node pool. Upon receiving the stop command, each node can control its garbage collection process to enter a sleep mode after completing the current garbage scan task, while simultaneously replying with a specified message. For nodes that do not reply with the specified message within a specified time, it is determined whether the node's heartbeat is normal. If the heartbeat is normal, a forced termination command is sent to that node. Upon receiving the forced termination command, the node terminates its garbage collection process. If it is determined that the garbage collection processes of all nodes are in a sleep state, an updated topology is determined based on the nodes with normal heartbeats. The topology and start command are then broadcast to each node with normal heartbeats. In this way, the garbage collection processes in the nodes with normal heartbeats can determine the master nodes for the metadata objects in each node based on the updated topology, and the garbage collection processes in each master node perform garbage collection on each metadata object. Through the technical solution disclosed herein, the situation of dual masters for a single metadata object can be avoided, ensuring data consistency.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 An exemplary system architecture diagram in which the waste recycling method of this disclosure can be applied is shown;

[0012] Figure 2 This is a schematic flowchart of one embodiment of the waste recycling method disclosed herein;

[0013] Figure 3 This is a schematic flowchart of another embodiment of the waste recycling method disclosed herein;

[0014] Figure 4 This is a schematic flowchart of yet another embodiment of the waste recycling method disclosed herein;

[0015] Figure 5 This is a schematic diagram of the structure of one embodiment of the waste recycling device disclosed herein;

[0016] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device disclosed herein. Detailed Implementation

[0017] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0018] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0019] Where there is no conflict, the embodiments and features described herein can be combined with each other.

[0020] To make the technical solutions and advantages of this disclosure clearer, the following description, in conjunction with the accompanying drawings and specific embodiments, will provide a more detailed account of this disclosure.

[0021] Figure 1 An exemplary system architecture 100 is shown that can be applied to embodiments of the waste recycling methods or waste recycling apparatus disclosed herein.

[0022] like Figure 1 As shown, the system architecture 100 may include a central coordinator 101, a node pool 102, and a distributed object storage pool 103. The central coordinator 101, node pool 102, and distributed object storage pool 103 are connected via network communication. The network may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0023] The central coordinator 101 can continuously perform heartbeat checks on each node in the node pool 102 to determine if each node's heartbeat is normal. It also continuously maintains the topology of each node in the node pool 102. When a topology update is detected, a stop command can be sent to each node in the node pool 102. After the garbage collection processes in each node have all gone into hibernation, the updated topology can be broadcast back to the nodes.

[0024] Each node in node pool 102 can store metadata objects. The data in these metadata objects can be the metadata of data objects stored in distributed object storage pool 103. One data object can correspond to one metadata object. The metadata can include a garbage tagging bitmap. Each node can run a garbage collection process, which can scan the garbage tagging bitmap in each metadata object to determine the garbage rate of the data object and perform garbage collection based on the garbage rate.

[0025] Each node in the distributed object storage pool 103 can store data objects, which can include various types of data. Data objects are in an "on" state during normal writing and in a "off" state when the pool is full.

[0026] It should be noted that the waste recycling method provided in this embodiment is generally executed by the central coordinator 101. Accordingly, the waste recycling device is generally located in the central coordinator 101.

[0027] It should be understood that Figure 1 The number of central coordinators, node pools, and distributed object storage pools shown is merely illustrative. Depending on implementation needs, any number of central coordinators, node pools, and distributed object storage pools can be used.

[0028] Figure 2 A flow 200 of one embodiment of the waste recycling method of this disclosure is shown. For example... Figure 2 As shown, the waste recycling method in this embodiment may include the following steps:

[0029] Step 201: Continuously perform heartbeat detection on each node in the node pool to determine the number of nodes with normal heartbeats.

[0030] In this embodiment, the entity executing the waste recycling method (e.g.) Figure 1 The central coordinator 101 shown can continuously perform heartbeat checks on each node in the node pool to determine the number of nodes with normal heartbeats. Specifically, the executing entity can send a liveness detection message to each node in the node pool every preset time interval (e.g., 1 second). Here, the liveness detection message can be information with preset fields. After receiving the liveness detection message, each node can reply with a preset message to the executing entity within the preset time interval. In this way, the executing entity can determine that the node's heartbeat is normal. The executing entity can count the number of nodes with normal heartbeats in the node pool.

[0031] Step 202: In response to the determination that the above quantity has changed, a stop command is broadcast to each node in the node pool so that the garbage collection process of each node will hibernate after completing the current garbage scanning task and reply with a specified message.

[0032] If the executing entity detects a change in the aforementioned quantities—that is, a node in the node pool crashes or a new node is added—it can broadcast a stop command to all nodes in the pool. Upon receiving the stop command, each node in the pool can control its garbage collection process to enter hibernation mode after completing its current garbage scan task, and simultaneously reply to the executing entity with a specified message. In Linux systems, processes may enter hibernation mode (such as S3 / S4 sleep states), where system resource usage is extremely low, but the process can still be woken up by external events. By replying with a specified message, the executing entity can be explicitly informed that the node's garbage collection process is hibernating.

[0033] Step 203: For nodes that do not respond to the specified message within the specified time, if the node's heartbeat is normal, send a forced termination command to the node to terminate the node's garbage collection process.

[0034] If a node fails to respond to a specified message within a specified time period, the executing entity can further determine whether these nodes are functioning normally. If a node's heartbeat is still normal, it is considered to be still online, but currently in an unresponsive state (process dead). In this case, a forced termination command can be sent to that node to terminate its garbage collection process.

[0035] Step 204: In response to the determination that the garbage collection process of each node is in a dormant state, determine the updated topology based on the nodes with normal heartbeats.

[0036] After confirming that the garbage collection process on all nodes is inactive, the updated topology can be determined based on the nodes with normal heartbeats. Specifically, the updated topology can be determined based on the connections and / or locations between the nodes.

[0037] Step 205: Broadcast the updated topology and start command to each node with a normal heartbeat, so that the garbage collection process in each node with a normal heartbeat can determine the primary node as the metadata object in each node according to the updated topology, and the garbage collection process in each primary node can perform garbage collection based on each metadata object.

[0038] After the updated topology is determined, it can be broadcast to all nodes with normal heartbeats. Simultaneously, a start command is sent to each node. Upon receiving the updated topology, each node can determine a new primary node for each metadata object based on the structure and the names of the metadata objects. Then, the garbage collection process on the new primary node performs garbage collection based on the metadata objects. It is understandable that a single node can store multiple metadata objects, and different metadata objects can correspond to different primary nodes.

[0039] Specifically, the garbage collection process in the new master node can scan the garbage marker bitmap in the metadata object to determine the garbage rate of the data object corresponding to the metadata object. If the garbage rate is greater than a preset threshold, garbage collection can be performed. During garbage collection, valid data in the data object can be moved to an empty disk, and then garbage data can be deleted.

[0040] The garbage collection method provided in the above embodiments of this disclosure can continuously perform heartbeat detection on each node in the node pool to determine the number of nodes with normal heartbeats. If the number changes, a stop command is broadcast to each node in the node pool. After receiving the stop command, each node can control its garbage collection process to enter a sleep mode after completing the current garbage scanning task, and simultaneously reply with a specified message. For nodes that do not reply with the specified message within a specified time, it is determined whether the node's heartbeat is normal. If the heartbeat is normal, a forced termination command is sent to the node. After receiving the forced termination command, the node terminates its garbage collection process. If it is determined that the garbage collection processes of all nodes are in a sleep state, an updated topology is determined based on the nodes with normal heartbeats. The topology and start command are broadcast to each node with normal heartbeats. In this way, the garbage collection processes in the nodes with normal heartbeats can determine the master nodes for the metadata objects in each node based on the updated topology, and the garbage collection processes in each master node perform garbage collection on each metadata object. Through the technical solution of this disclosure, the situation of dual masters for a single metadata object can be avoided, ensuring data consistency.

[0041] See also Figure 3This illustrates a process 300 of another embodiment of the waste recycling method according to the present disclosure. Figure 3 As shown, the method in this embodiment may include the following steps:

[0042] Step 301: Continuously perform heartbeat detection on each node in the node pool to determine the number of nodes with normal heartbeats.

[0043] Step 302: In response to determining that the quantity has changed, a stop command is broadcast to each node in the node pool so that the garbage collection process of each node will hibernate after completing the current garbage scanning task and reply with a specified message.

[0044] Step 303: For a node that does not reply to the specified message within a specified time period, if the node's heartbeat is normal, send a forced termination command to the node to terminate the node's garbage collection process.

[0045] Step 304: In response to the determination that the garbage collection process of each node is in a dormant state, determine the updated topology based on the nodes with normal heartbeats.

[0046] Step 305: Broadcast the updated topology and start command to each node with a normal heartbeat, so that the garbage collection process in each node with a normal heartbeat can determine the primary node as the metadata object in each node according to the updated topology, and the garbage collection process in each primary node can perform garbage collection based on each metadata object.

[0047] Step 306: In response to determining that the nodes that did not reply to the specified message include the primary node, determine the scan breakpoint of the garbage collection process of the primary node for each metadata object, and delete the primary node.

[0048] In this embodiment, if the node that does not respond to the specified message includes the primary node, i.e., the primary node process is dead, the garbage collection process of the primary node can first be identified and the breakpoints of the scan of each metadata object can be recorded. Then, the primary node can be deleted.

[0049] Step 307: For each metadata object, in response to the update of the primary node of the metadata object, a start command and a scan breakpoint are sent to the updated primary node so that the garbage collection process in the updated primary node continues to perform garbage scanning from the scan breakpoint.

[0050] For a single metadata object, if the primary node of that metadata object changes after a node topology update, the executing entity can send a start command and the aforementioned scan breakpoint to the new primary node of that metadata object. In this way, the garbage collection process in the new primary node can continue garbage scanning from the scan breakpoint.

[0051] In some optional implementations of this embodiment, if the new primary node is not a replica node of the metadata object, the data in the original primary node can be moved to the new primary node before sending the start command.

[0052] The following example will further illustrate this embodiment using a specific application scenario:

[0053] Initial state: The metadata object Meta_X has three copies located on nodes A, B, and C. Based on the object name and topology, node A is determined to be the "master" node, and is currently being scanned and analyzed by the garbage collection process on it.

[0054] Fault Occurrence: Node A suddenly crashed due to a hardware failure. The central coordinator detected the loss of connection to Node A within 3 seconds via heartbeat detection.

[0055] Stop broadcast: The coordinator immediately broadcasts a stop command to all surviving nodes (B, C, D, E...).

[0056] Waiting and Confirmation: Nodes B, C, and other services quickly stop their current work and send a confirmation.

[0057] Fault tolerance: The coordinator does not wait for a response from node A (because it has failed), but removes it directly from the waiting list.

[0058] New Map Release and Switchover: After the central coordinator collects confirmations from all healthy nodes, it releases a new topology (marking node A as invalid). Based on the object name and topology of Meta_X, it recalculates and finds that node B is the new highest priority replica holder, becoming the new primary node of Meta_X.

[0059] Second-level recovery: The garbage collection process on node B starts immediately, directly reading the existing Meta_X copy on the local disk and continuing the scanning or garbage collection task from the scan breakpoint. The entire process does not require copying any metadata from other nodes, achieving seamless failover in seconds.

[0060] The following example illustrates this embodiment further using another specific application scenario:

[0061] Before expansion: The cluster has 4 nodes (Node1-4), and data and services are evenly distributed.

[0062] Adding new nodes: The administrator adds two new nodes to the cluster: Node5 and Node6.

[0063] Underlying data rebalancing: The disk selection algorithm of the underlying distributed storage begins to work, aiming to rebalance the data across the six nodes while minimizing the amount of data migrated. The algorithm selects a portion of metadata objects (such as Meta_Y) and migrates their replicas from the original nodes (such as Node1) to the new nodes (such as Node5). This process is transparent to the upper-layer services.

[0064] Topology Update: The central coordinator generates and prepares to activate a new topology (V2) containing Node5 and Node6.

[0065] Smooth switchover: The coordinator broadcasts a stop command, and services such as garbage collection on all nodes stop after completing their current tasks. The coordinator then activates the new topology and broadcasts a start command.

[0066] A new equilibrium is established: all nodes recalculate their master-slave relationships based on the new Map V2. For example, Meta_Y's highest-priority replica is now on Node5, so Node5 becomes its new master node and starts the relevant GC services. Some of the GC load is automatically and smoothly distributed to the new nodes, and the entire system enters a new, larger equilibrium state after a short pause.

[0067] The garbage collection method provided in the above embodiments of this disclosure allows the new primary node to immediately start the garbage collection task without any data synchronization when the original primary node fails, since a complete copy of the metadata objects already exists on its local disk. The bottleneck in recovery is only the start-up and shutdown time of lightweight service processes.

[0068] See also Figure 4 This illustrates a process 400 of yet another embodiment of the waste recycling method according to the present disclosure. Figure 4 As shown, the method in this embodiment, in Figure 2 In addition to the steps in the illustrated embodiments, the following steps may also be included:

[0069] Step 401: Obtain the status and creation time of each data object.

[0070] In this embodiment, the executing entity can also obtain the status and creation time of each data object. Here, the status of a data object can include "on" and "off". When a data object is not full, its status is "on". When a data object is full, its status is set to "off". The creation time can be the creation time of the data object. Specifically, upon receiving a data write request, the target data in the data write request can be determined, and the data object to be written to the target data can be determined. When determining the data object, a disk selection algorithm can be used to ensure the balance of writes. This ensures that the distribution of metadata objects is also balanced, and the load on the subsequent garbage collection service is also balanced.

[0071] Step 402: For a single data object, in response to determining that the state of the data object is on, determine whether the duration of the state maintenance of the data object is greater than a first preset duration.

[0072] For each data object, if its state is "on", it can be further determined whether the duration of the state maintenance is greater than a first preset duration. If the duration of the state maintenance is greater than the first preset duration, the write operation to the data object is considered to have timed out.

[0073] Step 403: In response to determining that the state maintenance duration of the data object is greater than the first preset duration, the data writing process is controlled to set the state of the data object to off.

[0074] If the write operation to the data object times out, the foreground data write process can be controlled to set the data object's state to "off". Specifically, this can be done by changing the field representing the state of the data object to a preset value indicating "off".

[0075] Step 404: In response to the determination that the data writing process failed to change the state of the data object, determine whether the state maintenance time of the data object is greater than the second preset time.

[0076] In this embodiment, if the foreground data writing process fails to change the state of the data object, the possible reason is that the data object is still in the writing state. To ensure the integrity of the data writing, it can continue to wait until the state of the data object is maintained for a duration longer than a second preset duration. It is understood that the second preset duration is longer than the first preset duration.

[0077] Step 405: In response to determining that the state maintenance duration of the data object is greater than the second preset duration, control the garbage collection process to set the state of the data object to off.

[0078] If the state of the data object is maintained for a duration longer than a second preset duration, the background garbage collection process can be further controlled to set the state of the data object to "off". Once the state of the data object is off, the garbage collection process can immediately begin scanning the garbage marker bitmap of the data object to determine the garbage rate of the data object for garbage collection.

[0079] In some optional implementations of this embodiment, in order to ensure the consistency of the data object state settings, both the foreground data writing process and the background garbage collection process need to use a strong consistency interface when changing the state of the data object.

[0080] The following example will further illustrate this embodiment using a specific application scenario:

[0081] An error occurred: An operation log object OLOG_99 failed to complete the write process normally due to a client error or a brief node failure. Its state was stuck in the open state, and its garbage marker bitmap was also not updated.

[0082] Foreground daemon intervention (first layer of protection): The system maintains a global olog state manager, continuously monitoring all open-state operation log objects. If OLOG_99 has not been converted to a closed state for 24 hours since its creation, the foreground daemon will determine that it has timed out. This daemon will forcibly change the state of OLOG_99 to closed and remove it from the write queue. This operation is completed by calling the underlying distributed storage's strong consistency interface, ensuring state consistency within the cluster.

[0083] Background GC service intervention (secondary protection): Even if the foreground daemon fails to force shutdown (e.g., due to its own failure), the background garbage collection process provides redundancy. When periodically scanning local metadata objects, the garbage collection process checks the creation time of each open-state olog. If it finds that OLOG_99 was created more than 48 hours ago (this threshold is greater than the foreground threshold), the GC service will directly set the status field in its metadata object to closed.

[0084] Space reclamation: Once the state of OLOG_99 is closed by any mechanism, it will immediately enter the scanning field of the garbage collection process. The garbage collection process analyzes its garbage bitmap (which may be all 0s, i.e., no valid data, or partially updated due to cache loss) and decides, based on the strategy, whether to release all its space directly or release it after moving out valid data.

[0085] In some optional implementations of this embodiment, for a single data object, in response to receiving a deletion instruction for data in the data object, the garbage marker bitmap of the data object is updated in memory; in response to determining that the state corresponding to the data is off, the metadata object corresponding to the data object is updated according to the garbage marker bitmap.

[0086] In this implementation, for a single data object, if a deletion instruction is received for data within that data object, only the garbage marker bitmap of that data object needs to be updated in memory. This process continues until the corresponding state of the data is "off," at which point the garbage marker bitmap is then updated to the corresponding metadata object. This significantly reduces the number of lowercase metadata operations, fundamentally solving the metadata write amplification problem. The garbage collection process only iteratively scans the metadata objects stored locally on its node. It does not need to query remote metadata objects over the network, thus achieving extremely high processing performance and decoupled scalability.

[0087] The following is a specific application scenario to further illustrate this implementation method:

[0088] Data writing: The client requests to write a data block Data_A.

[0089] Allocate olog: The system allocates a new operation log object OLOG_100 and sets its state to open. At the same time, a corresponding metadata object Meta_OLOG_100 is created in the distributed storage pool, initially in the open state, to record GC information such as its garbage marking bitmap.

[0090] Data persistence: Based on the disk selection algorithm, if OLOG_100 is determined to be the write object of Data_A, then Data_A will be written to OLOG_100.

[0091] Data deletion / overwrite: In subsequent operations, Data_A is deleted by the client. The system does not immediately modify Meta_OLOG_100 on the disk, but instead marks the bit corresponding to Data_A in the garbage mark bitmap of OLOG_100 in the memory of its primary node and replica nodes (assuming the primary node is node A) as 1 (indicating garbage).

[0092] Closing olog: When OLOG_100 is full, the system sets its state to closed. At this time, the GC service of node A will batch write all the garbage marks belonging to OLOG_100 that are cached in memory to the Meta_OLOG_100 object on the local disk, thus completing the persistence of metadata.

[0093] Garbage collection: The garbage collection process of node A iteratively scans the local metadata objects. When it scans the Meta_OLOG_100 object with the status of closed, it reads its garbage bitmap and calculates that the garbage rate is 70% (higher than the threshold).

[0094] Data migration and space release: The garbage collection process reads the remaining valid data (unmarked blocks) from OLOG_100 and writes it to a new OLOG_200. Then, it calls the distributed storage interface to release the physical space occupied by the entire OLOG_100. This completes one garbage collection cycle.

[0095] In the garbage collection method provided by the above embodiments of this disclosure, the cooperation between the front-end data writing process and the background garbage collection process ensures that even if a component temporarily fails, the suspended data objects will eventually be identified and cleaned up, fundamentally avoiding permanent leakage of storage space. Furthermore, both front-end and back-end modifications to the state are performed through a strongly consistent interface, guaranteeing the atomicity of operations and avoiding state chaos caused by concurrent modifications. The entire repair process is completely transparent to the front-end business and is performed automatically.

[0096] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a waste recycling device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0097] like Figure 5 As shown, the waste recycling device 500 in this embodiment includes: a heartbeat detection unit 501, an instruction broadcasting unit 502, a forced termination unit 503, a topology update unit 504, and a waste recycling unit 505.

[0098] The heartbeat detection unit 501 is configured to continuously perform heartbeat detection on each node in the node pool to determine the number of nodes with normal heartbeats.

[0099] The instruction broadcasting unit 502 is configured to broadcast a stop instruction to each node in the node pool in response to a change in a certain number, so that the garbage collection process of each node will hibernate after completing the current garbage scan task and reply with a specified message.

[0100] The forced termination unit 503 is configured to send a forced termination command to a node that has not responded to a specified message within a specified time period, if the node's heartbeat is normal, in order to terminate the node's garbage collection process.

[0101] Topology update unit 504 is configured to determine the updated topology based on the nodes with normal heartbeats in response to determining that the garbage collection process of each node is in a dormant state.

[0102] Garbage collection unit 505 is configured to broadcast an updated topology and start command to each node with a normal heartbeat, so that the garbage collection process in each node with a normal heartbeat can determine the primary node as the metadata object in each node according to the updated topology, and the garbage collection process in each primary node can perform garbage collection based on each metadata object.

[0103] In addition, an electronic device is also proposed in the technical solution of this application.

[0104] Figure 6A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure is shown.

[0105] like Figure 6 As shown, the electronic device may include a processor 601, a memory 602, a bus 603, and a computer program stored on the memory 602 and executable on the processor 601, wherein the processor 601 and the memory 602 communicate with each other via the bus 603. When the processor 601 executes the computer program, it implements the steps of the above method, including, for example: continuously performing heartbeat detection on each node in the node pool to determine the number of nodes with normal heartbeats; in response to a change in the determined number, broadcasting a stop instruction to each node in the node pool so that the garbage collection process of each node will hibernate after completing the current garbage scanning task and reply with a specified message; for a node that does not reply with the specified message within a specified time, if the node's heartbeat is normal, sending a forced termination instruction to the node to terminate the node's garbage collection process; in response to determining that the garbage collection processes of each node are all hibernating, determining an updated topology based on the nodes with normal heartbeats; broadcasting the updated topology and a start instruction to each node with normal heartbeats so that the garbage collection processes in each node with normal heartbeats determine the primary node as the metadata object in each node based on the updated topology, and the garbage collection processes in each primary node perform garbage collection based on each metadata object.

[0106] In addition, one embodiment of this disclosure also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the above-described method, including, for example, continuously performing heartbeat detection on each node in a node pool to determine the number of nodes with normal heartbeats; in response to a change in the determined number, broadcasting a stop instruction to each node in the node pool to cause the garbage collection process of each node to hibernate after completing the current garbage scanning task and reply with a specified message; for a node that does not reply with the specified message within a specified time period, if the node's heartbeat is normal, sending a forced termination instruction to the node to terminate the node's garbage collection process; in response to determining that the garbage collection processes of each node are all hibernating, determining an updated topology based on the nodes with normal heartbeats; broadcasting the updated topology and a start instruction to each node with normal heartbeats to cause the garbage collection processes in each node with normal heartbeats to determine the primary node as the metadata object in each node based on the updated topology, and the garbage collection processes in each primary node to perform garbage collection based on each metadata object.

[0107] In summary, the technical solution disclosed herein continuously monitors the heartbeat of each node in the node pool to determine the number of nodes with normal heartbeats. If this number changes, a stop command is broadcast to all nodes in the node pool. Upon receiving the stop command, each node can control its garbage collection process to enter a sleep mode after completing the current garbage scan task, while simultaneously replying with a specified message. For nodes that do not reply with the specified message within a specified time, it is determined whether the node's heartbeat is normal. If the heartbeat is normal, a forced termination command is sent to that node. Upon receiving the forced termination command, the node terminates its garbage collection process. If it is determined that all nodes' garbage collection processes are in a sleep state, an updated topology is determined based on the nodes with normal heartbeats. The topology and start command are broadcast to all nodes with normal heartbeats. In this way, the garbage collection processes in the nodes with normal heartbeats can determine the master nodes for the metadata objects in each node based on the updated topology, and the garbage collection processes in each master node perform garbage collection on each metadata object. Through the technical solution disclosed herein, the dual-master situation for a single metadata object can be avoided, ensuring data consistency.

[0108] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A waste recycling method, comprising: Continuously perform heartbeat checks on each node in the node pool to determine the number of nodes with normal heartbeats; In response to determining that the quantity has changed, a stop command is broadcast to each node in the node pool so that the garbage collection process of each node will hibernate after completing the current garbage scanning task and reply with a specified message; For nodes that do not respond to the specified message within the specified time period, if the node's heartbeat is normal, a forced termination command is sent to the node to terminate the node's garbage collection process. In response to the determination that the garbage collection process of each node is in a dormant state, the updated topology is determined based on the nodes with normal heartbeats. The updated topology and start command are broadcast to each node with a normal heartbeat, so that the garbage collection process in each node with a normal heartbeat can determine the primary node as the metadata object in each node according to the updated topology, and the garbage collection process in each primary node performs garbage collection based on each metadata object.

2. The method according to claim 1, wherein, The method further includes: In response to the determination that a node that has not responded to a specified message includes the primary node, determine the scan breakpoint of the garbage collection process of the primary node for each metadata object, and delete the primary node; For each metadata object, in response to the update of the primary node of the metadata object, the start command and the scan breakpoint are sent to the updated primary node so that the garbage collection process in the updated primary node continues to perform garbage scanning from the scan breakpoint.

3. The method according to claim 2, wherein, The method further includes: In response to the determination that the updated master node is not a copy of the metadata object, the data of the metadata object in the original master node is moved to the updated master node.

4. The method according to claim 1, wherein, The method further includes: Get the status and creation time of each data object; For a single data object, in response to determining that the state of the data object is on, it is determined whether the duration of the state of the data object is maintained is greater than a first preset duration; In response to determining that the state of the data object is maintained for a duration longer than the first preset duration, the data writing process is controlled to set the state of the data object to off.

5. The method according to claim 4, wherein, The method further includes: In response to determining that the data writing process has failed to change the state of the data object, determine whether the state maintenance duration of the data object is greater than a second preset duration; In response to determining that the state of the data object is maintained for a duration longer than the second preset duration, the garbage collection process is controlled to set the state of the data object to off.

6. The method according to claim 5, wherein, The method further includes: Both the data writing process and the garbage collection process modify the state process settings of the data object through a strong consistency interface.

7. The method according to claim 5, wherein, The method further includes: For a single data object, in response to receiving a deletion instruction for data in that data object, the garbage marker bitmap of that data object is updated in memory; In response to determining that the state corresponding to the data is off, the metadata object corresponding to the data object is updated according to the garbage tag bitmap.

8. A waste recycling device, comprising: The heartbeat detection unit is configured to continuously detect the heartbeat of each node in the node pool to determine the number of nodes with normal heartbeats. The instruction broadcasting unit is configured to broadcast a stop instruction to each node in the node pool in response to determining that the quantity has changed, so that the garbage collection process of each node will hibernate after completing the current garbage scanning task and reply with a specified message; The forced termination unit is configured to send a forced termination command to a node that has not responded to the specified message within a specified time period, if the node's heartbeat is normal, to terminate the node's garbage collection process. The topology update unit is configured to determine the updated topology based on the nodes with normal heartbeats in response to the determination that the garbage collection process of each node is in a dormant state. The garbage collection unit is configured to broadcast the updated topology and start instructions to each node with a normal heartbeat, so that the garbage collection process in each node with a normal heartbeat can determine the primary node as the metadata object in each node according to the updated topology, and the garbage collection process in each primary node can perform garbage collection based on each metadata object.

9. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the garbage collection method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the garbage collection method as described in any one of claims 1 to 7.

Images