A GC mechanism processing method, device and medium

Abstract

The application discloses a GC mechanism processing method and device and medium, and is suitable for the technical field of solid state disks. When the RGW service is started, a GC main thread, a queue and a work thread are created, wherein the GC main thread starts the GC mechanism; garbage data obtained by the GC main thread traversing a storage pool is stored in the queue; and the queue sends the garbage data to the work thread in a de-queue mode so that the work thread deletes the garbage data. The method uses the thread division mode, the GC main thread starts the GC mechanism, when the GC mechanism is started, garbage collection is processed, the main thread is responsible for traversing the garbage data, and the work thread deletes the garbage data, thereby effectively improving the processing speed of the GC mechanism, relieving the pressure of only executing garbage collection through a single thread at present, and guaranteeing the stability of the system. Meanwhile, the multiple threads execute the deletion of the garbage data, high concurrency is realized, and available disk space is released.

Description

Technical Field

[0001] This invention relates to the field of solid-state drive technology, and in particular to a GC mechanism processing method, apparatus and medium. Background Technology

[0002] Object storage is a novel network storage architecture that combines the high-speed direct disk access of Storage Area Network (SAN) with the distributed sharing characteristics of Network Attached Storage (NAS), offering broad application prospects and value. As a distributed storage solution, its application scenarios primarily involve large-scale data storage. Customers typically store tens of millions or even hundreds of millions of object data items within storage clusters, while simultaneously performing numerous read, write, and delete operations. This process generates a significant amount of junk data, including tail object data larger than 4MB resulting from deletion operations, and overwriting existing object data during upload. Given such massive datasets and the substantial amount of junk data generated during operations, rapidly deleting junk data and reclaiming available disk space without severely impacting cluster performance is crucial for cluster availability and user experience.

[0003] Currently, the garbage collection (GC) mechanism in object storage works as follows: When the object storage service (Rados Gateway, RGW) starts, the GC starts a single thread to traverse the tail of the linked list stored in the key-value structure of the Rados objects, and then deletes the objects. While the implementation is simple and easy to understand, the two-level traversal results in significant time consumption. Furthermore, the current GC mechanism relies on a single thread, which is responsible for traversing all Rados objects and performing deletion operations. This process can easily lead to data accumulation. Additionally, if the user's disk write speed exceeds the GC mechanism's disk space reclamation speed, the user's disk space may reach its maximum availability due to excessive garbage data, impacting user experience.

[0004] Therefore, how to avoid data accumulation in order to improve the processing speed of the GC mechanism is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a GC mechanism processing method, apparatus and medium to avoid data accumulation and improve the speed of GC mechanism processing.

[0006] To address the aforementioned technical problems, this invention provides a GC mechanism processing method, comprising:

[0007] When the RGW service starts, it creates a GC main thread, a queue, and worker threads, with the GC main thread enabling the GC mechanism.

[0008] Garbage data obtained by the GC main thread traversing the storage pool is stored in a queue;

[0009] The queue is controlled by dequeuing to send garbage data to worker threads so that worker threads can delete the garbage data.

[0010] Preferably, the storage pool is established in the following way:

[0011] When a client of the RGW service performs a deletion operation, the object data generated by the deletion operation is deleted to obtain garbage data, which includes object header data and object tail data.

[0012] Garbage data is stored in a linked list and combined with the object name corresponding to the garbage data into a key-value structure and stored in the storage pool.

[0013] Correspondingly, object data is deleted to obtain garbage data, including:

[0014] Delete the object header data, and keep the object tail data as garbage data.

[0015] Preferably, when there are multiple worker threads, garbage data is sent to the worker threads by dequeuing, including:

[0016] The control queue sends garbage data to the worker threads in a polling manner.

[0017] Preferably, the control queue sends garbage data to the worker threads via a polling method, including:

[0018] Get the current working status of the worker thread, which includes the idle state and the data deletion state;

[0019] When the current worker thread is in the data deletion state, the working state of the next current worker thread is obtained until the current worker thread is in the idle state. Then, the garbage data is sent to the current worker thread corresponding to the idle state for deletion.

[0020] Preferably, when all current worker threads are in the data deletion state, the method further includes:

[0021] Set up heartbeat data to monitor worker threads;

[0022] When a change in the working state of a worker thread is detected, garbage data is sent to the worker thread whose working state has changed for deletion.

[0023] Preferably, it further includes:

[0024] When creating a storage pool, create garbage data records;

[0025] After the worker thread deletes garbage data, it controls the worker thread to generate a completion message and send the completion message to the GC main thread;

[0026] When the number of completed messages received reaches a threshold, garbage data records in the storage pool are pruned.

[0027] Preferably, creating a worker thread includes:

[0028] Create worker threads using a vector container.

[0029] To address the aforementioned technical problems, the present invention also provides a GC mechanism processing device, comprising:

[0030] Create a module to create the GC main thread, queues, and worker threads when the RGW service starts, with the GC main thread enabling the GC mechanism;

[0031] The storage module is used to store garbage data obtained by the GC main thread traversing the storage pool into a queue.

[0032] The sending module is used to control the queue to send garbage data to worker threads via dequeueing so that worker threads can delete the garbage data.

[0033] To address the aforementioned technical problems, the present invention also provides a GC mechanism processing device, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor is used to implement the GC mechanism processing methods described above when executing computer programs.

[0036] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the GC mechanism processing method described above.

[0037] This invention provides a GC mechanism processing method, which includes creating a GC main thread, a queue, and worker threads when the RGW service starts. The GC main thread initiates the GC mechanism; garbage data obtained by the GC main thread traversing the storage pool is stored in the queue; and the queue is controlled to send garbage data to worker threads for deletion via dequeueing. This method, through thread division of labor, effectively improves the processing speed of the GC mechanism by having the GC main thread initiate the GC mechanism and perform garbage collection when it is active, while the main thread is responsible for traversing the garbage data and the worker threads perform garbage data deletion. This alleviates the pressure of single-threaded garbage collection and ensures system stability. Simultaneously, multiple threads perform garbage data deletion, achieving high concurrency and freeing up available disk space.

[0038] In addition, the present invention also provides a GC mechanism processing device and medium, which have the same beneficial effects as the GC mechanism processing method described above. Attached Figure Description

[0039] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart of a GC mechanism processing method provided in an embodiment of the present invention;

[0041] Figure 2 This is a structural diagram of a GC mechanism processing device provided in an embodiment of the present invention;

[0042] Figure 3 This is a structural diagram of another GC mechanism processing device provided in an embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0044] The core of this invention is to provide a GC mechanism processing method, device, and medium to avoid data accumulation and improve the speed of GC mechanism processing.

[0045] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] It should be noted that the GC mechanism processing method provided by this invention is applicable to disk space reclamation operations in object storage, mainly involving some deletion operations or other garbage data generated during data clearing, and plays an important role in reclaiming available disk space.

[0047] Figure 1 A flowchart of a GC mechanism processing method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0048] S11: When the RGW service starts, it creates the GC main thread, queues, and worker threads, with the GC main thread enabling the GC mechanism.

[0049] S12: Garbage data obtained by the GC main thread traversing the storage pool is stored in the queue;

[0050] S13: Control the queue to send garbage data to worker threads by dequeuing so that worker threads can delete the garbage data.

[0051] Specifically, object storage describes a method for solving and processing discrete units. Objects exist within a single layer without further hierarchical structures and are characterized by extended metadata. It provides object-based data storage services on a distributed system platform, accessing data through a Representational State Transfer Application Programming Interface (RESTful API) for data read / write operations and a rich set of Software Development Kits (SDKs). Object storage features unstructured storage, multiple nodes, a flat structure, and elastic scaling. Its purpose is to store unstructured data such as documents, images, and videos, facilitating data access.

[0052] In real-world projects, memory overflow and memory leaks occur from time to time, which are unavoidable bugs. These potential bugs can sometimes affect the normal operation of a project. If your project does not allocate business memory reasonably, it will directly affect the project's concurrent processing. When garbage collection becomes a bottleneck for the system to achieve higher concurrency, we need to implement necessary monitoring and adjustment of these "automated" technologies. Understanding the GC implementation mechanism is the prerequisite for all our monitoring and adjustment.

[0053] Countless objects exist in a vast amount of memory. Before garbage collection (GC), these objects need to be accurately marked and categorized into live objects and garbage objects. If any object is missed during this process, it will have to wait for the next GC marking and collection, which will impact GC efficiency. Furthermore, it is crucial to avoid incorrect marking, such as marking perfectly live objects as garbage. Collecting perfectly live objects could cause various program crashes.

[0054] Currently, there are two algorithms for marking objects: reference counting and reachability analysis. Reference counting allocates a field in the object header to store the object's reference count. Once the object is referenced by another object, the count is incremented by 1. If the reference becomes invalid, the count is decremented by 1. When the reference count is 0, it means the object is no longer referenced and can be garbage collected. It only requires checking the count, making it relatively simple and efficient. However, it has a serious problem: it cannot solve the problem of circular references. Reachability analysis, on the other hand, first needs to find currently active references according to rules, calling them GC Roots. Then, starting from the GC Roots as root nodes, it traverses the object reference graph, marking traversable (reachable) objects as live and treating the rest as useless. This invention does not specifically limit the method used for marking data, as long as it can traverse the garbage data and delete it through worker threads.

[0055] When the RGW service starts, a GC main thread is created, which is used to traverse garbage data. At the same time, worker threads and queues are initialized and stored. The queues store the garbage data traversed by the GC main thread for users.

[0056] It's important to note that enabling the GC mechanism on the main GC thread is essentially controlling its on / off switch. This changes the previous approach where only the main GC thread traversed and deleted garbage data. Instead, the main GC thread traversed the garbage data in the storage pool, while worker threads performed the deletion. This multi-threaded processing doesn't involve the main thread and worker threads simultaneously and equally distributing the garbage data in the storage pool and then deleting it individually. Such an approach would increase the access interfaces of the underlying logic. Furthermore, the uneven distribution of garbage data in the storage pool could lead to one thread frequently traversing and deleting, causing data accumulation, while the GC mechanism of other threads doesn't process it frequently enough, thus failing to resolve the existing problem.

[0057] Furthermore, worker threads cannot traverse garbage data; the main thread only needs to monitor it. Since the GC mechanism is initiated by the main thread, if worker threads traverse and delete garbage data, it will add another worker thread responsible for traversing the garbage data. When the GC mechanism is turned off, that is, when garbage processing by the GC mechanism is not required, the worker thread will still continue to traverse the garbage data, causing the GC mechanism to lose control. Therefore, this embodiment of the invention adopts a data division of labor approach, where the GC main thread initiates the GC mechanism, traverses the garbage data in the storage pool and sends it to a queue, and then sends the garbage data to worker threads through dequeueing, where the worker threads are responsible for deleting the garbage data.

[0058] In addition, the number of worker threads can be set according to the actual performance of the storage system, which can be adjusted and optimized to meet high concurrency and effectively solve the problem of garbage data accumulation. It can also be set according to the spatial content of the garbage data and the amount processed each time. This invention does not make any specific limitations.

[0059] This invention provides a GC mechanism processing method, which includes creating a GC main thread, a queue, and worker threads when the RGW service starts. The GC main thread initiates the GC mechanism; garbage data obtained by the GC main thread traversing the storage pool is stored in the queue; and the queue is controlled to send garbage data to worker threads for deletion via dequeueing. This method, through thread division of labor, effectively improves the processing speed of the GC mechanism by having the GC main thread initiate the GC mechanism and perform garbage collection when it is active, while the main thread is responsible for traversing the garbage data and the worker threads perform garbage data deletion. This alleviates the pressure of single-threaded garbage collection and ensures system stability. Simultaneously, multiple threads perform garbage data deletion, achieving high concurrency and freeing up available disk space.

[0060] Based on the above embodiments, in step S12, the garbage data is obtained by the GC main thread traversing the storage pool. The storage pool is established in the following way:

[0061] When a client of the RGW service performs a deletion operation, the object data generated by the deletion operation is deleted to obtain garbage data, which includes object header data and object tail data.

[0062] Garbage data is stored in a linked list and combined with the object name corresponding to the garbage data into a key-value structure and stored in the storage pool.

[0063] Correspondingly, object data is deleted to obtain garbage data, including:

[0064] Delete the object header data, and keep the object tail data as garbage data.

[0065] Specifically, in the RGW client, the object data generated by the deletion operation is deleted to obtain garbage data. In object storage, data larger than 4MB to be uploaded and saved is considered large object data. When this type of data is stored in the storage cluster, it is divided into a 4MB object header and an object tail exceeding 4MB. If the tail exceeds 4MB, it will be split into tail portions based on a 4MB size. When performing a deletion operation, the object header is deleted directly, and all tail portions of the object become garbage data that needs to be garbage collected by GC. Specifically, the object tail data is divided into multiple 4MB object tail data and stored as garbage data in a linked list, combined with the object name to form a tail key-value structure and stored in the storage pool.

[0066] Its storage pool is named meta.log, generating 517 rados objects, named GC.0-516. This shard stores key-value data. Each key-value entry's key is the name of the object to which a deletion or other operation was performed, and the value is the garbage data of that object, i.e., the tail portion of that object. The value is stored as a linked list containing all the garbage data corresponding to that object, i.e., the object's tail portion, forming a shard. The data in the storage pool exists in the form of rados objects, and the "GC shards" are the 517 rados objects in the meta.log storage pool, named gc.0-516.

[0067] Rados objects are the core of the entire object storage system, primarily composed of OSDs, Monitors, and Clients. OSDs are the storage nodes of the cluster, used for data storage and maintenance. In essence, an OSD is a server with an operating system installed; its hardware includes a processor, a certain amount of memory, a network interface card (NIC), and a hard drive (where the data is actually stored). Each OSD possesses a certain computing power. Each OSD system has a daemon process called OSD Daemon. The Daemon is responsible for communicating with the Monitors cluster and other OSDs, maintaining system status, and collaborating with other OSDs to complete data storage and maintenance; it also communicates with Clients to perform various data object operations. Clients are the object storage clients, providing three interfaces: RBD (block storage), CephFS (file storage), and RadosGW (object storage) (this series focuses only on object storage), meaning clients encompassing all three. RadosGW provides an object storage interface compatible with S3 and Swift. Clients can send data to the RADOS cluster through these interfaces. RADOS stores this data as objects, each object being a file in the file system, stored on the OSD's storage device. The OSD Daemon is responsible for handling read and write operations on storage devices.

[0068] Monitors is a cluster of monitors in Rados. A Monitor is a standalone daemon; one Monitor can monitor one Ceph storage cluster. However, to avoid single points of failure for individual Monitors, a Monitors cluster is formed to ensure high availability. The Monitors cluster uses the Paxos distributed consensus algorithm to ensure data synchronization. Monitors records the status of all OSDs in the storage cluster, forming a primary replica of the cluster runtime graph, including cluster members, status, and changes. The cluster runtime graph is sent to all OSD nodes and clients. OSDs use the cluster runtime graph for data storage and maintenance; clients use the cluster runtime graph for data addressing.

[0069] When a client of the RGW service performs a deletion operation, the object data generated by the deletion operation is deleted to obtain garbage data. The object data includes object header data and object tail data. The garbage data is stored in a linked list and combined with the object name corresponding to the garbage data to form a KV structure and stored in a storage pool to complete the establishment of the storage pool.

[0070] Based on the above embodiments, there can be one or more worker threads. When multiple worker threads exist, step S13, which controls the queue to send garbage data to the worker threads via dequeueing, includes:

[0071] The control queue sends garbage data to the worker threads in a polling manner.

[0072] Specifically, when the queue is not empty, garbage data is dequeued and sent to worker threads in a round-robin fashion. The round-robin method can be achieved by using an incrementing counter to divide the array length, pairing the beginning and end of the array together, and then pairing the last element with the first element. The process is repeated until the last element is reached, and the result is returned to the first element. Alternatively, the index can be used to perform a modulo operation on the array size to return to the beginning of the array, thus achieving round-robin processing. Alternatively, worker threads can be used one at a time to check the data. This invention is not specifically limited to any particular method, as long as it effectively utilizes worker threads to delete garbage data.

[0073] In one preferred embodiment, the control queue sends garbage data to the worker thread via a polling method, including:

[0074] Get the current working status of the worker thread, which includes the idle state and the data deletion state;

[0075] When the current worker thread is in the data deletion state, the working state of the next current worker thread is obtained until the current worker thread is in the idle state. Then, the garbage data is sent to the current worker thread corresponding to the idle state for deletion.

[0076] The system retrieves the current worker thread's status, which includes an idle state and a data deletion state. An idle state indicates the current worker thread is not deleting data, while a data deletion state indicates the current worker thread is deleting garbage data. The queue sends garbage data to worker threads, which need to poll the current worker thread's status. When the current worker thread's status is in the data deletion state, the system retrieves the status of the next current worker thread until the current worker thread's status becomes idle. Then, the queue sends the garbage data to the idle worker thread for deletion.

[0077] After the GC main thread is created and started, it remains active, continuously traversing the key-value structure data on the 517 fragments in the meta.log storage pool, and traversing all object garbage data in the linked list stored in the value, pushing the results into the queue. At this time, the GC main thread acts as a monitor; when garbage data is sent from the queue to worker threads, its pointer variable is also sent to the main thread. When a worker thread deletes garbage data, it sends the return result back to the main thread, at which point the pointer variable in the main thread changes.

[0078] Furthermore, to avoid resource contention issues when worker threads are deleting garbage data, thread locks can be used. For example, if worker thread 1 and worker thread 2 both access the same global variable or the same garbage data, other garbage data cannot be processed, resulting in significant time consumption and impacting deletion efficiency. Therefore, it is necessary to allow only one thread to access one piece of garbage data at a time, introducing thread locks to protect the data. This invention does not specifically limit the type of lock, but deadlocks must be avoided. If a thread attempts to acquire the same mutex twice, it will enter a deadlock state, and other less obvious methods can also cause deadlocks. Critical resource: A resource that can be accessed by all programs. Critical section: The location where critical resources are stored, locked sequentially.

[0079] The control queue provided in this embodiment of the invention sends garbage data to worker threads via a polling method. Specifically, it includes obtaining the current working state of the worker thread, which includes an idle state and a data deletion state. When the current working thread is in the data deletion state, it obtains the working state of the next current worker thread until the current working thread's working state becomes idle. Then, it sends the garbage data to the current worker thread corresponding to the idle state for deletion processing. This avoids introducing new garbage data when worker threads have not completely deleted the old garbage data, which could lead to slow thread operation and data congestion, thus improving the data deletion processing speed.

[0080] Based on the above embodiments, if all current worker threads are in the data deletion state, and the queue contains new garbage data, the method further includes:

[0081] Set up heartbeat data to monitor worker threads;

[0082] When a change in the working state of a worker thread is detected, garbage data is sent to the worker thread whose working state has changed for deletion.

[0083] Specifically, with persistent connections, there might be a long period of no data exchange. Theoretically, the connection should remain open indefinitely, but in reality, it's difficult to detect if a node in the middle fails. More critically, some nodes (firewalls) will automatically disconnect connections that haven't exchanged data for a certain period. In this situation, heartbeats can be used to maintain the persistent connection and keep it alive. The heartbeat mechanism sends a fixed message to the server every few minutes. Upon receiving the heartbeat, the server replies with a fixed message. If the server doesn't receive a message from the client within a few minutes, it considers the connection disconnected.

[0084] The program needs to execute batch tasks in the background at specified times. Correspondingly, a monitoring mechanism is required to observe the operation of these worker threads. Setting up heartbeat data is helpful in detecting the worker thread's status. In Transmission Control Protocol (TCP) network communication, abnormal disconnections between clients and servers frequently occur, necessitating real-time monitoring of the connection status. A common solution is to incorporate a heartbeat mechanism into the program.

[0085] When receiving and sending data, I designed a daemon process (thread) to periodically send heartbeat packets. After the client / server receives the packet, it immediately returns the corresponding packet to detect whether the other party is online in real time.

[0086] Thread pools address the efficient communication between clients and servers, handling the sending and receiving of requests. They also address the server's ability to process these requests concurrently. When a server receives a request from a client, it places it in a maintained task queue (provided the queue is not full). When the task queue is not empty, threads in the thread pool (pre-created, with threads in a waiting-to-be-alive state) retrieve tasks from the queue for processing. It's important to note that thread pools have attributes such as the number of currently active threads, the number of threads actively engaged, and the step size for expanding or shrinking the pool. The expansion (when more than 80% of threads are actively engaged) and termination of threads in the thread pool are handled by the manager thread. Because the task queue is a shared resource, operations on it require locking.

[0087] When a change in the working state of a worker thread is detected, garbage data is sent to the worker thread whose working state has changed for deletion.

[0088] The present invention provides a method for setting heartbeat data to monitor worker threads; when a change in the working state of a worker thread is detected, garbage data is sent to the worker thread whose working state has changed for deletion, thereby improving the processing speed of data deletion.

[0089] Based on the above embodiments, the method further includes:

[0090] When creating a storage pool, create garbage data records;

[0091] After the worker thread deletes garbage data, it controls the worker thread to generate a completion message and send the completion message to the GC main thread;

[0092] When the number of completed messages received reaches a threshold, garbage data records in the storage pool are pruned.

[0093] Specifically, when creating the storage pool, garbage data records are created, recording garbage data on 517 fragments. After the worker thread deletes the garbage data, the worker thread is controlled to generate a completion message return result, which is sent to the GC main thread. The state is changed through pointer variables. When the number of garbage data records that need to be deleted is greater than the threshold, that is, when the threshold is reached, a unified pruning of records on GC fragments is performed.

[0094] The main thread is responsible for traversing garbage data on GC fragments and trimming garbage data records on GC fragments. Worker threads share the task of deleting garbage data. By cooperating with each other, the speed of GC processing can be effectively improved, while alleviating the pressure of existing single-threaded garbage collection and ensuring the stability of the system.

[0095] This invention provides a method for creating garbage data records when creating a storage pool. After a worker thread deletes garbage data, it generates a completion message and sends it to the GC main thread. When the number of received completion messages reaches a threshold, the garbage data records in the storage pool are pruned. The main thread is responsible for traversing the garbage data on the GC fragments and pruning the garbage data records on the GC fragments. This alleviates the pressure of existing single-threaded garbage collection and ensures system stability.

[0096] Based on the above embodiments, the creation of the worker thread in step S11 includes:

[0097] Create worker threads using a vector container.

[0098] Specifically, `vector` is a vector type that can hold data of many types, such as a number of integers, hence it is called a container. A `vector` container is used to start and store worker threads, and a `queue` is used to store garbage data encountered by the main garbage collector (GC) thread.

[0099] This invention provides a method for creating worker threads using a vector container. It can be seen that Vector encapsulates an array and delegates the expansion and contraction of the array to the container itself. Vector supports concurrent access by multiple threads because methods that modify the container are modified with the `synchronized` keyword (object lock, locking the entire container object), making it safer for threads.

[0100] The foregoing has described in detail various embodiments of the GC mechanism processing method. Based on this, the present invention also discloses a watch calibration device corresponding to the above method. Figure 2 This is a structural diagram of a GC mechanism processing device provided in an embodiment of the present invention. Figure 2 As shown, the GC mechanism processing device includes:

[0101] Create module 11, which is used to create the GC main thread, queue and worker threads when the RGW service starts, wherein the GC main thread enables the GC mechanism;

[0102] Storage module 12 is used to store garbage data obtained by the GC main thread traversing the storage pool into a queue;

[0103] The sending module 13 is used to control the queue to send garbage data to the worker thread by dequeuing so that the worker thread can delete the garbage data.

[0104] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments described in the method part for the embodiments of the device part, and will not be repeated here.

[0105] This invention provides a GC mechanism processing device, which includes creating a GC main thread, a queue, and worker threads when the RGW service starts. The GC main thread initiates the GC mechanism; garbage data obtained by the GC main thread traversing the storage pool is stored in the queue; and the queue is controlled to send garbage data to worker threads for deletion via dequeueing. This method, through thread division of labor, effectively improves the processing speed of the GC mechanism by having the GC main thread initiate the GC mechanism and perform garbage collection when it is active, while the main thread is responsible for traversing the garbage data and the worker threads perform garbage data deletion. This alleviates the pressure of current single-threaded garbage collection and ensures system stability. Simultaneously, multiple threads perform garbage data deletion, achieving high concurrency and freeing up available disk space.

[0106] Figure 3 A structural diagram of another GC mechanism processing device provided in an embodiment of the present invention is shown below. Figure 3 As shown, the device includes:

[0107] Memory 21 is used to store computer programs;

[0108] Processor 22 is used to implement the steps of GC mechanism processing methods when executing computer programs.

[0109] The GC mechanism processing device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0110] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 22 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 22 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0111] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 21 is used to store at least the following computer program 211, which, after being loaded and executed by the processor 22, is capable of implementing the relevant steps of the GC mechanism processing method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. The operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, the data involved in the GC mechanism processing method, etc.

[0112] In some embodiments, the GC mechanism processing device may further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.

[0113] Those skilled in the field can understand, Figure 3 The structure shown does not constitute a limitation on the GC mechanism processing device and may include more or fewer components than illustrated.

[0114] The processor 22 implements the GC mechanism processing method provided in any of the above embodiments by calling instructions stored in the memory 21.

[0115] This invention provides a GC mechanism processing device, which includes creating a GC main thread, a queue, and worker threads when the RGW service starts. The GC main thread initiates the GC mechanism; garbage data obtained by the GC main thread traversing the storage pool is stored in the queue; and the queue is controlled to send garbage data to worker threads for deletion via dequeueing. This method, through thread division of labor, effectively improves the processing speed of the GC mechanism by having the GC main thread initiate the GC mechanism and perform garbage collection when it is active, while the main thread is responsible for traversing the garbage data and the worker threads perform garbage data deletion. This alleviates the pressure of current single-threaded garbage collection and ensures system stability. Simultaneously, multiple threads perform garbage data deletion, achieving high concurrency and freeing up available disk space.

[0116] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by processor 22, implements the steps of the GC mechanism processing method described above.

[0117] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above GC mechanism processing method.

[0119] As a preferred embodiment, when there are 3 worker threads, the GC mechanism handles the situation as follows:

[0120] Step 1: When the RGW service starts, the GC main thread is created and initialized. During the initialization process of the GC main thread, 517 fragments, i.e., rados objects, will be created in the meta.log storage pool, named GC.0-516.

[0121] Step 2: While the GC main thread is initialized, a thread pool (workpool) is created. Then, the workpool creates three worker threads (workQ1-3) and stores the three worker threads in a vector. At the same time, a queue (quene) is created to store the garbage data traversed by the GC main thread.

[0122] Step 3: After the object header is deleted by the RGW client through operations such as deletion, the tail part of the object becomes garbage data. All the tail garbage data corresponding to the object is stored in a linked list and combined with the object name to form a KV structure, which is then stored in the 517 fragments (rados objects) created in the meta.log storage pool.

[0123] Step 4: After the GC main thread is created and started, it remains active, continuously traversing the KV structure data on the 517 fragments in the meta.log storage pool, and traversing all the garbage data of objects in the linked list where the value is stored. The traversed results are pushed into the queue queue. At the same time, when the queue is not empty, the data in the queue will be dequeued and sent to the three worker threads in turn for processing.

[0124] Step 5: When the three worker threads receive garbage data dequeued from the queue, they directly perform the data deletion operation and then return a completion message to the GC main thread. After receiving the message, the main thread performs a unified pruning of the records on the GC fragment when the number of garbage data records to be deleted exceeds the threshold. This completes the implementation process of the multi-threaded garbage collection mechanism (GC).

[0125] For an introduction to the GC mechanism processing method provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above GC mechanism processing method.

[0126] The GC mechanism processing method, apparatus, and medium provided by this invention have been described in detail above. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this invention without departing from the principles of this invention, and these improvements and modifications also fall within the protection scope of the claims of this invention.

[0127] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

Claims

1. A GC mechanism processing method, characterized in that, include: When the RGW service starts, it creates a GC main thread, a queue, and worker threads, wherein the GC main thread enables the GC mechanism; the creation of the GC main thread is used to traverse garbage data, and at the same time initializes and stores worker threads and a queue, wherein the queue stores the garbage data traversed by the GC main thread for the user. Garbage data obtained by the GC main thread traversing the storage pool is stored in the queue; The queue is controlled to send the garbage data to the worker thread by dequeueing, so that the worker thread can delete the garbage data. Correspondingly, the storage pool is established in the following way: When the client of the RGW service performs a deletion operation, the object data generated by the deletion operation is deleted to obtain the garbage data, wherein the object data includes object header data and object tail data; The garbage data is stored in a linked list and combined with the object name corresponding to the garbage data into a key-value structure and stored in the storage pool; Correspondingly, the object data is deleted to obtain the garbage data, including: Delete the object header data, and retain the object tail data as garbage data; Correspondingly, when there are multiple worker threads, the queue is controlled to send the garbage data to the worker threads via dequeueing, including: The queue is controlled to send the garbage data to the worker thread in a polling manner; wherein, the polling method is to use an incrementing counter to divide the array length, pair the beginning and end of the array into points, and pair the last and the first into a group, and traverse to the end and return to the first position by taking the remainder, or to use the index to perform a remainder operation on the array size to return to the beginning of the array, thereby realizing the polling of the array, or one worker thread at a time. When worker threads delete garbage data, a thread lock is used to ensure that only one thread can access one piece of garbage data at a time, thus protecting the data.

2. The GC mechanism processing method according to claim 1, characterized in that, The control of sending garbage data to the worker thread via a polling method includes: Get the current working status of the worker thread, where the working status includes idle status and data deletion status; When the current working thread is in the data deletion state, the next working state of the current working thread is obtained until the current working thread is in the idle state. Then, the garbage data is sent to the current working thread corresponding to the idle state for deletion.

3. The GC mechanism processing method according to claim 2, characterized in that, When all current worker threads are in the data deletion state, the process further includes: Set heartbeat data to monitor the worker thread; When a change in the working state of the worker thread is detected, the garbage data is sent to the worker thread whose working state has changed for deletion.

4. The GC mechanism processing method according to any one of claims 1 to 3, characterized in that, Also includes: When creating the storage pool, a garbage data record is created; After the worker thread deletes the garbage data, it controls the worker thread to generate a completion message and send the completion message to the GC main thread; When the number of completed messages received reaches a threshold, the garbage data records in the storage pool are pruned.

5. The GC mechanism processing method according to claim 1, characterized in that, Creating the worker thread includes: The worker thread is created using a vector container.

6. A GC mechanism processing device, characterized in that, include: A module is created to create a GC main thread, a queue, and worker threads when the RGW service starts, wherein the GC main thread enables the GC mechanism; The creation of the GC main thread is used to traverse garbage data, and at the same time initializes and stores worker threads and queues. The queue is used by users to store the garbage data traversed by the GC main thread. The storage module is used to store garbage data obtained by the GC main thread traversing the storage pool into the queue. The sending module is used to control the queue to send the garbage data to the worker thread by dequeueing so that the worker thread can delete the garbage data. Correspondingly, the storage pool is established in the following way: When the client of the RGW service performs a deletion operation, the object data generated by the deletion operation is deleted to obtain the garbage data, wherein the object data includes object header data and object tail data; The garbage data is stored in a linked list and combined with the object name corresponding to the garbage data into a key-value structure and stored in the storage pool; Correspondingly, the object data is deleted to obtain the garbage data, including: Delete the object header data, and retain the object tail data as garbage data; Correspondingly, when there are multiple worker threads, the queue is controlled to send the garbage data to the worker threads via dequeueing, including: The queue is controlled to send the garbage data to the worker thread in a polling manner; wherein, the polling method is to use an incrementing counter to divide the array length, pair the beginning and end of the array into points, and pair the last and the first into a group, and traverse to the end and return to the first position by taking the remainder, or to use the index to perform a remainder operation on the array size to return to the beginning of the array, thereby realizing the polling of the array, or one worker thread at a time. When worker threads delete garbage data, a thread lock is used to ensure that only one thread can access one piece of garbage data at a time, thus protecting the data.

7. A GC mechanism processing device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the GC mechanism processing method as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the GC mechanism processing method as described in any one of claims 1 to 5.

Images