Garbage recycling method and device, equipment and storage medium

Abstract

The invention relates to a garbage recycling method and device, equipment and a storage medium. The method comprises the following steps: monitoring traffic historical data of the distributed storage system and monitoring resource use data of storage equipment; predicting traffic trend data in a future time period by using a traffic prediction model according to the traffic historical data; garbage collection parameters corresponding to the future time period are determined according to the flow trend data; adjusting garbage collection parameters corresponding to the future time period according to the resource use data; and issuing the adjusted garbage collection parameters to an object gateway in the distributed storage system, and controlling the object gateway to execute a garbage collection task by using the adjusted garbage collection parameters in a future time period so as to clear garbage data in the storage device. According to the method, the garbage collection parameters are determined according to the dynamically predicted traffic trend data, so that the garbage collection parameters dynamically adapt to traffic data changes, the system performance is considered, the garbage collection parameters are matched with the system performance, and the garbage collection efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a garbage collection method, device, equipment and storage medium. Background Art

[0002] In a distributed storage system, a large amount of fragmented space is generated on the disk due to the frequent operations of clients such as uploading sharded data, overwriting data, and deleting data. As time goes by, the fragmented space on the disk continues to accumulate. If these fragmented spaces are not recovered in time, it will lead to a waste of storage resources and may affect the storage space utilization efficiency of the business.

[0003] Currently, before recycling the fragmented space of the disk, static garbage collection parameters are manually set, and the Rados Gateway (RGW) performs garbage collection tasks according to the garbage collection parameters to release the fragmented space occupied by disk garbage and complete the recycling of the fragmented space.

[0004] However, due to the use of static garbage collection parameters, the current garbage collection mechanism cannot dynamically adapt to changes in business traffic. Furthermore, after the data in the disk is marked as garbage data, the garbage collection task for processing the data will be stored in the garbage collection queue. When the object gateway executes the garbage collection task in the garbage collection queue, it needs to send an operation instruction to the storage device. In the process of the storage device executing the operation instruction, the central processing unit (CPU) resources will be occupied to perform data processing and logical judgment, and memory resources will also be occupied to cache data and intermediate results. The disk input / output (I / O) resource support is also required. When the garbage collection queue explodes or is congested, executing the garbage collection task will over-consume disk, CPU and memory resources, especially disk resources. If the garbage collection task is executed during the peak period of business traffic, the excessive consumption of such resources will cause client requests to be blocked, causing 504 errors, and ultimately reducing the availability of the service. Summary of the invention

[0005] The present application provides a garbage collection method, apparatus, device and storage medium to solve the problem that the current garbage collection mechanism cannot dynamically adapt to changes in business traffic due to the use of static garbage collection parameters.

[0006] In view of the above technical problems, the technical solution of this application is solved by the following embodiments:

[0007] An embodiment of the present application provides a garbage collection method, including: monitoring traffic history data of a distributed storage system and monitoring resource usage data of a storage device in the distributed storage system; inputting the traffic history data into a pre-trained traffic prediction model so that the traffic prediction model predicts traffic trend data of a future time period based on the traffic history data; determining garbage collection parameters corresponding to the future time period based on the traffic trend data; adjusting the garbage collection parameters corresponding to the future time period based on the resource usage data; and sending the adjusted garbage collection parameters to an object gateway in the distributed storage system, controlling the object gateway to perform garbage collection tasks using the adjusted garbage collection parameters within the future time period, so as to clear garbage data in the storage device.

[0008] Among them, determining the garbage collection parameters corresponding to the future time period based on the traffic trend data includes: using a preset linear function to determine the garbage collection parameters corresponding to the traffic trend data in the future time period; or using a pre-trained parameter determination model to determine the garbage collection parameters corresponding to the traffic trend data in the future time period.

[0009] Among them, adjusting the garbage collection parameters of the future time period according to the resource usage data includes: in a preset decision tree, determining the leaf nodes that the resource usage data meets; wherein each node in the decision tree corresponds to a judgment condition; each leaf node corresponding to the node is a judgment result corresponding to the judgment condition; the judgment result includes an adjustment method and an adjustment value; adjusting the garbage collection parameters according to the adjustment method and the adjustment value corresponding to the leaf node that the resource usage data meets.

[0010] Among them, the monitoring of the traffic history data of the distributed storage system and the monitoring of the resource usage data of the storage devices in the distributed storage system include: monitoring the traffic history data corresponding to the target storage device group in the distributed storage system and monitoring the resource usage data of the target storage device group; wherein the target storage device group includes at least one storage device; the target storage device group corresponds to at least one object gateway, and the object gateways corresponding to the target storage device group and other storage device groups are different; the sending of the adjusted garbage collection parameters to the object gateways in the distributed storage system includes: sending the adjusted garbage collection parameters to each of the object gateways corresponding to the target storage device group.

[0011] Among them, the monitoring of the traffic history data of the distributed storage system and the monitoring of the resource usage data of the storage devices in the distributed storage system include: monitoring the traffic history data corresponding to the storage devices within a preset range in the distributed storage system and monitoring the resource usage data of the storage devices within the preset range; wherein, for the storage devices within the preset range, the storage devices are connected to each object gateway in the distributed storage system; the sending of the adjusted garbage collection parameters to the object gateways in the distributed storage system includes: sending the adjusted garbage collection parameters to each of the object gateways connected to the storage devices within the preset range.

[0012] Among them, after sending the adjusted garbage collection parameters to the object gateway in the distributed storage system, and before controlling the object gateway to use the adjusted garbage collection parameters to perform the garbage collection task within the future time period, it also includes: controlling the object gateway to send a garbage collection notification message to the storage application corresponding to the garbage collection task, so as to adjust the current state of the storage application to the garbage collection state.

[0013] The method further includes: after the object gateway receives the data to be stored from the client, prohibiting the object gateway from sending the data to be stored to a storage application whose current state is a garbage collection state.

[0014] An embodiment of the present application also provides a garbage collection device, including: a monitoring module, which is used to monitor the traffic history data of a distributed storage system and monitor the resource usage data of a storage device in the distributed storage system; a prediction module, which is used to input the traffic history data into a pre-trained traffic prediction model, so that the traffic prediction model predicts the traffic trend data of a future time period based on the traffic history data; a determination module, which is used to determine the garbage collection parameters corresponding to the future time period based on the traffic trend data; an adjustment module, which is used to adjust the garbage collection parameters corresponding to the future time period based on the resource usage data; and a recovery module, which is used to send the adjusted garbage collection parameters to an object gateway in the distributed storage system, and control the object gateway to perform garbage collection tasks using the adjusted garbage collection parameters within the future time period, so as to clear the garbage data in the storage device.

[0015] An embodiment of the present application also provides a garbage collection device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to: execute a garbage collection program stored in the memory to implement any of the above-mentioned garbage collection methods.

[0016] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed to implement any of the above-mentioned garbage collection methods.

[0017] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages compared with the prior art: the method provided by the embodiment of the present application can monitor the traffic history data of the distributed storage system and monitor the resource usage data of the storage device in the distributed storage system; input the traffic history data into a pre-trained traffic prediction model so that the traffic prediction model predicts the traffic trend data of the future time period based on the traffic history data; determine the garbage collection parameters corresponding to the future time period based on the traffic trend data; adjust the garbage collection parameters corresponding to the future time period based on the resource usage data; send the adjusted garbage collection parameters to the object gateway in the distributed storage system, and control the object gateway to perform garbage collection tasks using the adjusted garbage collection parameters within the future time period so as to clear the garbage data in the storage device. The embodiments of the present application dynamically predict the traffic trend data for a future time period based on the monitored historical traffic data, and determine the garbage collection parameters based on the dynamically predicted traffic trend data, so that the garbage collection parameters dynamically adapt to the changes in the traffic data. The embodiments of the present application take the system performance into consideration, that is, the garbage collection parameters are adjusted based on the monitored resource usage data, so that the garbage collection parameters match the system performance, thereby improving the efficiency of garbage collection, improving the utilization of storage resources and the availability of services, and avoiding performance loss and client request blocking problems caused by garbage collection queue congestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0021] Figure 1is a flow chart of a garbage collection method according to an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the connection relationship between a storage device and an object gateway according to an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a connection relationship between a storage device and an object gateway according to another embodiment of the present application;

[0024] Figure 4 is a structural diagram of a garbage collection device according to an embodiment of the present application;

[0025] Figure 5 It is a structural diagram of a garbage collection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0027] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0028] The present application embodiment provides a garbage collection method. Figure 1 , which is a flow chart of a garbage collection method according to an embodiment of the present application.

[0029] Step S110 , monitoring the traffic history data of the distributed storage system and monitoring the resource usage data of the storage devices in the distributed storage system.

[0030] The distributed storage system may be a distributed file system (ceph).

[0031] Traffic history data refers to traffic data within a preset time period. The end point of the preset time period is the current time point or a time point before the current time point. Traffic data includes but is not limited to: the number of read and write operations per second (Input / Output Operations Per Second, referred to as IOPS) and bandwidth.

[0032] The storage device may be a device where a storage application (Object Storage Device, OSD for short) and a disk are located in a distributed storage system.

[0033] Resource usage data is used to reflect the resource usage of storage devices at the current point in time. Resource usage data includes but is not limited to: CPU usage, disk usage, and memory usage. Of course, resource usage data can also include: disk health and performance data as required.

[0034] Step S120, inputting the traffic history data into a pre-trained traffic prediction model, so that the traffic prediction model predicts the traffic trend data of a future time period based on the traffic history data.

[0035] The traffic prediction model is trained to predict traffic trend data for future time periods based on historical traffic data. The types of traffic prediction models include, but are not limited to, long short-term memory (LSTM) models.

[0036] The starting time point of the future time period is the current time point or a time point that has not arrived after the current time point. The length of the future time period can be determined according to needs.

[0037] Traffic trend data refers to traffic data in the future time period, for example, IOPS and bandwidth for the next day.

[0038] Step S130, determining garbage collection parameters corresponding to the future time period according to the traffic trend data.

[0039] The garbage collection parameters refer to the parameters to be followed during the garbage collection process. The flow trend data matches the garbage collection parameters. Furthermore, the flow intensity of the flow trend data is negatively correlated with the garbage collection intensity of the garbage collection parameters.

[0040] For example: as the intensity of traffic trend data increases, the intensity of garbage collection parameters decreases; as the intensity of traffic trend data decreases, the intensity of garbage collection parameters increases; this can reduce the garbage collection intensity during peak traffic periods and increase the garbage collection intensity during low traffic periods.

[0041] Step S140: adjusting garbage collection parameters corresponding to the future time period according to the resource usage data.

[0042] Since resource usage data plays a decisive role in data processing, in the embodiment of the present application, the garbage collection parameters not only need to be adapted to the traffic trend data, but also need to match the resource usage data so that system resources can support the smooth progress of garbage collection, thereby achieving multi-angle garbage collection efficiency improvement.

[0043] Step S150: Send the adjusted garbage collection parameters to the object gateway in the distributed storage system, and control the object gateway to perform garbage collection tasks using the adjusted garbage collection parameters within the future time period, so as to clear garbage data in the storage device.

[0044] In an embodiment of the present application, the flow history data of the distributed storage system is monitored and the resource usage data of the storage device in the distributed storage system is monitored; the flow history data is input into a pre-trained flow prediction model so that the flow prediction model predicts the flow trend data of the future time period according to the flow history data; the garbage collection parameters corresponding to the future time period are determined according to the flow trend data; the garbage collection parameters corresponding to the future time period are adjusted according to the resource usage data; the adjusted garbage collection parameters are sent to the object gateway in the distributed storage system, and the object gateway is controlled to perform the garbage collection task in the future time period using the adjusted garbage collection parameters to clear the garbage data in the storage device. The embodiment of the present application dynamically predicts the flow trend data of the future time period according to the monitored flow history data, and determines the garbage collection parameters according to the dynamically predicted flow trend data, so that the garbage collection parameters dynamically adapt to the changes in the flow data, and the embodiment of the present application considers the system performance, that is, the garbage collection parameters are adjusted according to the monitored resource usage data, so that the garbage collection parameters match the system performance, thereby improving the efficiency of garbage collection, improving the storage resource utilization and service availability, and avoiding the performance loss and client request blocking caused by garbage collection queue congestion.

[0045] In order to make the embodiments of the present application easier to understand, the garbage collection method of the embodiments of the present application is further described below.

[0046] In an embodiment of the present application, all traffic historical data processed by the distributed storage system can be monitored, and the overall resource usage data of all storage devices in the distributed storage system can be monitored; or, part of the traffic historical data processed by the distributed storage system can be monitored, and the overall resource usage data of the storage devices corresponding to the part of the traffic historical data can be monitored.

[0047] Furthermore, when processing traffic data, the monitoring tools provided by the distributed storage system can be used to monitor preset traffic data, such as monitoring the IOPS, bandwidth, etc. corresponding to the read, write, and delete operations of the client; when the traffic data is stored in the database, tags are added to the traffic data. For example, timestamps are added to the traffic data, and information about the processing device and storage device are added to the traffic data. Based on the tags added to the traffic data, historical traffic data within a preset time period is monitored.

[0048] Furthermore, if Figure 2 The diagram shows the connection relationship between the storage device and the object gateway according to an embodiment of the present application. Each storage device in the distributed storage system can be divided into multiple storage device groups in advance; each storage device group includes at least one storage device; each storage device group corresponds to at least one object gateway, that is, each storage device in the storage device group is connected to each object gateway corresponding to the storage device group; the object gateways corresponding to different storage device groups are different. In this case, the traffic history data corresponding to the target storage device group in the distributed storage system is monitored, and the resource usage data of the target storage device group is monitored; wherein the target storage device group includes at least one storage device; the target storage device group corresponds to at least one object gateway, and the object gateways corresponding to the target storage device group and other storage device groups are different. The number of target storage device groups can be multiple, and the above process can be executed separately for each target storage device group. In this way, different target storage device groups are isolated from each other, and the garbage collection parameters of the object gateways corresponding to the same target storage device group are the same, and the garbage collection parameters of the object gateways corresponding to different target storage device groups are different, so that different object gateways can perform garbage data collection according to their corresponding traffic conditions.

[0049] Furthermore, the traffic history data corresponding to the storage devices within a preset range in the distributed storage system is monitored, and the resource usage data of the storage devices within the preset range is monitored; wherein, for the storage devices within the preset range, the storage devices are connected to each object gateway in the distributed storage system. Furthermore, if Figure 3The figure shows a connection relationship diagram of a storage device and an object gateway according to another embodiment of the present application. For each storage device in a distributed storage system, the storage device can be connected to each object gateway of the distributed storage system respectively. The operation status of the storage device can be preset according to the usage requirements of the storage device. The types of operation status include: operation status and waiting status; the operation status indicates that the storage device is currently responsible for storage tasks; the waiting status indicates that the storage device is not currently responsible for storage tasks. The preset range can be storage devices whose current operation status is the operation status. Since the storage device in the waiting status can remain connected to the object gateway or disconnect from the object gateway, Figure 3 The connection relationship between the storage device in the waiting state and the object gateway is not shown. In this way, the storage device that undertakes the storage task can be selected from the storage devices according to the traffic size, so that the utilization rate of the storage device is optimized and the distributed storage system reaches the best operating state.

[0050] In an embodiment of the present application, a pre-trained traffic prediction model can be obtained so that the traffic prediction model predicts traffic trend data for a future time period based on monitored historical traffic data.

[0051] The traffic prediction model may be an LSTM neural network. When training the traffic prediction model, the traffic prediction model may learn traffic patterns and predict future trends of traffic data to form traffic trend data.

[0052] Furthermore, deep learning frameworks (such as TensorFlow, PyTorch, etc.) can be used to train traffic prediction models to improve the accuracy of traffic prediction models in predicting traffic underestimation and peaks. For training samples, historical traffic data can be collected from the database; historical traffic data can be preprocessed, including: data cleaning for historical traffic data, such as removing outliers and filling missing values ​​with mean filling; normalization of historical traffic data after data cleaning to unify the numerical range of historical traffic data; selection of time series features useful for traffic prediction in historical traffic data, such as traffic data that can reflect traffic changes at different times such as year, month, day, week, hour, minute, etc. Another example: traffic data that can reflect traffic changes in different seasons such as spring, summer, autumn, and winter. If the preset time period and the future time period are continuous time, then a historical time point can be determined and used as a dividing point to divide each time series feature into two parts. The length of the former part is equal to the length of the preset time period and the former part of the time series features is used as a training sample, and the length of the latter part is equal to the length of the future time period and the latter part of the time series features is used as the true value corresponding to the training sample. In terms of training methods, the traffic prediction model can be trained using a back propagation algorithm until it is determined that the traffic prediction model converges or until the traffic prediction model reaches a preset number of iterations. After the traffic prediction model training is completed, the traffic prediction model is stored so that the traffic prediction model can be used to predict the traffic trend data of the future time period. For example: predicting the traffic trend data for the next day, the next week, the next month, or the next year.

[0053] Furthermore, the traffic history data is input into a pre-trained traffic prediction model, and the traffic trend data for a future time period predicted by the traffic prediction model based on the traffic history data is obtained; and the traffic trend data for a future time period predicted by the traffic prediction model based on the traffic history data is displayed on a preset display device. The traffic trend data for a future time period can be displayed in the form of a traffic curve graph. For example: based on the traffic history data, the traffic trend data for the next year is predicted, and the traffic changes for the next year can be seen in the traffic curve graph, and the traffic changes for each day, week, and month can be displayed in a more fine-grained manner.

[0054] In the embodiment of the present application, after obtaining the traffic trend data for the future time period, the garbage collection parameters corresponding to the future time period can be determined according to the traffic trend data.

[0055] Furthermore, a preset linear function may be used to determine the garbage collection parameter corresponding to the traffic trend data in the future time period, wherein the linear function is used to generate a garbage collection parameter that is negatively correlated with the traffic intensity of the traffic trend data in the future time period.

[0056] Furthermore, a pre-trained parameter determination model can be used to determine the garbage collection parameters corresponding to the traffic trend data in the future time period. The parameter determination model is trained to generate garbage collection parameters that are negatively correlated with the traffic intensity of the traffic trend data in the future time period. The parameter determination model for the embodiment of the present application can be a model using a genetic algorithm. By training the model using a genetic algorithm, the model has the ability to determine the garbage collection parameters corresponding to the traffic trend data in the future time period.

[0057] Further, the types of garbage collection parameters in the embodiment of the present application include, but are not limited to: the number of garbage collection tasks (rgw_gc_max_objs), the execution time of each garbage collection task (rgw_gc_processor_max_time), and the interval time between two adjacent garbage collection operations of the garbage collection task (rgw_gc_processor_period). In the embodiment of the present application, the smaller the traffic intensity at a certain time point in the traffic trend data, the greater the intensity of the garbage collection parameter corresponding to the time point, that is, the more the number of garbage collection tasks, the longer the execution time of each garbage collection task, and the shorter the interval time between two adjacent garbage collection operations of the garbage collection task; the greater the traffic intensity at a certain time point in the traffic trend data, the smaller the intensity of the garbage collection parameter corresponding to the time point, that is, the fewer the number of garbage collection tasks, the shorter the execution time of each garbage collection task, and the longer the interval time between two adjacent garbage collection operations of the garbage collection task.

[0058] In the embodiment of the present application, after the garbage collection parameters are preliminarily determined based on the traffic trend data, the garbage collection parameters need to be fine-tuned based on the resource usage data of the storage device.

[0059] Furthermore, in a preset decision tree, a leaf node to which the resource usage data conforms can be determined; wherein each node in the decision tree corresponds to a judgment condition; each leaf node corresponding to the node is a judgment result corresponding to the judgment condition; the judgment result includes an adjustment method and an adjustment value; the garbage collection parameters are adjusted according to the adjustment method and the adjustment value corresponding to the leaf node to which the resource usage data conforms.

[0060] The adjustment method refers to the direction of adjustment of the garbage collection parameters, for example: increase, decrease.

[0061] For example, resource usage data includes CPU usage, disk usage, and memory usage. By querying the decision tree, the leaf nodes that the resource usage data matches are determined, including:

[0062] Node 1: If CPU usage > 90%, check disk usage;

[0063] Leaf node 1 of node 1: if the disk usage rate is greater than 90%, increase the interval between two adjacent garbage collection operations of the garbage collection task according to the first value;

[0064] Leaf node 2 of node 1: if the disk usage is ≤90%, then the interval between two adjacent garbage collection operations of the garbage collection task is reduced according to the second value;

[0065] Node 2: If CPU usage is ≤ 90%, check memory usage;

[0066] Leaf node 1 of node 2: if the memory usage rate is greater than 90%, then the execution time of each garbage collection task is reduced according to the third value;

[0067] Leaf node 2 of node 2: If memory usage is ≤ 90%, keep garbage collection parameters unchanged.

[0068] In an embodiment of the present application, after the garbage collection parameters are adjusted, the adjusted garbage collection parameters need to be sent to the object gateway in the distributed storage system to control the object gateway to perform garbage collection tasks using the adjusted garbage collection parameters within the future time period so as to clear the garbage data in the storage device.

[0069] Further, if the monitored resource usage data is the resource usage data corresponding to the target storage device group, the adjusted garbage collection parameter is sent to each of the object gateways corresponding to the target storage device group.

[0070] Further, if the monitored resource usage data is resource usage data of a storage device within a preset range, the adjusted garbage collection parameter is sent to each of the object gateways connected to the storage device within the preset range.

[0071] The embodiment of the present application uses a traffic prediction model to predict network traffic and dynamically adjusts garbage collection parameters based on the prediction results. During peak traffic periods, the intensity of garbage collection is appropriately reduced to reduce the occupation of system resources; during low traffic periods, the intensity of garbage collection is increased to efficiently clean up disk fragments. In addition, the embodiment of the present application fine-tunes the garbage collection parameters by monitoring resource usage to ensure that the garbage collection speed is increased when resource utilization is low and the garbage collection speed is reduced when resource utilization is high, thereby avoiding client request blocking due to resource competition and reducing service availability. Moreover, the embodiment of the present application reduces the cost of manual intervention and operation and maintenance by dynamically adjusting the garbage collection parameters, thereby improving the degree of automation of the distributed storage system.

[0072] The garbage collection method of the embodiment of the present application can be deployed on an independent device or in a storage device. The device deploying the garbage collection method of the embodiment of the present application can communicate with each node in the distributed storage system so as to monitor the traffic history data of the distributed storage system and the resource usage data of the storage device in the distributed storage system, and can also send the adjusted garbage collection parameters to the object gateway in the distributed storage system, and control the object gateway to perform garbage collection tasks using the adjusted garbage collection parameters in a future time period, so as to clear the garbage data in the storage device.

[0073] In an embodiment of the present application, the object gateway can be controlled to send a garbage collection notification message to the storage application corresponding to the garbage collection task, so as to adjust the current state of the storage application to the garbage collection state. Furthermore, each object gateway selects a suitable storage application (OSD) according to the CRUSH algorithm when processing data. Generally speaking, an OSD is used to manage a disk, and eventually writes user data to the disk corresponding to the OSD through the selected OSD. Therefore, before the object gateway executes the garbage collection task, the embodiment of the present application first sends a garbage collection notification message to the OSD corresponding to the garbage collection task, and then controls the disk corresponding to the OSD to delete data. The OSD will write the received garbage collection notification message into the OSD log, so that it can be read through the OSD log that the OSD needs to perform garbage collection, and then the current state of the OSD is adjusted to the garbage collection state.

[0074] In an embodiment of the present application, after the object gateway receives the data to be stored from the client, the object gateway may be prohibited from sending the data to be stored to a storage application that is currently in a garbage collection state. The embodiment of the present application adds the garbage collection state of the disk to the OSD selection strategy of the object gateway and removes the OSD that is currently in the garbage collection state when the strategy selects the OSD, thereby avoiding the problem that the OSD that is in the garbage collection state has to perform both data processing and garbage collection, which affects the garbage collection efficiency.

[0075] In order to timely recycle the fragmented space of the disk, avoid the waste of storage resources and ensure the sufficiency of the storage space used by the business, and solve the problem of not over-consuming system resources such as disk, CPU and memory when the garbage collection queue is congested, causing client request blocking, and thus reducing the service availability, the embodiment of the present application proposes a dynamic adjustment method for garbage collection parameters, based on historical traffic data combined with LSTM neural network (traffic prediction model), predicting the traffic trend data of the business, and then dynamically, in real time and globally adjusting the garbage collection parameters of the object gateway according to the predicted traffic trend data, and also according to the actual performance of different machines (storage devices), the garbage collection parameters of the object gateway are adjusted in a differentiated and customized manner, so as to achieve staggered recycling of garbage data, improve garbage collection efficiency, avoid high-intensity recycling of garbage data during peak traffic periods, cause garbage collection queue congestion, and affect service availability. In addition, when the object gateway selects a suitable disk to store data, the embodiment of the present application increases the state recognition of the disk, selects the OSD whose current state is not the garbage collection state, and avoids the adverse effects of garbage data recycling on data reading and writing.

[0076] The present application also provides a garbage collection device. Figure 4 , which is a structural diagram of a garbage collection device according to an embodiment of the present application.

[0077] The garbage recycling device comprises:

[0078] The monitoring module 410 is used to monitor the traffic history data of the distributed storage system and monitor the resource usage data of the storage devices in the distributed storage system.

[0079] The prediction module 420 is used to input the traffic history data into a pre-trained traffic prediction model so that the traffic prediction model predicts the traffic trend data of a future time period based on the traffic history data.

[0080] The determination module 430 is used to determine the garbage collection parameters corresponding to the future time period according to the traffic trend data.

[0081] The adjustment module 440 is configured to adjust the garbage collection parameters corresponding to the future time period according to the resource usage data.

[0082] The recycling module 450 is used to send the adjusted garbage collection parameters to the object gateway in the distributed storage system, and control the object gateway to perform garbage collection tasks using the adjusted garbage collection parameters within the future time period to clear garbage data in the storage device.

[0083] The functions of the device described in the embodiment of the present application have been described in the above method embodiment. Therefore, for any details not provided in the description of this embodiment, please refer to the relevant descriptions in the above embodiment and will not be repeated here.

[0084] The present application also provides a garbage collection device, such as Figure 5 , which is a structural diagram of a garbage collection device according to an embodiment of the present application.

[0085] The garbage collection device includes: a processor 510 , a communication interface 520 , a memory 530 and a communication bus 540 . The processor 510 , the communication interface 520 , and the memory 530 communicate with each other via the communication bus 540 .

[0086] The memory 530 is used to store computer programs.

[0087] In one embodiment of the present application, the processor 510, when executing the program stored in the memory 530, implements the garbage collection method provided by any of the aforementioned method embodiments, including: monitoring the traffic history data of the distributed storage system and monitoring the resource usage data of the storage device in the distributed storage system; inputting the traffic history data into a pre-trained traffic prediction model so that the traffic prediction model predicts the traffic trend data of the future time period based on the traffic history data; determining the garbage collection parameters corresponding to the future time period based on the traffic trend data; adjusting the garbage collection parameters corresponding to the future time period based on the resource usage data; sending the adjusted garbage collection parameters to the object gateway in the distributed storage system, and controlling the object gateway to perform garbage collection tasks using the adjusted garbage collection parameters within the future time period, so as to clear the garbage data in the storage device.

[0088] Among them, determining the garbage collection parameters corresponding to the future time period based on the traffic trend data includes: using a preset linear function to determine the garbage collection parameters corresponding to the traffic trend data in the future time period; or using a pre-trained parameter determination model to determine the garbage collection parameters corresponding to the traffic trend data in the future time period.

[0089] Among them, adjusting the garbage collection parameters of the future time period according to the resource usage data includes: in a preset decision tree, determining the leaf nodes that the resource usage data meets; wherein each node in the decision tree corresponds to a judgment condition; each leaf node corresponding to the node is a judgment result corresponding to the judgment condition; the judgment result includes an adjustment method and an adjustment value; adjusting the garbage collection parameters according to the adjustment method and the adjustment value corresponding to the leaf node that the resource usage data meets.

[0090] Among them, the monitoring of the traffic history data of the distributed storage system and the monitoring of the resource usage data of the storage devices in the distributed storage system include: monitoring the traffic history data corresponding to the target storage device group in the distributed storage system and monitoring the resource usage data of the target storage device group; wherein the target storage device group includes at least one storage device; the target storage device group corresponds to at least one object gateway, and the object gateways corresponding to the target storage device group and other storage device groups are different; the sending of the adjusted garbage collection parameters to the object gateways in the distributed storage system includes: sending the adjusted garbage collection parameters to each of the object gateways corresponding to the target storage device group.

[0091] Among them, the monitoring of the traffic history data of the distributed storage system and the monitoring of the resource usage data of the storage devices in the distributed storage system include: monitoring the traffic history data corresponding to the storage devices within a preset range in the distributed storage system and monitoring the resource usage data of the storage devices within the preset range; wherein, for the storage devices within the preset range, the storage devices are connected to each object gateway in the distributed storage system; the sending of the adjusted garbage collection parameters to the object gateways in the distributed storage system includes: sending the adjusted garbage collection parameters to each of the object gateways connected to the storage devices within the preset range.

[0092] Among them, after sending the adjusted garbage collection parameters to the object gateway in the distributed storage system, and before controlling the object gateway to use the adjusted garbage collection parameters to perform the garbage collection task within the future time period, it also includes: controlling the object gateway to send a garbage collection notification message to the storage application corresponding to the garbage collection task, so as to adjust the current state of the storage application to the garbage collection state.

[0093] The method further includes: after the object gateway receives the data to be stored from the client, prohibiting the object gateway from sending the data to be stored to a storage application whose current state is a garbage collection state.

[0094] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the garbage collection method provided in any of the above method embodiments are implemented. Since the garbage collection method has been described in detail above, for any details not fully described in the description of this embodiment, please refer to the relevant descriptions in the above embodiments, and no further description will be given here.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0097] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0098] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A garbage collection method, characterized in that: include: Monitoring the historical traffic data of the distributed storage system and monitoring the resource usage data of the storage devices in the distributed storage system; Inputting the traffic history data into a pre-trained traffic prediction model so that the traffic prediction model predicts traffic trend data for a future time period based on the traffic history data; Determining garbage collection parameters corresponding to the future time period according to the traffic trend data; adjusting garbage collection parameters corresponding to the future time period according to the resource usage data; The adjusted garbage collection parameters are sent to the object gateway in the distributed storage system, and the object gateway is controlled to perform garbage collection tasks using the adjusted garbage collection parameters within the future time period, so as to clear the garbage data in the storage device.

2. The method according to claim 1, characterized in that The determining, according to the traffic trend data, the garbage collection parameter corresponding to the future time period includes: Determine the garbage collection parameter corresponding to the traffic trend data in the future time period by using a preset linear function; or, The pre-trained parameter determination model is used to determine the garbage collection parameters corresponding to the traffic trend data in the future time period.

3. The method according to claim 1, characterized in that The adjusting the garbage collection parameters for the future time period according to the resource usage data includes: In a preset decision tree, determine the leaf nodes that the resource usage data meets; wherein each node in the decision tree corresponds to a judgment condition; each leaf node corresponding to the node is a judgment result corresponding to the judgment condition; the judgment result includes an adjustment method and an adjustment value; The garbage collection parameters are adjusted according to the adjustment method and the adjustment value corresponding to the leaf node to which the resource usage data conforms.

4. The method according to claim 1, characterized in that The monitoring of the traffic history data of the distributed storage system and the monitoring of the resource usage data of the storage devices in the distributed storage system include: Monitoring traffic history data corresponding to a target storage device group in a distributed storage system and monitoring resource usage data of the target storage device group; The target storage device group includes at least one storage device; the target storage device group corresponds to at least one object gateway, and the target storage device group and other storage device groups have different corresponding object gateways; The step of sending the adjusted garbage collection parameter to the object gateway in the distributed storage system includes: The adjusted garbage collection parameter is sent to each of the object gateways corresponding to the target storage device group.

5. The method according to claim 1, characterized in that The monitoring of the traffic history data of the distributed storage system and the monitoring of the resource usage data of the storage devices in the distributed storage system include: Monitoring the traffic history data corresponding to the storage devices within a preset range in the distributed storage system and monitoring the resource usage data of the storage devices within the preset range; wherein, for the storage devices within the preset range, the storage devices are connected to each object gateway in the distributed storage system; The step of sending the adjusted garbage collection parameter to the object gateway in the distributed storage system includes: The adjusted garbage collection parameter is sent to each of the object gateways connected to the storage devices within the preset range.

6. The method according to claim 1, characterized in that After sending the adjusted garbage collection parameters to the object gateway in the distributed storage system and before controlling the object gateway to perform garbage collection tasks using the adjusted garbage collection parameters in the future time period, the method further includes: The object gateway is controlled to send a garbage collection notification message to a storage application corresponding to the garbage collection task, so as to adjust a current state of the storage application to a garbage collection state.

7. The method according to claim 6, characterized in that The method further comprises: After the object gateway receives the data to be stored from the client, the object gateway is prohibited from sending the data to be stored to a storage application whose current state is a garbage collection state.

8. A garbage collection device, characterized in that: include: A monitoring module, used to monitor the traffic history data of the distributed storage system and the resource usage data of the storage devices in the distributed storage system; A prediction module, used for inputting the traffic history data into a pre-trained traffic prediction model, so that the traffic prediction model predicts the traffic trend data of a future time period based on the traffic history data; A determination module, configured to determine garbage collection parameters corresponding to the future time period according to the traffic trend data; An adjustment module, configured to adjust garbage collection parameters corresponding to the future time period according to the resource usage data; The recycling module is used to send the adjusted garbage collection parameters to the object gateway in the distributed storage system, and control the object gateway to perform garbage collection tasks using the adjusted garbage collection parameters within the future time period so as to clear the garbage data in the storage device.

9. A garbage recycling device, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to: execute a garbage collection program stored in the memory to implement the garbage collection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed to implement the garbage collection method according to any one of claims 1 to 7.

Images