Method and device for determining control bandwidth, storage medium and electronic equipment

Abstract

The invention discloses a control bandwidth determination method and device, a storage medium and electronic equipment, and relates to the field of storage systems.The method comprises the steps that the total downrefreshing data size in the process of executing each historical downrefreshing event and downrefreshing time consumption of each historical downrefreshing event are determined, determining a mean bandwidth corresponding to the plurality of historical downbrushing events according to the total downbrushing data volume and the downbrushing time consumption, and determining a reference bandwidth according to the mean bandwidth and a real-time bandwidth in the garbage collection execution process; according to the reference bandwidth and the residual space of the buffer pool, the control bandwidth in the process of writing the target data block into the data pool from the buffer pool can be determined. Through the embodiment of the invention, the problem that a large amount of resources of the storage system are consumed due to the fact that the buffer pool continuously accumulates the data which are not downswiped in the data downswipe process of the storage system in the related technology can be solved.

Description

Technical Field

[0001] The present application relates to the field of storage systems, and in particular to a method and device for determining control bandwidth, a storage medium, and an electronic device. Background Art

[0002] In scenarios where distributed storage uses erasure correction for fault tolerance, append-write processing is often used to avoid write amplification issues caused by erasure correction. Non-stripe-aligned write I / O requests are first persisted in the buffer pool. Multiple non-stripe-aligned I / O data is then aggregated and written to the data pool in a stripe-aligned manner. Stripe-aligned writes to the data pool improve overall data write performance. This approach can provide higher storage performance. However, in actual implementations, writing data to the buffer pool and writing stripe-aligned data to the data pool are typically performed by separate threads. To improve buffer pool performance, higher-performance storage hardware and more efficient data persistence methods are typically used. Consequently, buffer pool write performance is typically significantly higher than stripe-aligned write performance. This performance mismatch can lead to a continuous accumulation of unflushed data in the buffer pool, resulting in significant consumption of storage system resources and causing issues such as insufficient memory and buffer space, impacting the performance and stability of the entire distributed storage system.

[0003] Therefore, the problem that during the process of flushing data, the buffer pool of the storage system in the related art will continuously accumulate unflushed data, resulting in a large consumption of storage system resources, has not yet been effectively solved. Summary of the Invention

[0004] The present application provides a method and apparatus for determining control bandwidth, a storage medium, and an electronic device to at least solve the problem in the related art that, during the process of refreshing data in a storage system, a buffer pool continuously accumulates unflushed data, resulting in a large consumption of storage system resources.

[0005] The present application provides a method for determining a control bandwidth, comprising: calculating a total amount of flushed data written from a buffer pool to a data pool and a flushing time corresponding to each historical flushing event during the execution of each historical flushing event, wherein the historical flushing event is an event in which a data block is flushed from the buffer pool to the data pool within a past time period; determining an average bandwidth corresponding to multiple historical flushing events based on the total amount of flushed data and the flushing time, and determining a baseline bandwidth corresponding to a target flushing event based on the average bandwidth and the real-time bandwidth during garbage collection, wherein the target flushing event is an event in which a target data block is flushed from the buffer pool to the data pool; determining a control bandwidth corresponding to the target flushing event based on the baseline bandwidth and the remaining space of the buffer pool, and executing the target flushing event based on the control bandwidth.

[0006] The present application also provides a device for determining a control bandwidth, comprising: a calculation module, configured to calculate a total amount of flushed data written from a buffer pool to a data pool and a flushing time corresponding to each historical flushing event during the execution of each historical flushing event, wherein the historical flushing event is an event in which a data block is flushed from the buffer pool to the data pool within a past time period; a first determination module, configured to determine an average bandwidth corresponding to multiple historical flushing events based on the total amount of flushed data and the flushing time, and to determine a baseline bandwidth corresponding to a target flushing event based on the average bandwidth and the real-time bandwidth during garbage collection, wherein the target flushing event is an event in which a target data block is flushed from the buffer pool to the data pool; a second determination module, configured to determine a control bandwidth corresponding to the target flushing event based on the baseline bandwidth and the remaining space of the buffer pool, and to execute the target flushing event based on the control bandwidth.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for determining a control bandwidth when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for determining the control bandwidth are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned control bandwidth determination methods when executed by a processor.

[0010] Through the present application, the total amount of data flushed from the buffer pool to the data pool and the flushing time corresponding to each historical flushing event are calculated during the execution of each historical flushing event; the average bandwidth corresponding to multiple historical flushing events is determined based on the total amount of data flushed and the flushing time, and the benchmark bandwidth corresponding to the target data block flushed from the buffer pool to the data pool is determined based on the average bandwidth and the real-time bandwidth during the garbage collection process; the control bandwidth corresponding to the target data block flushed from the buffer pool to the data pool is determined based on the benchmark bandwidth and the remaining space in the buffer pool, and the target data block is flushed to the data pool based on the control bandwidth. In other words, the embodiment of the present application determines the total amount of data flushed and the flushing time corresponding to each historical flushing event during the execution of each historical flushing event, and then determines the average bandwidth corresponding to multiple historical flushing events based on the total amount of data flushed and the flushing time, and determines the benchmark bandwidth based on the average bandwidth and the real-time bandwidth during the garbage collection process; the control bandwidth in the process of flushing the target data block from the buffer pool to the data pool can be determined based on the benchmark bandwidth and the remaining space in the buffer pool. The embodiments of the present application can solve the problem in the related art that during the process of flushing data in the storage system, the buffer pool will continuously accumulate unflushed data, resulting in a large consumption of storage system resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 This is a hardware structure block diagram of a computer terminal for a method for determining a control bandwidth according to an embodiment of the present application;

[0013] Figure 2 is a flow chart of a method for determining a control bandwidth according to an embodiment of the present application;

[0014] Figure 3 This is a schematic diagram of determining the total amount of data to be refreshed according to an embodiment of the present application;

[0015] Figure 4 This is a structural block diagram of a device for determining a control bandwidth according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0017] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0018] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0019] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the method for determining the control bandwidth depends, the specific application environment architecture or the specific hardware architecture is described herein.

[0020] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a method of determining a control bandwidth according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor (MicroProcessing Unit, abbreviated as MPU) or a programmable logic device (Field-Programmable Gate Array, abbreviated as FPGA) and a memory 104 for storing data. The computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0021] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the interactive state in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0022] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0023] The embodiments of the present application provide a method for determining a control bandwidth. The following is an explanation of the technical terms involved in the embodiments of the present application:

[0024] Buffer pool: Front-end business requests first arrive at the buffer pool, which aggregates data from multiple requests and writes them to the data pool in a stripe-aligned manner.

[0025] Data pool: stores data and processes data read and write requests.

[0026] Figure 2 This is a flow chart of a method for determining a control bandwidth according to an embodiment of the present application, which can be applied to Figure 1 In a computer terminal, such as Figure 2 As shown, the process includes the following steps:

[0027] Step S202, calculating the total amount of data flushed from the buffer pool to the data pool and the flushing time corresponding to each historical flushing event during the execution of each historical flushing event, wherein the historical flushing event is an event in which a data block is flushed from the buffer pool to the data pool in a past time period;

[0028] Among them, the total amount of data flushed from the buffer pool to the data pool during the execution of each historical flush event means: while executing each historical flush event, there may be data blocks flushed from the buffer pool, and the sum of the data volume of these data blocks and the data volume of the data blocks corresponding to each historical flush event is the total amount of data flushed.

[0029] Step S204: determining an average bandwidth corresponding to multiple historical flush events based on the total flush data volume and the flush time, and determining a baseline bandwidth corresponding to a target flush event based on the average bandwidth and the real-time bandwidth during garbage collection, wherein the target flush event is an event of flushing a target data block from the buffer pool to the data pool;

[0030] That is to say, on the one hand, the buffer pool needs to aggregate the data written by the front-end business and flush it to the data pool; on the other hand, it also needs to flush the aggregated data generated by garbage collection to the data pool.

[0031] Step S206 : determining a control bandwidth corresponding to the target refresh event according to the reference bandwidth and the remaining space in the buffer pool, and executing the target refresh event based on the control bandwidth.

[0032] Through the method for determining the control bandwidth of the present application, the total amount of data flushed during the execution of each historical flush event and the flushing time of each historical flush event are determined, and then the average bandwidth corresponding to multiple historical flush events is determined based on the total amount of data flushed and the flushing time, and the benchmark bandwidth is determined based on the average bandwidth and the real-time bandwidth during the garbage collection process; the control bandwidth in the process of flushing the target data block from the buffer pool to the data pool can be determined based on the benchmark bandwidth and the remaining space in the buffer pool. Through the embodiments of the present application, the problem in the related art that the buffer pool of the storage system will continuously accumulate unflushed data during the data flushing process, resulting in a large consumption of storage system resources, can be solved.

[0033] Optionally, the calculation of the total amount of data flushed from the buffer pool to the data pool during the execution of each historical flush event in step S202 includes: determining a first data amount of first data that has been added to the to-be-flushed list and has not yet been flushed before the execution of each historical flush event, and determining a second data amount of second data that has been added to the to-be-flushed list and has not yet been flushed after the execution of each historical flush event; determining a third data amount of the data block corresponding to each historical flush event; and determining the total amount of data flushed according to a first formula, wherein the first formula is: d m =M0+M m +M1,d m is the total amount of data to be refreshed, M0 is the first amount of data, Mm is the third data amount, and M1 is the second data amount.

[0034] It can be understood that the above-mentioned first data volume is the total data volume of all data blocks that already existed in the queue to be flushed of the buffer pool before the start of each historical flush event but had not completed the step of flushing and writing to the data pool; the above-mentioned second data volume is the total data volume of data blocks newly added to the queue to be flushed of the buffer pool after the end of each historical flush event but have not yet started to be flushed.

[0035] Figure 3 is a schematic diagram of determining the total amount of data to be refreshed according to an embodiment of the present application. Figure 3 As shown: Figure 3 It contains d0-d67 data blocks, the refresh start time of each data block can be represented by t0, the refresh end time of each data block can be represented by t1, and the data size of each data block can be represented by m.

[0036] in, Figure 3 In the figure, d3 starts to be flushed from time T0. Before the flushing starts, there are already three data blocks d0, d1 and d2 in the queue (i.e., the queue to be flushed in the buffer pool). The flushing of d3 ends at time T1. Between T0 and T1, d4, d5 and d6 are being flushed, and the flushing is not completed at time T1.

[0037] The total amount of data to be flushed at this time should be the sum of the data sizes of d0-d67 data blocks. m =M0+M m +M1, M0 is the sum of the data sizes of the three data blocks d0, d1 and d2, M m is the data size of d3, and M1 is the sum of the data sizes of the three data blocks d4, d5 and d6.

[0038] Optionally, the calculation of the refresh time corresponding to each historical refresh event in step S202 includes: obtaining refresh attribute information corresponding to each historical refresh event, wherein the refresh attribute information includes: the refresh start time corresponding to each historical refresh event and the refresh end time corresponding to each historical refresh event; determining the refresh time corresponding to each historical refresh event according to a second formula, wherein the second formula is: d t =T1-T0,d t is the time taken for the next refresh, T1 is the end time of the next refresh, and T0 is the start time of the next refresh.

[0039] It is understandable that the flushing time corresponding to each historical flushing event also needs to be determined. Specifically: the flushing start time of each historical flushing event needs to be determined, that is, the time point when the data block starts to be written from the buffer pool to the data pool; and the flushing end time of each historical flushing event needs to be determined, that is, the time when the data block is completely written to the data pool and the write is confirmed to be successful.

[0040] The time taken to refresh the image can be determined by the difference between the refresh end time and the refresh start time.

[0041] Optionally, the step S204 of determining the average bandwidth corresponding to multiple historical refresh events according to the total refresh data volume and the refresh time includes:

[0042] (1) Calculating the second bandwidth corresponding to each historical refresh event according to the total refresh data volume and the refresh time, and adding multiple second bandwidths to the bandwidth list, specifically:

[0043] Calculating the second bandwidth corresponding to each historical refresh event based on the total refresh data volume and the refresh time includes: determining a third size relationship between the total refresh data volume and the third data volume of the data block corresponding to each historical refresh event; when the third size relationship indicates that the total refresh data volume is equal to the third data volume, determining the preset maximum bandwidth as the second bandwidth corresponding to each historical refresh event; when the third size relationship indicates that the total refresh data volume is greater than the third data volume, determining the second bandwidth according to a fourth formula, wherein the fourth formula is: b=d m ÷d t , b is the second bandwidth, d m is the total amount of data refreshed, d t As mentioned above, the refresh is time-consuming.

[0044] It is understandable that after the total refresh data volume and refresh time are determined, the second bandwidth corresponding to each historical refresh event can be determined according to the total refresh data volume and refresh time. Specifically:

[0045] Determine the third size relationship between the total flush data volume and the third data volume of the data block corresponding to each historical flush event. That is, compare the total amount of data written from the buffer pool to the data pool during each historical flush event with the actual amount of data processed in each historical flush event. The following situations may exist:

[0046] 1) If the total flushed data volume is equal to the third data volume, M0 and M1 can be determined to be both 0. At this time, during the execution of the historical flush event, the data pool's write speed is sufficient to handle all pending data in the buffer pool, and there are no additional concurrent processing tasks. In other words, the current data pool write performance far exceeds the front-end business write rate, and the system does not have a significant write bottleneck.

[0047] When the write capability of the data pool is strong enough to meet or even exceed the demand, the system can use the maximum bandwidth (i.e., the preset maximum bandwidth) to flush data to fully utilize the write performance of the data pool.

[0048] 2) When it is determined that the total amount of data refreshed is greater than the third amount of data, it can be determined that at least one of M0 and M1 is not 0. At this time, it can be determined that in addition to executing the current historical refresh event, there are other data blocks that are being refreshed concurrently or are ready but have not started to be refreshed, that is, there is a certain degree of write concurrency in the system.

[0049] When the above-mentioned write concurrency occurs, the second bandwidth can be determined by the quotient of the total amount of data refreshed and the refresh time.

[0050] By calculating the second bandwidth of each historical refresh event through the above embodiment, the system can identify the actual write speed of the data pool and its trend over time, thereby monitoring the write performance of the data pool in real time and promptly identifying performance bottlenecks or anomalies.

[0051] (2) Determine a first size relationship between the number of bandwidths included in the bandwidth list and the preset number of bandwidths. Specifically, there are the following two situations:

[0052] 1) When the first size relationship indicates that the bandwidth quantity is not equal to the preset bandwidth quantity, deleting some bandwidths in the bandwidth list or supplementing bandwidths in the bandwidth list so that the bandwidth quantity is equal to the preset bandwidth quantity, specifically:

[0053] In the case where the first size relationship indicates that the bandwidth quantity is greater than the preset bandwidth quantity, determine the refresh end time of each historical refresh event corresponding to each bandwidth in the bandwidth list, and determine the second size relationship between the refresh end time and the preset time; determine a target time earlier than the preset time among multiple refresh end times according to the second size relationship, and delete the second bandwidth of the historical refresh event corresponding to the target time from the bandwidth list; in the case where the first size relationship indicates that the bandwidth quantity is less than the preset bandwidth quantity, determine a second difference between the preset bandwidth quantity and the bandwidth quantity, and determine the second difference as the target quantity of bandwidth to be added to the bandwidth list; add the preset maximum bandwidth of the target quantity to the bandwidth list to make the bandwidth quantity equal to the preset bandwidth quantity.

[0054] It is understandable that, when it is determined that the bandwidth quantity of the bandwidths in the bandwidth list is greater than or less than the preset bandwidth quantity, the bandwidths in the bandwidth list may be adjusted so that the bandwidth quantity of the bandwidths in the bandwidth list is equal to the preset bandwidth quantity. Specifically:

[0055] If the number of bandwidths in the bandwidth list is greater than the preset number of bandwidths, the refresh end time of each historical refresh event in the bandwidth list is checked and compared with the preset time to identify data points that do not meet the timeliness of the current analysis. The preset time should reflect the length of the bandwidth statistics cycle that the system intends to maintain. Once all historical refresh events that are earlier than the preset time are identified, the second bandwidth of the historical refresh event is deleted from the bandwidth list. This maintains the timeliness and relevance of the bandwidth list and ensures that the calculated average bandwidth reflects the recent write performance status of the data pool.

[0056] When it is determined that the number of bandwidths in the bandwidth list is less than the preset number of bandwidths, the second difference between the preset number of bandwidths and the number of bandwidths in the current bandwidth list is calculated. The second difference represents the number of historical refresh events that are missing from the list to achieve the preset statistical sample size. The preset maximum bandwidth value of the target number is added to the bandwidth list until the number of bandwidths in the list is equal to the preset number of bandwidths. Among them, the preset maximum bandwidth can be the ideal upper limit of the write speed of the data pool. When there are insufficient samples in the bandwidth list, the maximum bandwidth is used to fill it, which can prevent statistical errors caused by insufficient sample size and ensure that the calculated mean bandwidth is within a reasonable range.

[0057] This technical solution of dynamically adjusting the number of samples in the bandwidth list ensures that it reflects the recent write performance of the data pool while maintaining an appropriate statistical sample size. This ensures that the bandwidth list remains timely and relevant, ensuring that the calculated average bandwidth reflects the recent write performance of the data pool while also preventing statistical errors caused by insufficient sample size and ensuring that the calculated average bandwidth remains within a reasonable range.

[0058] 2) When the first size relationship indicates that the number of bandwidths is equal to the preset number of bandwidths, determining the refresh end time of the historical refresh event corresponding to each bandwidth in the bandwidth list, and calculating a first difference between the current time and the refresh end time; determining the mean bandwidth according to a third formula, wherein the third formula is: B avg is the mean bandwidth, B i is the i-th bandwidth in the bandwidth list, r is the attenuation parameter, r is used to indicate the influence of the first difference on the weight of each bandwidth in the bandwidth list, t0 is the current time, t i is the refresh end time of the historical refresh event corresponding to the i-th bandwidth in the bandwidth list, (t0-t i ) is the first difference.

[0059] It is understandable that, when it is determined that the number of bandwidths is equal to the preset number of bandwidths, a weighted average calculation may be performed on the bandwidths in the bandwidth list to determine the average bandwidth of the data pool. Specifically:

[0060] Calculate the first difference between the current time and the refresh end time of each historical refresh event. The first difference reflects the age of each historical refresh event, that is, the time difference between each historical refresh event and the current time. The smaller the difference, the newer the historical refresh event. i ), the decay parameter r is also introduced, which determines the degree of influence of the first difference on the weight. The larger the decay parameter, the greater the negative impact of the time difference on the weight. The weight is calculated by the exponential decay function, and the formula is The mean bandwidth is then determined using the third formula above.

[0061] The above average bandwidth calculation method fully considers the timeliness and importance of each historical refresh event, and thus can provide a more accurate data pool write performance prediction, helping front-end businesses to more reasonably plan data write speeds and avoid resource waste or overload.

[0062] Optionally, the above-mentioned step S206 determines the control bandwidth corresponding to the target refresh event based on the baseline bandwidth and the remaining space of the buffer pool, including: determining a fourth size relationship between the space size of the remaining space and the preset space size; when the fourth size relationship indicates that the space size is greater than the preset space size, determining the sum of the baseline bandwidth and the target value as the control bandwidth; when the fourth size relationship indicates that the space size is less than or equal to the preset space size, determining the baseline bandwidth as the control bandwidth.

[0063] It is understandable that after determining the baseline bandwidth, the remaining space in the buffer pool must also be considered. The control bandwidth is determined based on the remaining space in the buffer pool. Specifically:

[0064] Compare the remaining space in the buffer pool to the preset size. The preset size indicates that the buffer pool can maintain adequate write performance and stability without prematurely triggering garbage collection (GC) or other resource-consuming operations that could degrade system performance.

[0065] When the buffer pool's capacity exceeds the preset capacity, it has sufficient space to receive more write requests from front-end services without requiring an immediate flush. Therefore, the baseline bandwidth can be added to a target value, and the resulting sum is used as the control bandwidth. The target value can be a preset increment, designed to allow the buffer pool to receive more write requests when sufficient remaining space is available to cope with sudden business traffic. Increasing the control bandwidth means the buffer pool can receive and process front-end business data more quickly, thereby improving the throughput of the entire storage system.

[0066] If the buffer pool size is less than or equal to the preset size, it indicates that the buffer pool is running low and the write speed of front-end services needs to be controlled to prevent space exhaustion. In this case, the baseline bandwidth can be used as the control bandwidth. This means that the write speed of front-end services needs to be more strictly limited to ensure that data can be flushed smoothly to the data pool and prevent buffer pool space exhaustion.

[0067] The above technical solution can dynamically adjust the control bandwidth according to the remaining space in the buffer pool, fully utilize the high write performance of the buffer pool when there is sufficient space, and reasonably control the write speed of the buffer pool when space is tight, thereby ensuring stable operation of the system.

[0068] In order to better understand the process of the above-mentioned method for determining the control bandwidth, the implementation process of the above-mentioned method for determining the control bandwidth is described below in combination with an optional embodiment, but it is not intended to limit the technical solution of the embodiment of the present application.

[0069] In the related art, the process of writing data to the buffer pool and writing stripe-aligned data to the data pool is usually completed by independent threads. In order to improve the performance of the buffer pool, higher-performance storage hardware is usually configured, and a more efficient data persistence method is used. The write performance of the buffer pool is usually much greater than the write performance of the stripe-aligned data pool. Due to the mismatch in performance, the buffer pool will continue to accumulate unflushed data, resulting in a large consumption of storage system resources, causing problems such as insufficient memory and insufficient buffer space, affecting the performance and stability of the entire distributed storage system. For the above reasons, the system will usually actively limit the write performance of the buffer pool when the memory consumption is high or the buffer space consumption is high. For example, a fixed system bandwidth is calculated based on the memory consumption, buffer space consumption, etc., and the speed of the buffer pool request is controlled accordingly. This control method cannot use system resources more smoothly, nor can it make more effective use of the preset total amount of resources, and the performance results of the speed control are not smooth enough.

[0070] Based on the above issues, an optional embodiment of the present application proposes a method for coordinating the performance of the buffer pool and the data pool in the append write mode. Through the statistical analysis of the processing performance of the buffer pool data flushing request by each buffer pool storage service, the data pool flushing processing bandwidth is estimated, and this is used as the buffer pool processing bandwidth value to control the bandwidth of the front-end business write. In this way, the data flushing process does not need to accumulate a large amount of data in the memory, the buffer space can be used to a higher level, and because the bandwidth is basically fixed, the control of the front-end business speed is also smoother.

[0071] The method for performance coordination between the buffer pool and the data pool in the append write mode proposed in the optional embodiment of the present application mainly involves statistics on the flushing performance of the data pool in the buffer pool, the buffer pool estimates the data pool writing performance based on the flushing performance statistics, and the buffer pool limits the speed of the front-end business based on the data writing performance. Specifically, it includes: the buffer pool samples the flushing write time and the amount of data in the flushing from the beginning to the end of the flushing write in the process of flushing data, and calculates a sampling bandwidth; averages the most recent multiple sampling bandwidths to obtain the average bandwidth; calculates the available baseline bandwidth for the front-end business with the average bandwidth and the current real-time statistics of the garbage collection write bandwidth; then calculates the actual control bandwidth of the front-end business based on the currently remaining available buffer space; and controls the speed of the front-end business with the actual control bandwidth.

[0072] The main steps for the buffer pool to collect statistics on the data pool flushing performance include:

[0073] (1) After the buffer pool aggregates the data, it is added to the list of data to be flushed, and the flushing statistical attributes are added to the aggregated data, including the data flushing start time T0, the data flushing completion time T1, the data size Mm ;

[0074] (2) The buffer pool records the amount of data being flushed. After the aggregate data starts to be flushed, the data size of the aggregate data is added to the amount of data being flushed. After the aggregate data is flushed, the data size of the aggregate data is subtracted from the amount of data being flushed.

[0075] For a specific data to be flushed, the flush start time is T0 (i.e., the flush start time), and the flush writing completion time is T1 (i.e., the flush end time). m (ie, the third amount of data); before the next refresh begins, the amount of data in the next refresh is M0 (ie, the first amount of data, excluding M m ), after the download is completed, the amount of data in the download is M1 (i.e. the second amount of data, excluding M m ); Sampling and refreshing information, the refreshing time is: d t =T1-T0, the total amount of data refreshed during the period d m =M0+M m +M1; please refer to Figure 3 As shown, data block d3 is queued at T0 and starts to be flushed. At this time, data blocks d0, d1, and d2 are already in the queue and are being flushed. Data block d3 is dequeued at T1 after being written. Between T0 and T1, data blocks d4, d5, and d6 are queued and remain in the queue.

[0076] (3) When the sampling information d m Contains only its own information (M m ) (that is, when the third size relationship indicates that the total amount of data to be flushed is equal to the third amount of data), it indicates that the current business write performance is lower than the data pool write performance, and no parallel processing tasks are generated, then the maximum value B max (i.e. preset maximum bandwidth) as the current bandwidth value;

[0077] (4) When the sampling information d m When the data contains more than its own information (ie, the value of M0 and / or M1 is not 0, that is, when the third size relationship indicates that the total amount of data to be refreshed is greater than the third amount of data), d m d t Calculate the current bandwidth value b=d m ÷d t ; Take b as the current bandwidth value (i.e., the second bandwidth);

[0078] (5) The current bandwidth value is added to the bandwidth calculation list (i.e., bandwidth list). When the number of samples in the bandwidth calculation list exceeds the threshold, the earliest sample value (i.e., the historical down-refresh event corresponding to the target time) is removed.

[0079] The main steps for estimating data pool write performance include:

[0080] (1) Before writing performance estimation into the data pool, the sampling values whose sampling time exceeds the threshold in the bandwidth calculation list are cleared, and only the sampling values within the threshold time are retained;

[0081] (2) If the number of samples in the current bandwidth calculation list is lower than the calculation threshold, the maximum value B max As a supplement;

[0082] (3) The bandwidth sampling values in the bandwidth calculation list are weighted according to the distance between the sampling time and the current time (i.e., the first difference). The closer the distance to the current time, the higher the weight, and the mean bandwidth B is obtained. avg , the average bandwidth is the estimated bandwidth of the data pool, specifically:

[0083] Assume there is a series of bandwidth values B1, B2, ..., B n , and their corresponding sampling times t1, t2, ..., t n , the current time is t0. Use the exponential decay function to define the weight, that is, the weight w i and Proportional to, where r is the parameter that controls the attenuation degree. Mean bandwidth B avg The calculation formula is:

[0084]

[0085] The main steps for the buffer pool to limit the speed of front-end services based on the calculated average bandwidth include:

[0086] (1) In the append write mode, the buffer pool needs to aggregate the data written by the front-end business and flush it to the data pool. On the other hand, it also needs to flush the aggregated data generated by garbage collection to the data pool. The average bandwidth B avg is the overall write performance of the data pool;

[0087] (2) In the append write mode, the background GC speed affects the system stability. Therefore, the background GC speed is prioritized. The real-time bandwidth of the GC process is counted by the GC module.

[0088] (3) According to the calculated mean bandwidth B avg Subtract the real-time bandwidth B counted by the GC module gc , which is the benchmark bandwidth B for front-end services f , that is, B f =B avg -B gc ;

[0089] (4) The reference bandwidth is the baseline for bandwidth control, specifically applied to the front-end control bandwidth B cIt can be adjusted according to the current remaining memory buffer (i.e. the size of the remaining space) and the device buffer size. When the remaining buffer is sufficient, B c Can be set higher than B f To cope with burst traffic, if the remaining buffer has reached the minimum buffer, then B c Set to B f , that is, control is performed according to the benchmark bandwidth.

[0090] (5) Front-end service bandwidth is based on B c The speed of the traffic can be limited by token bucket or other methods.

[0091] In summary, the buffer pool of the optional embodiment of the present application calculates the down-brush bandwidth of a single sampling by counting the total amount of data concurrently written by the data pool and the time taken to write the data this time; the buffer pool calculates the current average bandwidth of the data pool based on the down-brush bandwidth of multiple samples; then the buffer pool calculates the baseline bandwidth of the front-end business based on the average bandwidth and the real-time statistics of the garbage collection write bandwidth; and calculates the control bandwidth of the front-end business with reference to the current remaining situation of the buffer resources. The above technical solution can effectively estimate the bandwidth available for the front-end business, and use it as the basis for controlling the speed of the front-end business, so that the storage system does not need to accumulate a large amount of data in the memory, and the buffer space can also be used to a higher level. And because the bandwidth can be predicted, the control of the front-end business speed is also smoother.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0093] This embodiment also provides a device for determining a control bandwidth, which is used to implement the above embodiments and preferred implementations. Details that have already been described will not be repeated. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0094] Figure 4 is a structural block diagram of a device for determining a control bandwidth according to an embodiment of the present application, such as Figure 4 As shown, the device includes:

[0095] a calculation module 42 configured to calculate a total amount of data flushed from the buffer pool to the data pool and a flushing time corresponding to each historical flushing event during the execution of each historical flushing event, wherein the historical flushing event is an event in which a data block is flushed from the buffer pool to the data pool in a past time period;

[0096] a first determining module 44 configured to determine an average bandwidth corresponding to a plurality of historical flushing events based on the total flushing data volume and the flushing time, and to determine a baseline bandwidth corresponding to a target flushing event based on the average bandwidth and the real-time bandwidth during garbage collection, wherein the target flushing event is an event of flushing a target data block from the buffer pool to the data pool;

[0097] The second determining module 46 is configured to determine a control bandwidth corresponding to the target refresh event according to the reference bandwidth and the remaining space in the buffer pool, and execute the target refresh event based on the control bandwidth.

[0098] Through the control bandwidth determination device of the present application, the total amount of data flushed during the execution of each historical flush event and the flushing time of each historical flush event are determined, and then the average bandwidth corresponding to multiple historical flush events is determined based on the total amount of data flushed and the flushing time, and the benchmark bandwidth is determined based on the average bandwidth and the real-time bandwidth during the garbage collection process; the control bandwidth in the process of flushing the target data block from the buffer pool to the data pool can be determined based on the benchmark bandwidth and the remaining space in the buffer pool. Through the embodiments of the present application, the problem in the related art that the buffer pool of the storage system will continuously accumulate unflushed data during the data flushing process, resulting in a large consumption of storage system resources, can be solved.

[0099] In an exemplary embodiment, the calculation module 42 is further configured to determine a first data volume of first data that has been added to the to-be-flush list and has not yet been flushed before executing each historical flush event, and to determine a second data volume of second data that has been added to the to-be-flush list and has not yet been flushed after executing each historical flush event; determine a third data volume of the data block corresponding to each historical flush event; and determine the total flush data volume according to a first formula, wherein the first formula is: d m =M0+M m +M1,d m is the total amount of data to be refreshed, M0 is the first amount of data, M m is the third data amount, and M1 is the second data amount.

[0100] In an exemplary embodiment, the calculation module 42 is further configured to obtain the brush down attribute information corresponding to each historical brush down event, wherein the brush down attribute information includes: the brush down start time corresponding to each historical brush down event and the brush down end time corresponding to each historical brush down event; determine the brush down time corresponding to each historical brush down event according to a second formula, wherein the second formula is: d t =T1-T0,d t is the time taken for the next refresh, T1 is the end time of the next refresh, and T0 is the start time of the next refresh.

[0101] In an exemplary embodiment, the first determination module 44 is further configured to calculate the second bandwidth corresponding to each historical refresh event based on the total refresh data volume and the refresh time, and add multiple second bandwidths to a bandwidth list; determine a first size relationship between the number of bandwidths included in the bandwidth list and a preset number of bandwidths; if the first size relationship indicates that the number of bandwidths is not equal to the preset number of bandwidths, delete some bandwidths from the bandwidth list or supplement the bandwidths in the bandwidth list so that the number of bandwidths is equal to the preset number of bandwidths; if the first size relationship indicates that the number of bandwidths is equal to the preset number of bandwidths, determine the refresh end time of the historical refresh event corresponding to each bandwidth in the bandwidth list, and calculate a first difference between the current time and the refresh end time; and determine the mean bandwidth according to a third formula, wherein the third formula is: B avg is the mean bandwidth, B i is the i-th bandwidth in the bandwidth list, r is the attenuation parameter, r is used to indicate the influence of the first difference on the weight of each bandwidth in the bandwidth list, t0 is the current time, t i is the refresh end time of the historical refresh event corresponding to the i-th bandwidth in the bandwidth list, (t0-t i ) is the first difference.

[0102] In an exemplary embodiment, the first determination module 44 is further configured to, when the first size relationship indicates that the bandwidth quantity is greater than the preset bandwidth quantity, determine the refresh end time of each historical refresh event corresponding to each bandwidth in the bandwidth list, and determine a second size relationship between the refresh end time and the preset time; determine a target time earlier than the preset time from multiple refresh end times according to the second size relationship, and delete the second bandwidth of the historical refresh event corresponding to the target time from the bandwidth list; when the first size relationship indicates that the bandwidth quantity is less than the preset bandwidth quantity, determine a second difference between the preset bandwidth quantity and the bandwidth quantity, and determine the second difference as the target quantity of bandwidth to be added to the bandwidth list; and add the preset maximum bandwidth of the target quantity to the bandwidth list so that the bandwidth quantity is equal to the preset bandwidth quantity.

[0103] In an exemplary embodiment, the first determining module 44 is further configured to determine a third size relationship between the total refresh data amount and the third data amount of the data block corresponding to each historical refresh event; when the third size relationship indicates that the total refresh data amount is equal to the third data amount, determine the preset maximum bandwidth as the second bandwidth corresponding to each historical refresh event; when the third size relationship indicates that the total refresh data amount is greater than the third data amount, determine the second bandwidth according to a fourth formula, wherein the fourth formula is: b=d m ÷d t , b is the second bandwidth, d m is the total amount of data refreshed, d t As mentioned above, the refresh is time-consuming.

[0104] In an exemplary embodiment, the second determination module 46 is further used to determine a fourth size relationship between the space size of the remaining space and the preset space size; when the fourth size relationship indicates that the space size is greater than the preset space size, the sum of the baseline bandwidth and the target value is determined as the control bandwidth; when the fourth size relationship indicates that the space size is less than or equal to the preset space size, the baseline bandwidth is determined as the control bandwidth.

[0105] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the method for determining the control bandwidth.

[0106] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned embodiments of the method for determining the control bandwidth when running.

[0107] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0108] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned methods for determining the control bandwidth are implemented.

[0109] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned control bandwidth determination method embodiments are implemented.

[0110] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] The above describes in detail a method for determining a control bandwidth provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is intended only to facilitate understanding of the method and core concept of the present application. It should be noted that those skilled in the art may make various improvements and modifications to the present application without departing from the principles of the present application, and such improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for determining a control bandwidth, characterized in that: include: Calculate the total amount of data flushed from the buffer pool to the data pool and the flushing time corresponding to each historical flush event during the execution of each historical flush event, wherein the historical flush event is an event in which a data block is flushed from the buffer pool to the data pool in the past time period; Determining an average bandwidth corresponding to a plurality of historical flush events based on the total flush data volume and the flush time, and determining a baseline bandwidth corresponding to a target flush event based on the average bandwidth and the real-time bandwidth during garbage collection, wherein the target flush event is an event of flushing a target data block from the buffer pool to the data pool; A control bandwidth corresponding to the target refresh event is determined according to the reference bandwidth and the remaining space of the buffer pool, and the target refresh event is executed based on the control bandwidth.

2. The method for determining the control bandwidth according to claim 1, wherein: Calculate the total amount of data flushed from the buffer pool to the data pool during each historical flush event, including: Determine a first data volume of first data that has been added to the to-be-flush list and has not yet been flushed before executing each of the historical flush events, and determine a second data volume of second data that has been added to the to-be-flush list and has not yet been flushed after executing each of the historical flush events; Determining a third data volume of the data block corresponding to each historical refresh event; The total amount of refresh data is determined according to the first formula, wherein the first formula is: m =M0+M m +M1,d m is the total refresh data amount, M0 is the first data amount, c is the third data amount, and M1 is the second data amount.

3. The method for determining the control bandwidth according to claim 1, wherein: Calculating the refresh time corresponding to each historical refresh event includes: Acquire the refresh attribute information corresponding to each historical refresh event, wherein the refresh attribute information includes: the refresh start time corresponding to each historical refresh event and the refresh end time corresponding to each historical refresh event; The refresh time corresponding to each historical refresh event is determined according to a second formula, wherein the second formula is: d t =T1-T0,d t is the time taken for the next refresh, T1 is the end time of the next refresh, and T0 is the start time of the next refresh.

4. The method for determining the control bandwidth according to claim 1, wherein: Determining average bandwidths corresponding to multiple historical refresh events according to the total refresh data volume and the refresh time, including: Calculating the second bandwidth corresponding to each historical refresh event according to the total refresh data volume and the refresh time, and adding the plurality of second bandwidths to the bandwidth list; Determining a first size relationship between the number of bandwidths included in the bandwidth list and the preset number of bandwidths; If the first size relationship indicates that the bandwidth quantity is not equal to the preset bandwidth quantity, deleting some bandwidths in the bandwidth list or supplementing bandwidths in the bandwidth list so that the bandwidth quantity is equal to the preset bandwidth quantity; When the first size relationship indicates that the number of bandwidths is equal to the preset number of bandwidths, determining a refresh end time of a historical refresh event corresponding to each bandwidth in the bandwidth list, and calculating a first difference between a current time and the refresh end time; The mean bandwidth is determined according to a third formula, wherein the third formula is: B avg is the mean bandwidth, B i is the i-th bandwidth in the bandwidth list, r is the attenuation parameter, r is used to indicate the influence of the first difference on the weight of each bandwidth in the bandwidth list, t0 is the current time, t i is the refresh end time of the historical refresh event corresponding to the i-th bandwidth in the bandwidth list, (t0-t i ) is the first difference.

5. The method for determining the control bandwidth according to claim 4, wherein: Deleting some bandwidths in the bandwidth list or supplementing the bandwidths in the bandwidth list so that the number of bandwidths is equal to the preset number of bandwidths includes: When the first size relationship indicates that the number of bandwidths is greater than the preset number of bandwidths, determining a refresh end time of each historical refresh event corresponding to each bandwidth in the bandwidth list, and determining a second size relationship between the refresh end time and the preset time; determining a target time earlier than the preset time from a plurality of refresh end times according to the second size relationship, and deleting the second bandwidth of the historical refresh event corresponding to the target time from the bandwidth list; If the first size relationship indicates that the bandwidth quantity is less than the preset bandwidth quantity, determining a second difference between the preset bandwidth quantity and the bandwidth quantity, and determining the second difference as a target quantity of bandwidth to be added to the bandwidth list; The target number of preset maximum bandwidths is added to the bandwidth list so that the bandwidth number is equal to the preset bandwidth number.

6. The method for determining the control bandwidth according to claim 4, wherein: Calculating the second bandwidth corresponding to each historical refresh event according to the total refresh data volume and the refresh time includes: Determine a third size relationship between the total refresh data amount and the third data amount of the data block corresponding to each historical refresh event; When the third size relationship indicates that the total refresh data amount is equal to the third data amount, determining the preset maximum bandwidth as the second bandwidth corresponding to each historical refresh event; When the third size relationship indicates that the total refresh data amount is greater than the third data amount, the second bandwidth is determined according to a fourth formula, wherein the fourth formula is: b=d m ÷d t , b is the second bandwidth, d m is the total amount of data refreshed, d t As mentioned above, the refresh is time-consuming.

7. The method for determining the control bandwidth according to claim 1, wherein: Determining a control bandwidth corresponding to the target refresh event according to the reference bandwidth and the remaining space of the buffer pool includes: Determining a fourth size relationship between the size of the remaining space and the preset space size; If the fourth size relationship indicates that the space size is greater than the preset space size, determining a sum of the reference bandwidth and the target value as the control bandwidth; When the fourth size relationship indicates that the space size is smaller than or equal to the preset space size, the reference bandwidth is determined as the control bandwidth.

8. A device for determining a control bandwidth, characterized in that: include: a calculation module, configured to calculate a total amount of data flushed from the buffer pool to the data pool and a flushing time corresponding to each historical flushing event during the execution of each historical flushing event, wherein the historical flushing event is an event in which a data block is flushed from the buffer pool to the data pool within a past time period; a first determining module, configured to determine an average bandwidth corresponding to a plurality of historical flushing events based on the total flushing data volume and the flushing time, and determine a baseline bandwidth corresponding to a target flushing event based on the average bandwidth and a real-time bandwidth during garbage collection, wherein the target flushing event is an event of flushing a target data block from the buffer pool to the data pool; The second determining module is configured to determine a control bandwidth corresponding to the target refresh event according to the reference bandwidth and the remaining space of the buffer pool, and execute the target refresh event based on the control bandwidth.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for determining the control bandwidth as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for determining the control bandwidth as claimed in any one of claims 1 to 7 are implemented.

Images