A method, device and storage medium for controlling traffic under flushing of cached data

Abstract

The application relates to the technical field of distributed storage, and discloses a cache data flushing flow control method and device, equipment and a storage medium, which comprises the following steps: acquiring the maximum service bandwidth and the maximum IOPS that can be carried by an HDD device at the same time; before each service thread and / or flushing thread writes data to the HDD device, calculating the bandwidth resource amount and the IOPS resource amount required by the current write operation; judging whether the bandwidth resource amount required is not more than the maximum service bandwidth and whether the IOPS resource amount required is not more than the maximum IOPS; if yes, executing the write operation and releasing the occupied bandwidth resource and IOPS resource after the write operation is executed; and if no, waiting for the release of the bandwidth resource and the IOPS resource. In this way, the HDD disk service bandwidth and IOPS can be simply and effectively controlled, the disk performance can be fully utilized, and the flushing performance of the cache data of an ICFS distributed layered storage system can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage, in particular to a cache data flushing flow control method and device, equipment and storage medium. BACKGROUND

[0002] The Inspur Cluster File System (ICFS) is a distributed file storage system, which is widely used in video monitoring, broadcasting media, biotechnology and high performance scenarios.

[0003] At present, in the ICFS storage system, a hierarchical storage architecture is usually used, that is, both a hard disk drive (HDD) device with large storage capacity and low price and a solid state disk (SSD) device with relatively small storage capacity but fast access speed and better random and concurrent performance are used. In order to improve the write business performance, a single object storage device (OSD) usually uses an SSD as a cache device and an HDD as a main storage device. After being judged, the write business issued from the upper layer to the OSD will be written to different devices. For example, as shown in FIG. 1, random small write business data will be written to the cache device, and sequential write and large block write business will be directly written to the main storage device; the data on the cache device will be flushed to the main storage device later. This strategy of selecting a write device according to the write business form can improve the overall response time of the write business request. Figure 1

[0004] However, the addition of the cache device also brings additional workload to the HDD main storage device. Due to the limited storage space of the SSD cache device, when the front-end business is written, the dirty data on the cache device needs to be constantly flushed to the HDD main storage device. The dirty data flushing brings additional write pressure to the HDD main storage device, which will compete with the business input / output (IO) for disk resources. Since the HDD main storage device can carry limited amount of business within a period of time, when the write pressure is too large, it will cause a series of problems such as business jamming and performance fluctuation.

[0005] Therefore, how to reduce the problems of HDD disk busy, business blocking and performance fluctuation caused by the additional write IO of flushing when multiple businesses and flushing threads write data to the HDD device at the same time is a technical problem to be solved by those skilled in the art. SUMMARY

[0006] ​Therefore, the application aims to provide a cache data flushing flow control method, device, equipment and storage medium, which can effectively improve the cache data flushing performance of the ICFS distributed hierarchical storage system and reduce the business performance fluctuation caused by flushing.

[0007] A cache data flushing flow control method applied to an ICFS distributed hierarchical storage system, comprising:

[0008] Obtaining the maximum business bandwidth and the maximum IOPS that can be carried by the HDD device at the same time and recording them;

[0009] Before each business thread and / or flushing thread writes data to the HDD device, calculating the bandwidth resource amount and the IOPS resource amount required by the current write operation;

[0010] Judging whether the required bandwidth resource amount exceeds the recorded maximum business bandwidth and whether the required IOPS resource amount exceeds the recorded maximum IOPS;

[0011] If yes, performing the write operation and releasing the occupied bandwidth resource and IOPS resource after the write operation is performed; if no, waiting for the release of the bandwidth resource and IOPS resource.

[0012] Preferably, in the above cache data flushing flow control method provided by the application, the recording comprises:

[0013] Building a throttle valve unit; the throttle valve unit comprises a first counter for recording the maximum business bandwidth and the occupied bandwidth resource, and a second counter for recording the maximum IOPS and the occupied IOPS resource;

[0014] Initializing the maximum business bandwidth recorded by the first counter and the maximum IOPS recorded by the second counter.

[0015] Preferably, in the above cache data flushing flow control method provided by the application, the judging whether the required bandwidth resource amount exceeds the recorded maximum business bandwidth and whether the required IOPS resource amount exceeds the recorded maximum IOPS comprises:

[0016] According to the required bandwidth resource amount and the required IOPS resource amount, applying for bandwidth resource and IOPS resource to the throttle valve unit;

[0017] The first counter of the throttle valve unit is used to determine whether the bandwidth required to be occupied exceeds the recorded maximum service bandwidth, and the second counter of the throttle valve unit is used to determine whether the IOPS resource required to be occupied exceeds the recorded maximum IOPS.

[0018] Preferably, in the cache data flushing traffic control method provided by the embodiment of the application, the throttle valve unit further comprises a FIFO queue for waking up the waiting thread.

[0019] The release of the bandwidth resource and the IOPS resource includes:

[0020] A condition variable is added to the FIFO queue, and the thread is put to sleep, and after the thread is woken up by the condition variable, the bandwidth or the IOPS resource is re-applied to the throttle valve unit.

[0021] Preferably, in the cache data flushing traffic control method provided by the embodiment of the application, before the write operation is performed, the method further comprises:

[0022] The throttle valve unit is called to mark the bandwidth resource and the IOPS resource required to be occupied.

[0023] Preferably, in the cache data flushing traffic control method provided by the embodiment of the application, the release of the occupied bandwidth resource and the IOPS resource includes:

[0024] The throttle valve unit is called to mark and release the occupied bandwidth resource and the IOPS resource.

[0025] Preferably, in the cache data flushing traffic control method provided by the embodiment of the application, after the release of the occupied bandwidth resource and the IOPS resource, the method further comprises:

[0026] If the FIFO queue is not empty, the condition variable that is woken up by the earliest enqueued condition variable in the FIFO queue is dequeued, and the bandwidth or the IOPS resource is re-applied to the throttle valve unit.

[0027] The embodiment of the application further provides a cache data flushing traffic control device applied to an ICFS distributed layered storage system, and the device comprises:

[0028] A traffic index acquisition module is configured to acquire the maximum service bandwidth and the maximum IOPS that can be borne by the HDD device at the same time.

[0029] A resource amount calculation module is configured to calculate the bandwidth resource amount and the IOPS resource amount required to be occupied by the write operation before each service thread and / or flushing thread writes data to the HDD device.

[0030] The available resource judging module is used for judging whether the bandwidth resource amount to be occupied does not exceed the maximum service bandwidth and whether the IOPS resource amount to be occupied does not exceed the maximum IOPS; if yes, the write operation is executed, and the occupied bandwidth resource and IOPS resource are released after the write operation is executed; if no, the release of the bandwidth resource and the IOPS resource is waited.

[0031] The embodiment of the application further provides a cache data flushing flow control device, comprising a processor and a memory, wherein the processor implements the cache data flushing flow control method provided by the embodiment of the application when executing the computer program stored in the memory.

[0032] The embodiment of the application further provides a computer readable storage medium for storing a computer program, wherein the computer program is executed by the processor to implement the cache data flushing flow control method provided by the embodiment of the application.

[0033] It can be seen from the above technical solution that the cache data flushing flow control method provided by the application is applied to the ICFS distributed hierarchical storage system, and comprises the following steps: obtaining and recording the maximum service bandwidth and the maximum IOPS that can be carried by the HDD device at the same time; calculating the bandwidth resource amount and the IOPS resource amount to be occupied by the current write operation before each service thread and / or flushing thread writes data to the HDD device; judging whether the bandwidth resource amount to be occupied does not exceed the recorded maximum service bandwidth and whether the IOPS resource amount to be occupied does not exceed the recorded maximum IOPS; if yes, executing the write operation and releasing the occupied bandwidth resource and IOPS resource after the write operation is executed; if no, waiting for the release of the bandwidth resource and the IOPS resource.

[0034] The cache data flushing flow control method provided by the application takes into account the IOPS and bandwidth flow control, can realize the flushing flow control of the ICFS distributed hierarchical storage system, can simply and effectively control the HDD disk service bandwidth and IOPS, can fully utilize the HDD disk performance, can effectively improve the flushing performance of the cache data of the ICFS distributed hierarchical storage system, and can reduce the service performance fluctuation caused by flushing. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0036] Figure 1 The architecture diagram for writing existing write service data into a cache device or a main storage device;

[0037] Figure 2 The flowchart of the cache data flushing traffic control method provided by the embodiment of the present application;

[0038] Figure 3 The specific flowchart of the cache data flushing traffic control method provided by the embodiment of the present application;

[0039] Figure 4 The structure schematic diagram of the cache data flushing traffic control device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0040] In the ICFS distributed hierarchical storage system, due to the limited storage space of the SSD cache device, when the front-end service is written, the dirty data on the cache device needs to be constantly flushed to the HDD device. The dirty data flushing brings additional write pressure to the HDD device, which will compete with the business IO for disk resources. Since the HDD device can carry limited amount of business within a period of time, when the write pressure is too large, it will cause a series of problems such as business jamming and performance fluctuation.

[0041] In order to solve the above technical problems, the present application provides a cache data flushing traffic control method. The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0042] The cache data flushing traffic control method provided by the present application is applied to an ICFS distributed hierarchical storage system, as shown in the figure, and the method comprises the following steps: Figure 2

[0043] S201, acquiring the maximum service bandwidth and the maximum IOPS that the HDD device can carry at the same time, and recording them;

[0044] ​It should be noted that IOPS (Input / Output Operations Per Second) refers to the number of IOs here. The present application takes into account both IOPS and bandwidth, two forms of traffic control indicators. The maximum service bandwidth and the maximum IOPS that can be carried by the HDD device at the same time can be obtained in an estimated manner or in other manners, which is not limited here.

[0045] S202, before each service thread and / or flushing thread writes data to the HDD device, the bandwidth resource amount and the IOPS resource amount required by the current write operation are calculated;

[0046] In actual applications, for example, random write and small block write service data are written to the cache device, while sequential write and large block write service data are directly written to the HDD device. The data on the cache device is flushed to the HDD device later. In the present application, the bandwidth resource amount and the IOPS resource amount required by the current write operation are calculated before the data is directly written to the HDD device or the cache data is flushed to the HDD device. Figure 1

[0047] S203, it is judged whether the bandwidth resource amount required exceeds the maximum service bandwidth and whether the IOPS resource amount required exceeds the maximum IOPS;

[0048] It should be noted that the present application abstracts the write performance of the disk as a kind of available resource. When the judgment operation in step S203 is performed, it is judged whether the bandwidth resource amount required exceeds the recorded maximum service bandwidth and whether the IOPS resource amount required exceeds the recorded maximum IOPS, which can be understood as whether the available resource is sufficient.

[0049] If yes, step S104 is performed; if no, step S105 is performed;

[0050] S204, the write operation is performed, and the occupied bandwidth resource and IOPS resource are released after the write operation is performed; that is, when the bandwidth resource amount required does not exceed the recorded maximum service bandwidth and the IOPS resource amount required does not exceed the recorded maximum IOPS, the write operation can be directly performed, and the occupied bandwidth resource and IOPS resource are released after the write operation is performed.

[0051] S205, the release of the bandwidth resource and the IOPS resource is waited for; that is, when the bandwidth resource amount required exceeds the recorded maximum service bandwidth and / or the IOPS resource amount required exceeds the recorded maximum IOPS, the release of the bandwidth resource and the IOPS resource is waited for.

[0052] ​In the cache data flushing flow control method provided by the embodiment of the present application, the IOPS and bandwidth are considered, the flushing flow control of the ICFS distributed hierarchical storage system can be realized, the HDD service bandwidth and IOPS can be simply and effectively controlled, the HDD performance can be fully utilized, the flushing performance of the cache data of the ICFS distributed hierarchical storage system is effectively improved, and the service performance fluctuation caused by the flushing is reduced.

[0053] Further, in the specific implementation, in the cache data flushing flow control method provided by the embodiment of the present application, the step S201 records, as shown in the following table, specifically can include the following steps. Figure 3

[0054] It should be noted that, in the present application, when the throttle valve unit is constructed, the write performance of the disk is abstracted as a throttle valve unit realized by a resource, the maximum IOPS and bandwidth of the write request handled by the HDD disk in a uniform time can be uniformly controlled through the throttle valve unit. The data organization form of the throttle valve unit includes the first counter used to record the occupied bandwidth resource and the maximum service bandwidth, and the second counter used to record the occupied IOPS resource and the maximum IOPS. The occupied bandwidth resource can be recorded as bandwidth_count, the maximum service bandwidth can be recorded as bandwidth_max, the occupied IOPS resource can be recorded as iops_count, and the maximum IOPS can be recorded as iops_max. After the maximum service bandwidth and the maximum IOPS of the HDD device in the same time are obtained, the bandwidth_max and iops_max parameters of the throttle valve unit can be initialized.

[0055] In the specific implementation, in the cache data flushing flow control method provided by the embodiment of the present application, the step S203 judges whether the required occupied bandwidth resource amount exceeds the recorded maximum service bandwidth, and whether the required occupied IOPS resource amount exceeds the recorded maximum IOPS, specifically can include the following steps. First, according to the required occupied bandwidth resource amount and the required occupied IOPS resource amount, the bandwidth resource and the IOPS resource are applied to the throttle valve unit. Then, the first counter of the throttle valve unit is used to judge whether the required occupied bandwidth exceeds the recorded maximum service bandwidth, and the second counter of the throttle valve unit is used to judge whether the required occupied IOPS resource amount exceeds the recorded maximum IOPS.

[0056] ​Specifically, all write operations to the HDD device can first try to apply bandwidth resources and IOPS resources to the throttle valve unit, and the throttle valve unit determines whether the available resources are sufficient, that is, whether bandwidth_count<=bandwidth_max and iops_count<=iops_max; when the available resources are insufficient, the resources must be waited for release, and when the available resources are sufficient, the write operation is directly executed without waiting, and the occupied resources are released after the execution is completed.

[0057] Further, in the above cache data flushing traffic control method provided by the embodiment of the application, the throttle valve unit can further include a FIFO queue (the member is a condition variable) for waking up the waiting thread, which can be denoted as waiting_conds.

[0058] It should be understood that FIFO (First Input First Output) simply refers to a first-in first-out queue in a data structure, which is a sequential execution method. The first instruction is completed and retired first, and the second instruction is executed following the first instruction. The instruction is a program code of a computer in response to a user operation, which is transparent to the user. Compared with a common data buffer, the FIFO does not have the function of external read and write address, and thus is simpler to use.

[0059] Therefore, step S205 waits for the release of the bandwidth resources and the IOPS resources, and specifically can include: adding a condition variable to the FIFO queue waiting_conds and suspending the thread, and after the thread is woken up by the condition variable, the bandwidth or IOPS resources are applied to the throttle valve unit again.

[0060] In the above cache data flushing traffic control method provided by the embodiment of the application, before the write operation in step S204 is executed, as shown in Figure 3 , the method can further include: calling the throttle valve unit to mark the bandwidth resources and the IOPS resources that need to be occupied. Specifically, if the available resources are sufficient, the resources do not need to be waited for release, and the bandwidth_count and the iops_count can be updated.

[0061] In the above cache data flushing traffic control method provided by the embodiment of the application, step S204 releases the occupied bandwidth resources and IOPS resources, and as shown in Figure 3 , the method can include: calling the throttle valve unit to mark and release the occupied bandwidth resources and IOPS resources.

[0062] In actual application, the throttle unit needs to be called to release the previously acquired resources regardless of the success or failure of the business. When the throttle resources are released, the bandwidth_count and iops_count can be updated.

[0063] In the specific implementation, after the occupied bandwidth resources and IOPS resources are released in step S204, as shown in the following formula, the FIFO queue can be used to control the release of the resources. Figure 3 If the FIFO queue is not empty, the earliest-in-queue conditional variable corresponding to the waiting thread is dequeued, and then the bandwidth or IOPS resources are re-applied to the throttle unit. In this way, the FIFO conditional variable queue is used to control the awakening of the sleeping waiting thread, and the thread corresponding to the conditional variable in the waiting queue is immediately notified after the available resources are released, so that there is no redundant waiting, and the disk performance and utilization can be effectively improved.

[0064] The above-mentioned cache data flushing traffic control method provided by the embodiments of the present application is described below with a specific example. The specific steps are as follows:

[0065] Step 1: Estimate the maximum business bandwidth and IOPS that can be carried by the HDD device at the same time, and initialize the bandwidth_max and iops_max parameters of the throttle unit.

[0066] Step 2: Before issuing the write business / flush business to the HDD device each time, calculate the bandwidth and IOPS resource amount required by the write operation this time and record it.

[0067] Step 3: Before the business / flush thread executes the write business to the HDD device, the bandwidth resources or iops resources are first applied to the throttle unit, and the throttle unit judges whether the available resources are sufficient (whether bandwidth_count<=bandwidth_max and iops_count<=iops_max). If the available resources are insufficient, a conditional variable is added to the waiting_conds, and the thread is put to sleep, and after the variable is awakened, the application is re-attempted. If the available resources are sufficient, the bandwidth_count and iops_count are updated without waiting.

[0068] Step 4: Execute the write business. Regardless of the success or failure of the business, the previously acquired throttle resources need to be released.

[0069] Step 5: After the business process is completed, call the throttle valve to release the previously acquired resources. When releasing the used throttle valve resources, first update bandwidth_count and iops_count. If the waiting_conds queue is not empty, dequeue the earliest enqueued condition variable from the queue and wake up its corresponding waiting thread.

[0070] This invention abstracts disk write performance as an available resource. By executing steps one to five above, a throttling valve unit is used to uniformly manage the occupation and release of disk write performance resources, thus simply and effectively achieving flow control for cached data flushing.

[0071] Based on the same inventive concept, this invention also provides a cached data flushing traffic control device. Since the principle of this device in solving the problem is similar to that of the aforementioned cached data flushing traffic control method, the implementation of this device can refer to the implementation of the cached data flushing traffic control method, and the repeated parts will not be described again.

[0072] In practical implementation, the cache data flushing traffic control device provided in this embodiment of the invention is applied to the ICFS distributed tiered storage system, such as... Figure 4 As shown, it specifically includes:

[0073] Traffic metrics acquisition module 11 is used to acquire the maximum service bandwidth and maximum IOPS that the HDD device can carry at the same time.

[0074] The resource quantity calculation module 12 is used to calculate the amount of bandwidth resources and IOPS resources required for each write operation before each business thread and / or brush thread writes data to the HDD device.

[0075] The available resource judgment module 13 is used to determine whether the amount of bandwidth resources to be occupied does not exceed the maximum service bandwidth and whether the amount of IOPS resources to be occupied does not exceed the maximum IOPS; if so, a write operation is performed and the occupied bandwidth resources and IOPS resources are released after the write operation is performed; if not, the module waits for the bandwidth resources and IOPS resources to be released.

[0076] In the cache data flushing traffic control device provided in the embodiments of the present invention, the flushing traffic control of the ICFS distributed tiered storage system can be realized through the interaction of the above three modules. It can simply and effectively control the HDD disk service bandwidth and IOPS, make full use of HDD disk performance, effectively improve the flushing performance of cache data in the ICFS distributed tiered storage system, and reduce the service performance fluctuations caused by flushing.

[0077] Furthermore, in a specific implementation, the cached data flushing flow control device provided in the embodiments of the present invention further includes:

[0078] a throttle valve module, configured to build a throttle valve unit; the throttle valve unit can include a first counter configured to record a maximum service bandwidth and occupied bandwidth resources, and a second counter configured to record a maximum IOPS and occupied IOPS resources;

[0079] an initialization module, configured to initialize the maximum service bandwidth recorded by the first counter, and the maximum IOPS recorded by the second counter.

[0080] In a specific implementation, in the cache data underbrush traffic control device provided by the embodiment of the application, the available resource judgment module can specifically include:

[0081] a resource application sub-module, configured to apply for bandwidth resources and IOPS resources from the throttle valve unit according to the amount of occupied bandwidth resources and the amount of occupied IOPS resources;

[0082] a resource judgment sub-module, configured to judge whether the occupied bandwidth resources exceed the recorded maximum service bandwidth by using the first counter of the throttle valve unit, and whether the amount of occupied IOPS resources exceeds the recorded maximum IOPS by using the second counter of the throttle valve unit.

[0083] Further, in a specific implementation, in the cache data underbrush traffic control device provided by the embodiment of the application, the throttle valve unit can further include a FIFO queue configured to wake up a waiting thread; based on this, the available resource judgment module can specifically further include:

[0084] a resource waiting sub-module, configured to add a condition variable to the FIFO queue and put the thread to sleep, and wait for the condition variable to wake up the thread.

[0085] In a specific implementation, in the cache data underbrush traffic control device provided by the embodiment of the application, the available resource judgment module can specifically further include:

[0086] a resource release sub-module, configured to call the throttle valve unit to mark and release the occupied bandwidth resources and IOPS resources.

[0087] In a specific implementation, in the cache data underbrush traffic control device provided by the embodiment of the application, the resource waiting sub-module is further configured to, after the resource release sub-module releases the occupied bandwidth resources and IOPS resources, if the FIFO queue is not empty, wake up the corresponding waiting thread from the FIFO queue by dequeuing the earliest-in-queue condition variable.

[0088] The more specific working processes of the above-mentioned various modules can refer to the corresponding contents disclosed in the foregoing embodiments, and will not be described here in detail.

[0089] Correspondingly, the application also discloses a cache data underbrush traffic control device.

[0090] More specific processes of the above method can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0091] Further, the application also discloses a computer readable storage medium for storing a computer program; the computer program is executed by a processor to implement the foregoing disclosed cache data underbrush traffic control method.

[0092] More specific processes of the above method can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0093] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device, equipment and storage medium disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0094] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0095] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0096] To sum up, the cache data flushing flow control method provided by the embodiment of the application is applied to the ICFS distributed hierarchical storage system, and comprises the following steps: obtaining the maximum service bandwidth and the maximum IOPS that can be carried by the HDD device at the same time, and recording the same; before each service thread and / or flushing thread writes data to the HDD device, calculating the bandwidth resource amount and the IOPS resource amount required by the current write operation; determining whether the bandwidth resource amount required is not more than the recorded maximum service bandwidth, and whether the IOPS resource amount required is not more than the recorded maximum IOPS; if yes, performing the write operation, and releasing the occupied bandwidth resource and IOPS resource after the write operation is performed; if no, waiting for the release of the bandwidth resource and the IOPS resource. The above method takes into account the flow control of the IOPS and the bandwidth in two forms, can realize the flushing flow control of the ICFS distributed hierarchical storage system, can simply and effectively control the HDD disk service bandwidth and the IOPS, and can fully utilize the HDD disk performance, effectively improve the flushing performance of the cache data of the ICFS distributed hierarchical storage system, and reduce the service performance fluctuation caused by flushing. In addition, the application also provides a corresponding device, equipment and computer readable storage medium for the cache data flushing flow control method, and further makes the above method more practical, and the device, the equipment and the computer readable storage medium have corresponding advantages.

[0097] Finally, it should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or equipment including the element.

[0098] The cache data flushing flow control method, device, equipment and storage medium provided by the application are described in detail above, and the principle and implementation mode of the application are described by applying specific examples in this document; the above embodiment description is only used to help understand the method and core idea of the application; meanwhile, for those skilled in the art, according to the idea of the application, the specific implementation mode and application range will be changed; in conclusion, the content of the specification should not be understood as a limitation of the application.

Claims

1. A method for controlling data cache underflush traffic, the method comprising: The application is applied to an ICFS distributed hierarchical storage system, comprising: Obtaining the maximum service bandwidth and the maximum IOPS that can be carried by the HDD device at the same time, and constructing a throttle valve unit; the throttle valve unit comprises a first counter for recording the maximum service bandwidth and occupied bandwidth resources, a second counter for recording the maximum IOPS and occupied IOPS resources, and a FIFO queue for waking up a waiting thread; Initializing the maximum service bandwidth recorded by the first counter and the maximum IOPS recorded by the second counter; Before each service thread and / or flushing thread writes data to the HDD device, calculating the amount of bandwidth resources and IOPS resources required to be occupied by the current write operation; Judging whether the amount of bandwidth resources required to be occupied exceeds the recorded maximum service bandwidth, and whether the amount of IOPS resources required to be occupied exceeds the recorded maximum IOPS; If the amount of bandwidth resources required to be occupied does not exceed the recorded maximum service bandwidth, and the amount of IOPS resources required to be occupied does not exceed the recorded maximum IOPS, the write operation is executed, and the occupied bandwidth resources and IOPS resources are released after the write operation is executed, if the FIFO queue is not empty, the earliest enqueued condition variable is dequeued from the FIFO queue to wake up the corresponding waiting thread, and then bandwidth or IOPS resources are re-applied to the throttle valve unit; If the amount of bandwidth resources required to be occupied exceeds the recorded maximum service bandwidth, and / or the amount of IOPS resources required to be occupied exceeds the recorded maximum IOPS, a condition variable is added to the FIFO queue, and the current thread is put to sleep, waiting for the condition variable to wake up the current thread, and then bandwidth or IOPS resources are re-applied to the throttle valve unit.

2. The method of claim 1, wherein, The judgment of whether the amount of bandwidth resources required to be occupied exceeds the recorded maximum service bandwidth, and whether the amount of IOPS resources required to be occupied exceeds the recorded maximum IOPS, comprises: According to the amount of bandwidth resources required to be occupied and the amount of IOPS resources required to be occupied, bandwidth resources and IOPS resources are applied to the throttle valve unit; Using the first counter of the throttle valve unit to judge whether the amount of bandwidth required to be occupied exceeds the recorded maximum service bandwidth, and using the second counter of the throttle valve unit to judge whether the amount of IOPS resources required to be occupied exceeds the recorded maximum IOPS.

3. The method of claim 2, wherein, Before the write operation is executed, it further comprises: Calling the throttle valve unit to mark the bandwidth resources and IOPS resources required to be occupied.

4. The method of claim 3, wherein, The release of the occupied bandwidth resources and IOPS resources comprises: Calling the throttle valve unit to mark and release the occupied bandwidth resources and IOPS resources.

5. A cache data underflush traffic control apparatus, characterized by, The application is applied to an ICFS distributed hierarchical storage system, comprising: A traffic index acquisition module for obtaining the maximum service bandwidth and the maximum IOPS that can be carried by the HDD device at the same time; a throttle valve construction module, configured to construct a throttle valve unit; the throttle valve unit comprises a first counter configured to record the maximum traffic bandwidth and occupied bandwidth resource, a second counter configured to record the maximum IOPS and occupied IOPS resource, and a FIFO queue configured to wake up a waiting thread; an initialization module, configured to initialize the maximum traffic bandwidth recorded by the first counter, and the maximum IOPS recorded by the second counter; a resource amount calculation module, configured to calculate the bandwidth resource amount and IOPS resource amount required to be occupied by a current write operation before each traffic thread and / or flush thread writes data to the HDD device; an available resource judgment module, configured to judge whether the bandwidth resource amount required to be occupied does not exceed the maximum traffic bandwidth, and whether the IOPS resource amount required to be occupied does not exceed the maximum IOPS; if the bandwidth resource amount required to be occupied does not exceed the recorded maximum traffic bandwidth, and the IOPS resource amount required to be occupied does not exceed the recorded maximum IOPS, the write operation is executed, and the occupied bandwidth resource and IOPS resource are released after the write operation is executed; if the FIFO queue is not empty, a condition variable with the earliest queue time is dequeued from the FIFO queue to wake up the corresponding waiting thread, and then bandwidth or IOPS resource is re-applied to the throttle valve unit; if the bandwidth resource amount required to be occupied exceeds the recorded maximum traffic bandwidth, and / or the IOPS resource amount required to be occupied exceeds the recorded maximum IOPS, a condition variable is added to the FIFO queue, and the current thread is put to sleep, waiting for the condition variable to wake up the current thread, and then bandwidth or IOPS resource is re-applied to the throttle valve unit.

6. A cache data underflush traffic control device, characterized by, A computer program product comprising a processor and a memory, wherein the processor implements the cache data flush traffic control method according to any one of claims 1 to 4 when executing the computer program stored in the memory.

7. A computer readable storage medium characterized in that, A computer program product for storing a computer program, wherein the computer program is executed by a processor to implement the cache data flush traffic control method according to any one of claims 1 to 4.

Images