Cache data flush adjustment method, device, electronic device and storage medium
By monitoring and adjusting the read and write IO response time of the storage server, combining the disk redundant array RAID type and software stack running time, dynamically adjusting the cache brushing speed, solving the problem of insufficient storage performance, realizing the improvement of storage performance and system Automatic optimization.
Patent Information
- Application Number
- CN202411207259.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-30
AI Technical Summary
In the field of storage, storage controllers need to speed up the speed of each processing stage and improve storage performance on the basis of ensuring data security and system stability. However, the existing technology is difficult to effectively solve the problem of flash speed control under cache, resulting in insufficient storage performance .
By monitoring the read and write IO response time of the target storage server, combining the disk redundant array RAID type and software stack running time, adjusting the cache brushing speed to optimize the read and write IO response time, dynamically adjusting the brushing speed to ensure that the system is in abnormal situations. Automatically optimize performance.
It realizes the improvement of storage performance on the basis of ensuring data security and system stability, optimizes the overall processing efficiency of IO services, and improves the read and write performance of the storage system.
Smart Images

Figure CN118708134B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a cache data flush adjustment method, device, electronic device and storage medium. Background Art
[0002] The storage controller is mainly used for managing the hard disks and the data stored on them, and provides data access services to the server. In the process of providing data access to the server, due to the huge number of IOs processed in a short period of time, the CPU in the storage controller undertakes a large number of unpacking, disk reading, and disk writing operations. In the storage field, a data I / O will go through multiple operation stages such as storage volume, storage pool, RAID, storage disk, etc. in the back-end processing.
[0003] How to speed up the processing speed at each stage and improve storage performance while ensuring data security and system stability is an important issue that needs to be urgently addressed in the industry. Summary of the invention
[0004] The present invention provides a cache data flush adjustment method, device, electronic equipment and storage medium, which are used to improve storage performance.
[0005] The present invention provides a cache data flush adjustment method, comprising the following steps:
[0006] Determine the expected response time of the cache flush of the target storage server based on the RAID type of the disk in the target storage server and the software stack running time during the cache data flush process;
[0007] Determine the expected read response time of the software stack based on the read IO standard response time of the disk and the running time of the software stack;
[0008] Based on monitoring the read and write of the cache of the target storage server in the current observation period, determining the actual read IO response time of the software stack and the actual write IO response time of the software stack;
[0009] Based on the cache flush speed of the current observation period, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time, the cache flush speed of the next observation period is adjusted.
[0010] According to a cache data flush adjustment method provided by the present invention, the actual read IO response time of the software stack and the actual write IO response time of the software stack are determined based on monitoring the read and write of the cache of the target storage server in the current observation period, including:
[0011] Monitor the read and write of the cache of the target storage server in the current observation period, and obtain the average response time of the read IO in the current observation period and the average response time of the write IO in the current observation period;
[0012] The average response time of the read IO is used as the actual read IO response time, and the average response time of the write IO is used as the actual write IO response time.
[0013] According to a cache data flush adjustment method provided by the present invention, the obtaining of the average response time of the read IO in the current observation period includes:
[0014] Obtain the number of all read IOs in the current observation period and the cumulative value of all read IO response times in the current observation period;
[0015] Based on the number of all read IOs and the accumulated value, an average response time of the read IO in the current observation period is determined.
[0016] According to a cache data flush adjustment method provided by the present invention, obtaining an average response time of write IO in the current observation period includes:
[0017] Obtain the number of all write IOs in the current observation period and the cumulative value of all write IO response times in the current observation period;
[0018] Based on the number of all write IOs and the accumulated value, an average response time of the write IO in the current observation period is determined.
[0019] According to a cache data flush adjustment method provided by the present invention, the method determines the cache flush expected response time of the target storage server based on the independent disk redundant array RAID type of the disk in the target storage server and the software stack running time during the cache data flush process, including:
[0020] When it is determined that the RAID type of the disk is RAID0 or RAID1, the expected response time of the cache flush is:
[0021] T_w_flush=T_w_disk+T_soft;
[0022] When it is determined that the RAID type of the disk is RAID5, the expected response time of the cache flush is:
[0023] T_w_flush=2×T_w_disk+2×T_r_disk+T_soft;
[0024] When it is determined that the RAID type of the disk is RAID6, the expected response time of the cache flush is:
[0025] T_w_flush=3×T_w_disk+3×T_r_disk+T_soft;
[0026] Among them, T_w_flush is the expected response time for cache flushing, T_soft is the software stack running time, T_r_disk is the standard response time for read IO, and T_w_disk is the standard response time for write IO.
[0027] According to a cache data flush adjustment method provided by the present invention, the expected read response time of the software stack is:
[0028] T_r_flush=T_r_disk+T_soft;
[0029] Among them, T_r_flush is the expected read response time of the software stack, T_soft is the software stack running time, and T_r_disk is the read IO standard response time.
[0030] According to a cache data flush adjustment method provided by the present invention, the cache flush speed of the next observation cycle is adjusted based on the cache flush speed of the current observation cycle, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time, including:
[0031] Determine a flush adjustment speed based on the cache flush speed of the current observation period, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time;
[0032] Based on the refresh adjustment speed, the cache refresh speed of the next observation period is adjusted.
[0033] According to a cache data flush adjustment method provided by the present invention, the flush adjustment speed is:
[0034] C_next=C_cur×{(T_w_flush-T_w_resp)×D 1 +(T_r_flush-T_r_resp)×D 2};
[0035] Among them, C_next is the flush adjustment speed, C_cur is the cache flush speed of the current observation period, T_w_flush is the cache flush expected response time, T_r_flush is the read expected response time, T_r_resp is the actual read IO response time, T_w_resp is the actual write IO response time, D 1 and D 2 is the weight coefficient, D 1 +D 2 =1.
[0036] According to a cache data flushing adjustment method provided by the present invention, after adjusting the cache flushing speed of the next observation period, the method further includes:
[0037] Based on the adjustment weight coefficient D 1 And the weight coefficient D 2 , further adjust the cache refresh speed.
[0038] According to a cache data flush adjustment method provided by the present invention, the current observation period is determined based on an actual response time of the cache flush and an expected response time of the cache flush.
[0039] According to a cache data flush adjustment method provided by the present invention, the current observation period is:
[0040] T_observation=(T_w_flush / T_resp)×100;
[0041] Among them, T_observation is the current observation period, T_w_flush is the expected response time of cache flush, and T_resp is the actual response time of cache flush.
[0042] The present invention also provides a cache data flush adjustment device, comprising the following modules:
[0043] A first expected time determination module, configured to determine the expected response time of a cache flush of the target storage server based on the RAID type of the disk in the target storage server and the software stack running time during the cache data flush process;
[0044] A second expected time determination module, configured to determine an expected read response time of a software stack based on a read IO standard response time of the disk and a running time of the software stack;
[0045] An actual time determination module, configured to determine an actual read IO response time of the software stack and an actual write IO response time of the software stack based on monitoring the read and write of the cache of the target storage server in a current observation period;
[0046] The adjustment module is used to adjust the cache flush speed of the next observation cycle based on the cache flush speed of the current observation cycle, the cache flush expected response time, the read expected response time, the actual read IO response time and the actual write IO response time.
[0047] The present invention also provides a storage server, comprising the cache data flushing adjustment device as described above.
[0048] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the cache data flushing adjustment methods described above is implemented.
[0049] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the cache data flushing adjustment method described in any one of the above methods is implemented.
[0050] The cache data flushing adjustment method, device, electronic device and storage medium provided by the present invention comprehensively consider the read and write IO response times. When an abnormal situation occurs in the back-end, there is no need to build a special abnormal scenario cache processing solution. The flushing speed can be adjusted automatically only by the response time. At the same time, the software stack below the cache automatically adjusts its state to enter the optimal state of the system, thereby optimizing the system read and write performance, improving the overall processing efficiency of the IO business, and thus improving the storage performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0052] Figure 1 It is a schematic diagram of the storage controller structure for introducing cache provided in the related method.
[0053] Figure 2 It is a schematic diagram of flushing cache data provided in the relevant method.
[0054] Figure 3 It is a flow chart of the cache data refresh adjustment method provided by the present invention.
[0055] Figure 4 It is a schematic diagram of the structure of the storage pool provided by the present invention.
[0056] Figure 5It is a structural schematic diagram of the cache data flush adjustment device provided by the present invention.
[0057] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] The storage controller is mainly used for managing the hard disks and the data stored on them at the storage end, and provides data access services to the server end. In the process of providing data access to the server end, due to the huge number of input and output (IO) processed in a short period of time, the central processing unit (CPU) in the storage controller undertakes a large number of unpacking, disk reading, and disk writing operations. In the storage field, a data I / O will go through multiple operation stages such as storage volume, storage pool, independent redundant disk array (RAID), storage disk, etc. in the back-end processing. How to speed up the processing speed of each stage and how to improve performance (response time, bandwidth, IOPS) on the basis of ensuring data security and system stability are urgent problems to be solved in the storage field.
[0060] In related methods, the CPU solves the speed difference between the processor core and the memory and introduces the principle of high-speed cache. The storage controller introduces a large amount of memory as a cache between the front-end data and the physical disk, thereby improving the performance of the storage system.
[0061] like Figure 1 As shown in the schematic diagram of the storage controller structure with introduced cache provided in the related method, the storage structure with introduced cache is completed from the IO request sent from the server to the cache level. Not only does it shorten the IO path, but the cache (the actual medium is memory) performance is improved by several orders of magnitude compared to the disk, thereby greatly improving the storage performance.
[0062] However, the cache capacity is much smaller than the capacity provided by the backend disk, and it will eventually be exhausted. Data needs to be flushed from the cache to the backend disk. Figure 2 As shown in the cache data flushing diagram provided in the related method, the flushing IO data in the cache is flushed from the cache to the storage disk to implement the data storage process.
[0063] How to control the cache flush speed to provide better storage performance is an important issue that needs to be solved in related methods.
[0064] In view of the defects of the related methods, the present invention provides a cache data flush adjustment method. Figure 3 is a flow chart of the cache data flush adjustment method provided by the present invention, such as Figure 3 As shown, the method includes the following:
[0065] Step 310, determining the expected response time of the cache flush of the target storage server based on the RAID type of the disk in the target storage server and the software stack running time during the cache data flush process;
[0066] Step 320, determining the expected read response time of the software stack based on the read IO standard response time of the disk and the software stack running time;
[0067] Step 330, based on monitoring the read and write of the cache of the target storage server in the current observation period, determine the actual read IO response time of the software stack and the actual write IO response time of the software stack;
[0068] Step 340, based on the cache flush speed of the current observation cycle, the cache flush expected response time, the read expected response time, the actual read IO response time and the actual write IO response time, adjust the cache flush speed of the next observation cycle.
[0069] The execution subject of the cache data flush adjustment method provided by the present invention may be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a network attached storage (NAS), or a personal computer (PC), etc., which is not specifically limited by the present invention.
[0070] The following takes a computer executing the cache data flushing adjustment method provided by the present invention as an example to describe the technical solution of the present invention in detail.
[0071] In step 310, based on the RAID type of the disks in the target storage server and the software stack running time during the cache data flushing process, the expected cache flushing response time of the target storage server is determined.
[0072] Cache data flushing refers to the process of writing data stored in the cache (usually those that have been recently accessed or modified) to a slower but larger storage device (such as a hard disk drive HDD or solid-state drive SSD). This process is to ensure data persistence and consistency, because cache devices are volatile, that is, when the power is turned off, the data stored in them will be lost.
[0073] The expected response time for cache flushing is a theoretical time, which is the theoretical time for cache data to be flushed to the disk.
[0074] It is understandable that since the data flushing is completed only after it is written to the disk, the cache flushing performance is mainly affected by two factors: one is the performance of the backend disk that provides capacity, and the other is the redundancy strategy used by the storage pool based on the disk, that is, the RAID type.
[0075] like Figure 4 As shown in the structural diagram of the storage pool provided by the present invention, the disks first form the required RAID according to the data security policy, and then the storage space composed of multiple disks is handed over to the storage pool for management, and finally provided to the customer. Considering the data serialization problem, the commonly used RAID types will have the following write penalties:
[0076] Write penalty means that one write IO needs to be completed by reading and writing the disk multiple times, and the reading and writing need to be serial; no write penalty means that one write IO only needs to be completed by writing the disk once, or multiple parallel writes can be completed.
[0077] RAID0, RAID1: no write penalty;
[0078] RAID5: There is a write penalty. One write IO requires two disk reads and two disk writes to complete.
[0079] RAID6: There is a write penalty. One write IO requires three disk reads and three disk writes to complete.
[0080] During the process of flushing cache data, some storage-related operations are implemented based on the software stack, such as primary and secondary backup storage of cached data, verification of cached data, etc. The software stack operation consumes time, thus generating the software stack operation time. The software stack operation time can be obtained by intercepting the operation timestamp.
[0081] Therefore, it can be understood that the expected response time of the cache flush of the target storage server can be determined based on the RAID type of the disk and the software stack running time during the cache data flush process.
[0082] It is understandable that the cache refresh expected response time formulated in this way is the optimal response time below the storage system cache. If it is lower than this value, the storage system performance cannot be brought into play. If it is higher than this value, the system will be overloaded, which will only prolong the system response time and cannot increase the system IO throughput.
[0083] In step 320, based on the read IO standard response time of the disk and the software stack running time, the expected read response time of the software stack is determined.
[0084] The expected read time of the software stack is a time determined by considering the response speed of the following read IO in the cache. Optionally, the determined expected read response time of the software stack can be:
[0085] T_r_flush=T_r_disk+T_soft;
[0086] Among them, T_r_flush is the expected read response time of the software stack, T_soft is the software stack running time, and T_r_disk is the read IO standard response time.
[0087] The read IO standard response time T_r_disk may be an optimal read IO response time of the disk, and may be determined based on optimal values recommended in various scenarios in the manual of the disk hardware manufacturer.
[0088] In step 330, based on monitoring the read and write of the cache of the target storage server in the current observation period, the actual read IO response time of the software stack and the actual write IO response time of the software stack are determined.
[0089] The actual response time is obtained by counting the average response time of all read and write IOs from the cache to the downstream software stack and the downstream software stack to the cache in the current observation period. That is, the number and response time of read and write IOs are accumulated in the observation period, and then:
[0090] T_average_r_respose=T_sum_r_respose / Count_r;
[0091] T_average_w_respose=T_sum_w_respose / Count_w;
[0092] Among them, T_average_r_respose is the average response time of read IO during the observation period, which can be used as the actual read IO response time of the software stack; T_sum_r_respose is the cumulative value of the read IO response time during the observation period; Count_r is the cumulative value of the number of read IOs during the observation period; T_average_w_respose is the average response time of write IO during the observation period, which can be used as the actual write IO response time of the software stack; T_sum_w_respose is the cumulative value of the write IO response time during the observation period; Count_w is the cumulative value of the number of write IOs during the observation period.
[0093] In step 340, the cache flush speed of the next observation cycle is adjusted based on the cache flush speed of the current observation cycle, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time.
[0094] Based on the above determined cache flush speed, cache flush expected response time, read expected response time, actual read IO response time and actual write IO response time of the current observation period, the cache flush speed of the next observation period is adjusted.
[0095] A disastrous consequence of a system overload caused by excessive cache flushing speed is that the host-side read IO response speed is seriously slowed down. Therefore, when adjusting the cache flushing speed, the read IO response speed of the software stack below the cache should also be taken into account, so that it has a certain influence ratio in the dynamic adjustment, thereby improving the read performance. Optionally, the specific adjustment principle can be as follows:
[0096] C_next=C_cur×{(T_w_flush-T_w_resp)×D 1 +(T_r_flush-T_r_resp)×D 2};
[0097] Where C_next is the cache flush speed of the next observation cycle, C_cur is the cache flush speed of the current observation cycle, T_w_flush is the expected response time of cache flush, T_r_flush is the expected response time of read, T_r_resp is the actual read IO response time, T_w_resp is the actual write IO response time, D 1 and D 2 is the weight coefficient.
[0098] It is understandable that when the refresh speed is too fast, the state of the observation feedback is that the refresh response time is too slow, and the refresh speed can be appropriately reduced, otherwise it can be appropriately increased. Based on the comprehensive consideration of the read and write IO response times, when abnormal situations occur in the backend (such as RAID reconstruction, inspection, bad disk), there is no need to build a special abnormal scenario cache processing solution. The refresh speed can be adjusted automatically only by the response time, and the software stack below the cache can automatically adjust the state to enter the optimal state of the system, so as to optimize the system read and write performance.
[0099] The cache data flushing adjustment method provided by the present invention takes into account both read and write IO response times. When an abnormal situation occurs in the backend, there is no need to build a special abnormal scenario cache processing solution. The flushing speed can be adjusted automatically only by the response time. At the same time, the software stack below the cache automatically adjusts its state to enter the optimal state of the system, thereby optimizing the system read and write performance, improving the overall processing efficiency of IO business, and thus improving storage performance.
[0100] In one embodiment, based on monitoring the reading and writing of the cache of the target storage server in the current observation cycle, determining the actual read IO response time of the software stack and the actual write IO response time of the software stack includes: monitoring the reading and writing of the cache of the target storage server in the current observation cycle, obtaining the average response time of the read IO in the current observation cycle and the average response time of the write IO in the current observation cycle; using the average response time of the read IO as the actual read IO response time, and using the average response time of the write IO as the actual write IO response time.
[0101] Wherein, the number of all read IOs in the current observation period and the cumulative value of all read IO response times in the current observation period are obtained; based on the number of all read IOs and the cumulative value, the average response time of the read IO in the current observation period is determined;
[0102] The number of all write IOs in the current observation period and the accumulated value of all write IO response times in the current observation period are obtained; based on the number of all write IOs and the accumulated value, an average response time of the write IO in the current observation period is determined.
[0103] Specifically, the actual response time is obtained by counting the average response time of all read and write IOs from the cache to the downstream software stack and the downstream software stack to the cache in the current observation period, that is, accumulating the number and response time of read and write IOs in the observation period, respectively, and then obtaining:
[0104] T_average_r_respose=T_sum_r_respose / Count_r;
[0105] T_average_w_respose=T_sum_w_respose / Count_w;
[0106] Among them, T_average_r_respose is the average response time of read IO in the current observation period, which can be used as the actual read IO response time of the software stack; T_sum_r_respose is the cumulative value of all read IO response times in the current observation period; Count_r is the number of all read IOs in the current observation period; T_average_w_respose is the average response time of write IO in the current observation period, which can be used as the actual write IO response time of the software stack; T_sum_w_respose is the cumulative value of all write IO response times in the current observation period; Count_w is the number of all write IOs in the current observation period.
[0107] In one embodiment, based on the Redundant Array of Independent Disks RAID type of the disk in the target storage server and the software stack running time during the cache data flushing process, the expected response time of the cache flushing of the target storage server is determined, including: when the RAID type of the disk is determined to be RAID0 type or RAID1 type, the expected response time of the cache flushing is: T_w_flush=T_w_disk+T_soft; when the RAID type of the disk is determined to be RAID5 type, the expected response time of the cache flushing is: T_w_flush=2×T_w_disk+2×T_r_disk+T_soft; when the RAID type of the disk is determined to be RAID6 type, the expected response time of the cache flushing is: T_w_flush=3×T_w_disk+3×T_r_disk+T_soft; wherein T_w_flush is the expected response time of the cache flushing, T_soft is the software stack running time, T_r_disk is the read IO standard response time, and T_w_disk is the write IO standard response time.
[0108] The disks are first organized into the required RAID according to the data security policy, and then the storage space composed of multiple disks is handed over to the storage pool for management, and finally provided to customers. Considering the data serialization problem, the commonly used RAID types will have the following write penalties:
[0109] Write penalty means that one write IO needs to be completed by reading and writing the disk multiple times, and the reading and writing need to be serial; no write penalty means that one write IO only needs to be completed by writing the disk once, or multiple parallel writes can be completed.
[0110] RAID0, RAID1: no write penalty;
[0111] RAID5: There is a write penalty. One write IO requires two disk reads and two disk writes to complete.
[0112] RAID6: There is a write penalty. One write IO requires three disk reads and three disk writes to complete.
[0113] There is no penalty for read IO requests under any redundancy strategy.
[0114] Based on the above characteristics, it can be determined that different RAID types have different expected response times for cache flushing.
[0115] Wherein, when it is determined that the RAID type of the disk is RAID0 or RAID1, the expected response time of the cache flush is:
[0116] T_w_flush=T_w_disk+T_soft;
[0117] When it is determined that the RAID type of the disk is RAID5, the expected response time of the cache flush is:
[0118] T_w_flush=2×T_w_disk+2×T_r_disk+T_soft;
[0119] When it is determined that the RAID type of the disk is RAID6, the expected response time of the cache flush is:
[0120] T_w_flush=3×T_w_disk+3×T_r_disk+T_soft;
[0121] Among them, T_w_flush is the expected response time for cache flushing, T_soft is the software stack running time, T_r_disk is the standard response time for read IO, and T_w_disk is the standard response time for write IO.
[0122] The read IO standard response time T_r_disk may be the optimal read IO response time of the disk, and the write IO standard response time T_w_disk may be the optimal write IO response time of the disk, which may be determined based on the optimal values recommended in various scenarios in the disk hardware manufacturer's manual.
[0123] In one embodiment, the expected read response time of the software stack is: T_r_flush=T_r_disk+T_soft; wherein T_r_flush is the expected read response time of the software stack, T_soft is the software stack running time, and T_r_disk is the read IO standard response time.
[0124] The expected read time of the software stack is a time determined by considering the response speed of the following read IO in the cache. Optionally, the determined expected read response time of the software stack can be:
[0125] T_r_flush=T_r_disk+T_soft;
[0126] Among them, T_r_flush is the expected read response time of the software stack, T_soft is the software stack running time, and T_r_disk is the read IO standard response time.
[0127] The read IO standard response time T_r_disk may be the optimal read IO response time of the disk, and may be determined based on optimal values recommended in various scenarios in the manual of the disk hardware manufacturer.
[0128] In one embodiment, based on the cache flush speed of the current observation cycle, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time, the cache flush speed of the next observation cycle is adjusted, including:
[0129] Determine a flush adjustment speed based on the cache flush speed of the current observation period, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time;
[0130] The lower brush adjustment speed is:
[0131] C_next=C_cur×{(T_w_flush-T_w_resp)×D 1 +(T_r_flush-T_r_resp)×D 2};
[0132] Among them, C_next is the flush adjustment speed, C_cur is the cache flush speed of the current observation period, T_w_flush is the cache flush expected response time, T_r_flush is the read expected response time, T_r_resp is the actual read IO response time, T_w_resp is the actual write IO response time, D 1 and D 2 is the weight coefficient, D 1 +D 2 =1;
[0133] Based on the refresh adjustment speed, the cache refresh speed of the next observation period is adjusted.
[0134] A disastrous consequence of a system overload caused by a high cache flushing speed is that the host-side read IO response speed is seriously slowed down. Therefore, when adjusting the cache flushing speed, the read IO response speed of the software stack below the cache should also be taken into account, so that it has a certain influence ratio in the dynamic adjustment, thereby improving the read performance.
[0135] The determined next brush adjustment speed C_next=C_cur×{(T_w_flush-T_w_resp)×D 1 +(T_r_flush-T_r_resp)×D 2}.
[0136] When the refresh speed is too fast, the feedback state is observed, that is, the response time of the refresh is too slow, and the refresh speed can be appropriately reduced. Otherwise, the refresh speed can be appropriately increased. This solution comprehensively considers the response time of both read and write IO. When abnormal situations occur in the backend (such as RAID reconstruction, inspection, and bad disk), there is no need to build a special abnormal scenario cache processing solution. The refresh speed can be adjusted automatically only by the response time. At the same time, the software stack below the cache automatically adjusts the state to enter the optimal state of the system, thereby optimizing the system read and write performance.
[0137] Among them, the difference between the expected response time and the actual response time of the two IOs, read and write (flush), is considered at the same time, and the refresh rate is adjusted in the form of negative feedback. At the same time, the influence of read and write in the adjustment is adjusted by the weight coefficient. That is, the weight coefficient D can be adjusted based on the 1 And the weight coefficient D 2 , further adjust the cache refresh speed. The selected weight coefficient value can be: D 1 =0.9, D 2 =0.1.
[0138] Optionally, the selection of the weight coefficient may be determined based on a negative feedback adjustment mechanism.
[0139] When the response time difference of the write (flush) operation continues to be large, it means that the refresh rate may be too low and the refresh rate needs to be increased to reduce the delay.
[0140] On the contrary, if the response time difference is small or negative (that is, the actual response time is less than the expected response time), you may need to appropriately reduce the refresh rate to avoid wasting resources.
[0141] A controller is introduced to dynamically adjust the refresh rate according to the response time difference. The controller can adopt a proportional-integral-derivative (PID) control algorithm or other suitable control strategies.
[0142] The controller outputs a corresponding adjustment signal to the down-brush system according to the size and direction of the difference to adjust the down-brush rate.
[0143] According to system requirements and application scenarios, set the proportion of read and write in the adjusted refresh rate. For example, if the system is more sensitive to the response time of read operations, you can set a higher proportion of read operations.
[0144] In one embodiment, the current observation period is determined based on the actual response time of the cache flush and the expected response time of the cache flush;
[0145] The current observation period is:
[0146] T_observation=(T_w_flush / T_resp)×100;
[0147] Among them, T_observation is the current observation period, T_w_flush is the expected response time of cache flush, and T_resp is the actual response time of cache flush.
[0148] Observed in dynamic time period, the period and actual response time show the following relationship:
[0149] T_observation=(T_w_flush / T_resp)×100;
[0150] Among them, T_observation is the current observation period, T_w_flush is the expected response time of cache flush, and T_resp is the actual response time of cache flush.
[0151] The advantages of refreshing the response time under the above dynamic period observation are:
[0152] The system feedback speed closely follows the system status. That is, when the actual response time of the backend increases, the observation cycle time becomes shorter, and the system quickly adjusts the refresh speed (reduces the refresh speed) to quickly improve the actual response time of the backend. When the actual response time of the backend is too low, the observation cycle time becomes longer, and the time for the system to adjust the refresh speed each time is appropriately increased to prevent excessive jitter in the backend response time.
[0153] The fixed period cannot adapt to the observation of different back-end states, and the optimal fixed period is difficult to determine.
[0154] Optionally, the initial value of the observation period may be set by assuming that the actual response time of the next refresh is equal to the expected response time.
[0155] The cache data flush adjustment device provided by the present invention is described below. The cache data flush adjustment device described below and the cache data flush adjustment method described above can be referred to each other.
[0156] like Figure 5 As shown, the device comprises:
[0157] A first expected time determination module 510 is used to determine the expected response time of the cache flush of the target storage server based on the RAID type of the disk in the target storage server and the software stack running time during the cache data flush process;
[0158] A second expected time determination module 520, configured to determine an expected read response time of a software stack based on a read IO standard response time of the disk and a running time of the software stack;
[0159] An actual time determination module 530, configured to determine an actual read IO response time of the software stack and an actual write IO response time of the software stack based on monitoring the read and write of the cache of the target storage server in a current observation period;
[0160] The adjustment module 540 is used to adjust the cache flush speed of the next observation cycle based on the cache flush speed of the current observation cycle, the cache flush expected response time, the read expected response time, the actual read IO response time and the actual write IO response time.
[0161] The cache data flushing adjustment device provided by the present invention takes into account both read and write IO response times. When an abnormal situation occurs in the backend, there is no need to build a special abnormal scenario cache processing solution. The flushing speed can be adjusted automatically only by the response time. At the same time, the software stack below the cache automatically adjusts its state to enter the optimal state of the system, thereby optimizing the system read and write performance, improving the overall processing efficiency of IO business, and thus improving storage performance.
[0162] In one embodiment, the actual time determination module 530 is specifically configured to:
[0163] Based on monitoring the reading and writing of the cache of the target storage server in the current observation period, determining the actual read IO response time of the software stack and the actual write IO response time of the software stack, including:
[0164] Monitor the read and write of the cache of the target storage server in the current observation period, and obtain the average response time of the read IO in the current observation period and the average response time of the write IO in the current observation period;
[0165] The average response time of the read IO is used as the actual read IO response time, and the average response time of the write IO is used as the actual write IO response time.
[0166] In one embodiment, the actual time determination module 530 is further specifically configured to:
[0167] The obtaining of the average response time of the read IO in the current observation period includes:
[0168] Obtain the number of all read IOs in the current observation period and the cumulative value of all read IO response times in the current observation period;
[0169] Based on the number of all read IOs and the accumulated value, an average response time of the read IO in the current observation period is determined.
[0170] In one embodiment, the actual time determination module 530 is further specifically configured to:
[0171] Get the average response time of write IO in the current observation period, including:
[0172] Obtain the number of all write IOs in the current observation period and the cumulative value of all write IO response times in the current observation period;
[0173] Based on the number of all write IOs and the accumulated value, an average response time of the write IO in the current observation period is determined.
[0174] In one embodiment, the first expected time determination module 510 is specifically configured to:
[0175] Determining the expected response time of the cache flush of the target storage server based on the RAID type of the disk in the target storage server and the software stack running time during the cache data flush process includes:
[0176] When it is determined that the RAID type of the disk is RAID0 or RAID1, the expected response time of the cache flush is:
[0177] T_w_flush=T_w_disk+T_soft;
[0178] When it is determined that the RAID type of the disk is RAID5, the expected response time of the cache flush is:
[0179] T_w_flush=2×T_w_disk+2×T_r_disk+T_soft;
[0180] When it is determined that the RAID type of the disk is RAID6, the expected response time of the cache flush is:
[0181] T_w_flush=3×T_w_disk+3×T_r_disk+T_soft;
[0182] Among them, T_w_flush is the expected response time for cache flushing, T_soft is the software stack running time, T_r_disk is the standard response time for read IO, and T_w_disk is the standard response time for write IO.
[0183] In one embodiment, the second expected time determination module 520 is specifically configured to:
[0184] The expected response time for reads from the software stack is:
[0185] T_r_flush=T_r_disk+T_soft;
[0186] Among them, T_r_flush is the expected read response time of the software stack, T_soft is the software stack running time, and T_r_disk is the read IO standard response time.
[0187] In one embodiment, the adjustment module 540 is specifically configured to:
[0188] Based on the cache flush speed of the current observation period, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time, adjusting the cache flush speed of the next observation period, including:
[0189] Determine a flush adjustment speed based on the cache flush speed of the current observation period, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time;
[0190] Based on the refresh adjustment speed, the cache refresh speed of the next observation period is adjusted.
[0191] In one embodiment, the adjustment module 540 is further specifically configured to:
[0192] Determine the brush down adjustment speed:
[0193] C_next=C_cur×{(T_w_flush-T_w_resp)×D 1 +(T_r_flush-T_r_resp)×D 2};
[0194] Among them, C_next is the flush adjustment speed, C_cur is the cache flush speed of the current observation period, T_w_flush is the cache flush expected response time, T_r_flush is the read expected response time, T_r_resp is the actual read IO response time, T_w_resp is the actual write IO response time, D 1 and D 2 is the weight coefficient, D1 +D 2 =1.
[0195] In one embodiment, the adjustment module 540 is further specifically configured to:
[0196] After adjusting the cache flush speed of the next observation period, the method further includes:
[0197] Based on the adjustment weight coefficient D 1 And the weight coefficient D 2 , further adjust the cache refresh speed.
[0198] In one embodiment, the actual time determination module 530 is specifically configured to:
[0199] The current observation period is determined based on the actual response time of the cache flush and the expected response time of the cache flush.
[0200] In one embodiment, the actual time determination module 530 is further specifically configured to:
[0201] Determine the current observation period as:
[0202] T_observation=(T_w_flush / T_resp)×100;
[0203] Among them, T_observation is the current observation period, T_w_flush is the expected response time of cache flush, and T_resp is the actual response time of cache flush.
[0204] An embodiment of the present invention further provides a storage server, which includes the cache data flushing adjustment device.
[0205] The storage server provided by the present invention takes into account both read and write IO response times. When an abnormal situation occurs in the back end, there is no need to build a special abnormal scenario cache processing solution. The refresh speed can be adjusted automatically only by the response time. At the same time, the software stack below the cache automatically adjusts its state to enter the optimal state of the system, thereby optimizing the system's read and write performance, improving the overall processing efficiency of IO business, and thus improving storage performance.
[0206] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the cache data flush adjustment method, the method comprising: determining the cache flush expected response time of the target storage server based on the independent disk redundant array RAID type of the disk in the target storage server and the software stack running time during the cache data flush process;
[0207] Determine the expected read response time of the software stack based on the read IO standard response time of the disk and the running time of the software stack;
[0208] Based on monitoring the reading and writing of the cache of the target storage server in the current observation period, determining the actual read IO response time of the software stack and the actual write IO response time of the software stack;
[0209] Based on the cache flush speed of the current observation period, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time, the cache flush speed of the next observation period is adjusted.
[0210] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0211] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the cache data flush adjustment method provided by the above methods, the method comprising: based on the independent disk redundant array RAID type of the disk in the target storage server, and the software stack running time during the cache data flush process, determining the cache flush expected response time of the target storage server;
[0212] Determine the expected read response time of the software stack based on the read IO standard response time of the disk and the running time of the software stack;
[0213] Based on monitoring the read and write of the cache of the target storage server in the current observation period, determining the actual read IO response time of the software stack and the actual write IO response time of the software stack;
[0214] Based on the cache flush speed of the current observation period, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time, the cache flush speed of the next observation period is adjusted.
[0215] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the cache data flush adjustment method provided by the above methods, the method comprising: determining the cache flush expected response time of the target storage server based on the independent disk redundant array RAID type of the disk in the target storage server and the software stack running time during the cache data flush process;
[0216] Determine the expected read response time of the software stack based on the read IO standard response time of the disk and the running time of the software stack;
[0217] Based on monitoring the read and write of the cache of the target storage server in the current observation period, determining the actual read IO response time of the software stack and the actual write IO response time of the software stack;
[0218] Based on the cache flush speed of the current observation period, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time, the cache flush speed of the next observation period is adjusted.
[0219] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0220] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for adjusting cache data flushing. It is characterized in that The method comprises: Determine the expected response time of the cache flush of the target storage server based on the RAID type of the disk in the target storage server and the software stack running time during the cache data flush process; Determine the expected read response time of the software stack based on the read IO standard response time of the disk and the running time of the software stack; Based on monitoring the read and write of the cache of the target storage server in the current observation period, determining the actual read IO response time of the software stack and the actual write IO response time of the software stack; Adjusting the cache flush speed of the next observation cycle based on the cache flush speed of the current observation cycle, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time; The step of determining the expected response time of the cache flush of the target storage server based on the RAID type of the disk in the target storage server and the software stack running time during the cache data flush process includes: When it is determined that the RAID type of the disk is RAID0 or RAID1, the expected response time of the cache flush is: T_w_flush=T_w_disk+T_soft; When it is determined that the RAID type of the disk is RAID5, the expected response time of the cache flush is: T_w_flush=2×T_w_disk+2×T_r_disk+T_soft; When it is determined that the RAID type of the disk is RAID6, the expected response time of the cache flush is: T_w_flush=3×T_w_disk+3×T_r_disk+T_soft; Where, T_w_flush is the expected response time for cache flushing, T_soft is the software stack running time, T_r_disk is the standard response time for read IO, and T_w_disk is the standard response time for write IO; The expected read response time of the software stack is: T_r_flush=T_r_disk+T_soft; Among them, T_r_flush is the expected read response time of the software stack, T_soft is the software stack running time, and T_r_disk is the read IO standard response time.
2. The cache data flushing adjustment method according to claim 1, It is characterized in that The determining the actual read IO response time of the software stack and the actual write IO response time of the software stack based on monitoring the read and write of the cache of the target storage server in the current observation period includes: Monitor the read and write of the cache of the target storage server in the current observation period, and obtain the average response time of the read IO in the current observation period and the average response time of the write IO in the current observation period; The average response time of the read IO is used as the actual read IO response time, and the average response time of the write IO is used as the actual write IO response time.
3. The cache data flushing adjustment method according to claim 2, It is characterized in that The obtaining of the average response time of the read IO in the current observation period includes: Obtain the number of all read IOs in the current observation period and the cumulative value of all read IO response times in the current observation period; Based on the number of all read IOs and the accumulated value, an average response time of the read IO in the current observation period is determined.
4. The cache data flushing adjustment method according to claim 2, It is characterized in that Get the average response time of write IO in the current observation period, including: Obtain the number of all write IOs in the current observation period and the cumulative value of all write IO response times in the current observation period; Based on the number of all write IOs and the accumulated value, an average response time of the write IO in the current observation period is determined.
5. The cache data flushing adjustment method according to claim 1, It is characterized in that The step of adjusting the cache flush speed of the next observation cycle based on the cache flush speed of the current observation cycle, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time includes: Determine a flush adjustment speed based on the cache flush speed of the current observation period, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time; Based on the refresh adjustment speed, the cache refresh speed of the next observation period is adjusted.
6. The cache data flushing adjustment method according to claim 5, It is characterized in that The lower brush adjustment speed is: C_next=C_cur×{(T_w_flush-T_w_resp)×D 1 +(T_r_flush-T_r_resp)×D 2 }; Among them, C_next is the flush adjustment speed, C_cur is the cache flush speed of the current observation period, T_w_flush is the cache flush expected response time, T_r_flush is the read expected response time, T_r_resp is the actual read IO response time, T_w_resp is the actual write IO response time, D 1 and D 2 is the weight coefficient, D 1 +D 2 =1.
7. The cache data flushing adjustment method according to claim 6, It is characterized in that After adjusting the cache flush speed of the next observation period, the method further includes: Based on the adjustment weight coefficient D 1 And the weight coefficient D 2 , further adjust the cache refresh speed.
8. The cache data flushing adjustment method according to claim 1, It is characterized in that The current observation period is determined based on the actual response time of the cache flush and the expected response time of the cache flush.
9. The cache data flush adjustment method according to claim 8, It is characterized in that The current observation period is: T_observation=(T_w_flush / T_resp)×100; Among them, T_observation is the current observation period, T_w_flush is the expected response time of cache flush, and T_resp is the actual response time of cache flush.
10. A cache data flush adjustment device, It is characterized in that include: A first expected time determination module, configured to determine the expected response time of a cache flush of the target storage server based on the RAID type of the disk in the target storage server and the software stack running time during the cache data flush process; A second expected time determination module, configured to determine an expected read response time of a software stack based on a read IO standard response time of the disk and a running time of the software stack; An actual time determination module, configured to determine an actual read IO response time of the software stack and an actual write IO response time of the software stack based on monitoring the read and write of the cache of the target storage server in a current observation period; An adjustment module, configured to adjust the cache flush speed of the next observation cycle based on the cache flush speed of the current observation cycle, the cache flush expected response time, the read expected response time, the actual read IO response time, and the actual write IO response time; The step of determining the expected response time of the cache flush of the target storage server based on the RAID type of the disk in the target storage server and the software stack running time during the cache data flush process includes: When it is determined that the RAID type of the disk is RAID0 or RAID1, the expected response time of the cache flush is: T_w_flush=T_w_disk+T_soft; When it is determined that the RAID type of the disk is RAID5, the expected response time of the cache flush is: T_w_flush=2×T_w_disk+2×T_r_disk+T_soft; When it is determined that the RAID type of the disk is RAID6, the expected response time of the cache flush is: T_w_flush=3×T_w_disk+3×T_r_disk+T_soft; Where, T_w_flush is the expected response time for cache flushing, T_soft is the software stack running time, T_r_disk is the standard response time for read IO, and T_w_disk is the standard response time for write IO; The expected read response time of the software stack is: T_r_flush=T_r_disk+T_soft; Among them, T_r_flush is the expected read response time of the software stack, T_soft is the software stack running time, and T_r_disk is the read IO standard response time.
11. A storage server, It is characterized in that It includes the cache data flush adjustment device as described in claim 10.
12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the cache data flushing adjustment method according to any one of claims 1 to 9 is implemented.
13. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the cache data flushing adjustment method according to any one of claims 1 to 9 is implemented.
