Page cache recovery method and device
By comprehensively evaluating page cache information and system load and dynamically adjusting the recycling strategy, the problem of operating system performance degradation caused by excessive page cache recycling is solved, and system stability and memory utilization efficiency are improved.
Patent Information
- Application Number
- CN202510945270.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-09
- Publication Date
- 2025-10-17
AI Technical Summary
In the prior art, excessive recycling of page cache may cause operating system performance degradation and even trigger an out-of-memory error (OOM), increasing the probability of operating system crashes.
By comprehensively considering the operating system's memory information, page cache information, file importance indicators, memory fragmentation level, and recently unused cache pages, the reclaimable page cache information is determined. When the cache occupancy ratio exceeds the dynamic threshold, the reclaim strategy is dynamically adjusted based on the operating system's load information and memory pressure information to avoid excessive page cache reclaim.
It effectively avoids the negative impact of page cache recycling on operating system performance, reduces the risk of crashes caused by insufficient memory, and improves system stability and performance.
Smart Images
Figure CN120803966A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of computer, in particular to a page cache recycling method and device. BACKGROUND
[0002] Linux page cache is a cache mechanism based on memory pages and oriented to files. In general, page cache can improve system performance. However, when too many files are cached in the page cache, a large amount of system memory will be occupied, resulting in insufficient system memory, which will negatively affect the performance and even cause an out-of-memory (OOM) error.
[0003] In the related art, the active file page cache value, the inactive file page cache value, and the dirty page cache value can be used to calculate the size of the actual recyclable page cache without affecting the storage data of the Linux system internal cache. When the to-be-recycled page cache value determined by the recyclable page cache value is too high relative to the Linux system memory value, a separate page cache recycling thread is used to recycle the page cache of the Linux system.
[0004] However, the method for recycling page cache in the related art may excessively recycle some types of page cache, which negatively affects the performance of the operating system and thus increases the probability of OOM problems. SUMMARY
[0005] Therefore, the present disclosure provides a page cache recycling method to solve the problem that the excessive recycling of some types of page cache in the related art negatively affects the performance of the operating system and thus increases the probability of OOM problems.
[0006] In a first aspect, the present disclosure provides a page cache recycling method, which includes: determining recyclable page cache information based on memory information of an operating system, page cache information, an importance indicator of a file, a memory fragmentation degree indicator, and a recently unused cache page; determining a recycling strategy of the page cache according to load information and memory pressure information of the operating system when the recyclable page cache information indicates that the cache occupancy ratio is greater than a dynamic threshold value; wherein the dynamic threshold value is determined based on a sliding average value of memory usage information within a target period; the memory pressure information is determined based on the memory information of the operating system; and recycling the recyclable page cache using the recycling strategy of the page cache.
[0007] The page cache recycling method in the present disclosure determines the recyclable page cache information based on the memory information of the operating system, the page cache information, the importance index of the file, the memory fragmentation degree index and the recently unused cache page, so as to comprehensively consider the source of the memory pressure information, determine the recyclable page cache information through multi-dimensional data, and avoid over-recycling some types of page cache. Then, when the cache occupancy ratio indicated by the recyclable page cache information is greater than the dynamic threshold, the recycling strategy of the page cache is determined according to the load information and the memory pressure information of the operating system, so as to dynamically adjust the recycling strategy of the page cache by using the dynamic threshold, avoid over-recycling the page cache in the high load state, and reduce the negative impact of page cache recycling on the performance of the operating system. Then, the recyclable page cache is recycled by using the recycling strategy of the page cache, so as to reasonably recycle the page cache, avoid the operating system crash or performance degradation caused by insufficient memory, and improve the stability of the operating system.
[0008] In a second aspect, the present disclosure provides a page cache recycling device, comprising: a page cache determination module configured to determine recyclable page cache information based on memory information of an operating system, page cache information, an importance index of a file, a memory fragmentation degree index and a recently unused cache page; a recycling strategy determination module configured to determine a recycling strategy of the page cache according to load information and memory pressure information of the operating system when the cache occupancy ratio indicated by the recyclable page cache information is greater than a dynamic threshold; wherein the dynamic threshold is determined based on a sliding average value of memory usage information in a target period; the memory pressure information is determined based on the memory information of the operating system; and a page cache recycling module configured to recycle the recyclable page cache by using the recycling strategy of the page cache.
[0009] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, which are communicatively connected to each other, and the memory stores computer instructions; the processor executes the computer instructions to perform the page cache recycling method of the first aspect or any of the corresponding embodiments thereof.
[0010] In a fourth aspect, the present disclosure provides a computer readable storage medium, which stores computer instructions for causing a computer to perform the page cache recycling method of the first aspect or any of the corresponding embodiments thereof.
[0011] In a fifth aspect, the present disclosure provides a computer program product, which comprises computer instructions for causing a computer to perform the page cache recycling method of the first aspect or any of the corresponding embodiments thereof. BRIEF DESCRIPTION OF DRAWINGS
[0012] In order to more clearly illustrate the technical solutions in the specific embodiments of the present disclosure or the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.
[0013] Figure 1 is a flowchart of a page cache recycling method according to an embodiment of the present disclosure;
[0014] Figure 2 is an exemplary flowchart of determining recyclable page cache information according to an embodiment of the present disclosure;
[0015] Figure 3 is a flowchart of another page cache recycling method according to an embodiment of the present disclosure;
[0016] Figure 4 is a structural block diagram of a page cache recycling device according to an embodiment of the present disclosure;
[0017] Figure 5 is a hardware structure diagram of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0018] In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of the present disclosure.
[0019] According to an embodiment of the present disclosure, a page cache recycling method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0020] In the present embodiment, a page cache recycling method is provided, which can be used in the computer system described above, Figure 1 is a flowchart of a page cache recycling method according to an embodiment of the present disclosure, as Figure 1 shown, the flow includes the following steps:
[0021] In step S101, based on the memory information of the operating system, the page cache information, the importance indicator of the file, the memory fragmentation degree indicator, and the recently unused cache page, the recyclable page cache information is determined.
[0022] In this embodiment, the memory information of the operating system is a set of memory-related data collected by the operating system through hardware monitoring and kernel management mechanisms, indicating the original data and objective description of the memory state, which can include memory capacity (total memory information, available memory information, used memory information), usage type (process occupation, cache / buffer, shared memory), performance indicator (memory access delay, page fault exception rate), hardware state (memory health, bandwidth utilization), etc.
[0023] The page cache information can include the size, distribution, and usage of the cache. The importance indicator of the file can be evaluated according to the access frequency and type of the file (such as system files and user files).
[0024] The memory fragmentation degree indicator is used to quantify the severity of fragmentation, which can be evaluated using indicators such as fragmentation rate, average free block size, free block number, allocation failure rate caused by fragmentation, and internal fragmentation rate.
[0025] The recently unused cache page can be identified by the Least Recently Used (LRU) algorithm or other strategies.
[0026] Specifically, the information of the LRU list is obtained to identify the recently unused page, which can be achieved by the following steps: first, the starting address of the LRU list can be found using kernel debugging tools or kernel symbol tables; then, the kernel-provided list operation function can be used to iterate through each node in the list; then, the relevant information of the page frame (including the flag bit of the page frame, the position information of the page frame in the LRU list, the address space structure of the page frame, and the access timestamp) can be accessed through the pointer to the page frame contained in each list node; then, whether the page frame is recently unused can be determined according to the relevant information of the page frame (such as the position information of the page frame in the LRU list and the access timestamp); finally, the process to which the page frame belongs and the file size and type are obtained.
[0027] Here, the flag bit of the page frame can be used to determine the state of the page frame, such as whether it is locked and / or whether it is a dirty page. The address space structure of the page frame can be used to obtain file information. The process to which the page frame belongs can be used to determine whether the page frame belongs to a critical process. The file size and type of the page frame can be used to determine whether the file corresponding to the page frame is important.
[0028] The memory management tool of the operating system of the computer system described above (such as a kernel thread, a daemon, a tool or a command, etc.) can determine which page cache can be recycled by comprehensively analyzing the memory information, the page cache information, the importance indicator of the file, the memory fragmentation degree indicator and the recently unused cache page of the operating system. For example, the memory information and the page cache information can be obtained by using a system tool (such as vmstat, / proc / meminfo), the memory usage can be collected and analyzed in real time by using a monitoring tool (such as Prometheus), and the unused cache page can be identified by using an LRU algorithm or a similar strategy.
[0029] In step S102, when the cache occupancy ratio of the recyclable page cache is greater than the dynamic threshold, the recycling strategy of the page cache is determined according to the load information and the memory pressure information of the operating system.
[0030] In this embodiment, the cache occupancy ratio of the recyclable page cache can be the ratio of the to-be-recycled page cache value to the total memory or the available memory of the system.
[0031] The dynamic threshold can be determined based on the sliding average value of the memory usage information in the target period. That is, the window size corresponding to the target period can be set in advance, and the memory usage data can be collected in the window to calculate the sliding average value to set the dynamic threshold. The window size here can determine the length of the data considered when calculating the sliding average value: a larger window can smooth out more short-term fluctuations, but it can also delay the response to changes in memory pressure information; a smaller window can respond more quickly to changes in memory pressure information, but it is also more susceptible to short-term fluctuations. The choice of window size can be a trade-off based on the characteristics and needs of the system. If the memory usage of the system fluctuates greatly, a larger window can be selected; if the system has a high requirement for the response speed of the memory pressure information, a smaller window can be selected. Generally, a reasonable window size can be between a few seconds and a few minutes.
[0032] The target period here can include a historical period and / or a future period.
[0033] Specifically, when the target period includes a historical period, / proc / meminfo file or other interfaces provided by the system can be used to obtain the system memory usage in the window corresponding to the historical period when collecting the data source of the memory usage data. Generally, data can be collected once per second or per few seconds, the collected data can be the percentage of used memory or the absolute value (byte number) of used memory, and the collected historical memory usage data can be stored in a data structure, such as a ring buffer or a queue. The ring buffer is more suitable for a fixed-size window, while the queue is more flexible and can handle different sizes of windows.
[0034] Specifically, when the target period includes a future period, the system memory usage demand within the window corresponding to the future period can be predicted based on a machine learning model. The specific steps are as follows: first, data collection and preprocessing, then model selection and training, then model evaluation and deployment, and then continuous monitoring and improvement.
[0035] Further, data collection can collect historical memory usage data through / proc / meminfo, system monitoring tools or monitoring indicators provided by cloud platforms, data containing timestamps and memory usage indicators (e.g. used memory, free memory, swap memory, etc.), then clean up outliers and missing values in the data, and extract meaningful features from the original data, including time features, rolling statistics, difference features, and ratio features. Scale the features to the same range.
[0036] Further, the time series prediction model can be one of the autoregressive integrated moving average model, long short-term memory network, gated recurrent unit or gradient boosting tree model. The collected data is used to train the selected model, and the data is divided into training set, validation set and test set. The training set is used to train the model, the validation set is used to adjust the model hyperparameters, and the test set is used to evaluate the generalization ability of the model.
[0037] Further, the performance of the trained model can be evaluated using the test set, and the evaluation metrics include mean squared error, root mean squared error and mean absolute error.
[0038] When determining the dynamic threshold based on the sliding average of the memory usage information in the target period, the calculation method of the sliding average is to divide the sum of all data in the target period window by the window size. For a window of size N, assuming the data at the current time point is x[n], the calculation formula of the sliding average avg[n] is as follows:
[0039] avg[n] = (x[n] + x[n-1] +... + x[n-N+1]) / N
[0040] Further, in order to improve the calculation efficiency, the recursive formula can be used to calculate the sliding average:
[0041] avg[n] = avg[n-1] + (x[n] - x[n-N]) / N
[0042] The above formula only needs to calculate the difference between the new data x[n] and the oldest data x[n-N], rather than recalculating the sum of all data.
[0043] In some specific examples, the dynamic threshold is determined based on a sliding average of the memory usage information in the target period, which can include that the dynamic threshold is determined based on a ratio of the sliding average of the memory usage information in the target period and a fixed offset; the sliding average and the fixed offset are reduced in value when the load of the operating system increases and are increased when the load of the operating system decreases.
[0044] Specifically, the threshold can be set as a ratio of the sliding average, or a fixed offset can be added, for example:
[0045] threshold=avg[n]*k+offset
[0046] where k is a coefficient (for example, 0.9 or 1.1, or a percentage), offset is an offset (for example, 5% or 100MB), and k and offset need to be adjusted according to actual conditions.
[0047] Here, the setting of the dynamic threshold can also consider the system load in addition to the sliding average of the memory usage information. For example, if the system load is high, the threshold can be set lower, i.e., the dynamic threshold is reduced when the system load increases, to reduce the frequency of memory recycling; if the system load is low, the dynamic threshold can be set higher, i.e., the dynamic threshold is increased when the system load decreases, to reduce the impact of dynamic threshold adjustment on system performance.
[0048] After the dynamic threshold is set, the memory management tool described above can determine an appropriate recycling strategy based on pre-set rules or agents according to the load information and memory pressure information of the operating system when the recyclable page cache information indicates that the cache occupancy ratio is greater than the dynamic threshold. For example, when the load information and memory pressure information meet the pre-set conditions, clean pages are preferentially recycled, dirty pages are delayed, and the like.
[0049] The load information of the operating system can include CPU load, input / output (I / O) load, and the like. The memory pressure information of the operating system can be evaluated according to memory usage, swap partition usage, and the like.
[0050] In some specific examples, the determination of the recycling strategy of the page cache according to the load information and memory pressure information of the operating system can include one, two, or three of the following:
[0051] recycling a first number of recently unused cache pages starting from the tail page of the recently unused cache pages;
[0052] recycling a second number of cache pages starting from the minimum value of the importance indicator of the file; and
[0053] The third number of cache pages are reclaimed starting from the maximum value of the memory fragmentation degree indicator.
[0054] In this example, by reclaiming the tail pages of the recently unused cache pages, the memory resource allocation can be optimized, the memory resources that have not been accessed for a long time are released, and the utilization rate of memory is improved. Reclaiming file pages with low file importance can reduce the memory occupation of files with low importance, so that the system can more efficiently process critical data and processes, thereby improving the response speed and overall performance of the system. Preferentially reclaiming cache pages with high memory fragmentation degree indicators helps to reduce the number of memory fragments and improve the continuity and efficiency of memory allocation. This not only improves the performance of memory management, but also reduces memory allocation failures and page swap frequencies caused by memory fragmentation. The first number, the second number, and the third number are only used to distinguish the number of different reclaimed objects, and can be set according to actual application scenarios or experience, which is not limited by the present application.
[0055] In step S103, the reclaimable cache pages are reclaimed using the reclaim strategy of the cache pages.
[0056] In this embodiment, the memory management tool described above can use the reclaim strategy of the cache pages determined in step S102 to reclaim the reclaimable cache pages, thereby releasing memory resources. For example, the cache pages are reclaimed using system commands (such as sync, echo 3> / proc / sys / vm / drop_caches); and / or reclaimed using automatic memory management mechanisms dependent on the operating system (such as kswapd of Linux).
[0057] The cache page reclaiming method in this embodiment determines the reclaimable cache page information based on the memory information of the operating system, the cache page information, the importance indicator of the file, the memory fragmentation degree indicator, and the recently unused cache pages, thereby comprehensively considering the sources of memory pressure information and determining the reclaimable cache page information through multi-dimensional data to avoid over-reclaiming some types of cache pages. Then, when the cache occupation ratio indicated by the reclaimable cache page information is greater than the dynamic threshold, the reclaim strategy of the cache pages is determined according to the load information and the memory pressure information of the operating system, so that the dynamic threshold is used to dynamically adjust the reclaim strategy of the cache pages, thereby avoiding over-reclaiming the cache pages at high load and reducing the negative impact of cache page reclaiming on the performance of the operating system. Then, the reclaimable cache pages are reclaimed using the reclaim strategy of the cache pages, thereby reasonably reclaiming the cache pages and avoiding system crashes or performance degradation caused by memory shortage, thereby improving the stability of the operating system.
[0058] In some optional embodiments, as shown in Figure 2 Figure 2 is an exemplary flowchart of determining the recyclable page cache information according to the embodiments of the present disclosure. That is, the step S101 can include:
[0059] In step S1011, the memory management tool obtains the system total memory information, the available memory information, the used memory information, the active page cache information, the inactive page cache information, the dirty page cache information, and the recently unused cache pages based on the interface provided by the kernel of the operating system.
[0060] In the present embodiment, the memory management tool can obtain the memory and page cache information of the system through the kernel interface of the operating system, for example, the system total memory information, the available memory information, the used memory information, the active page cache information, the inactive page cache information, the dirty page cache information, and the recently unused cache pages.
[0061] The system total memory is the total memory capacity of the system. The available memory information is the current available memory size. The used memory information is the current used memory size. The active page cache information is the page cache being used. The inactive page cache information is the set of memory pages that are not frequently accessed at present. The dirty page cache information is the page cache that has been modified but not written back to the disk. The recently unused cache page is the unused cache page identified by the LRU (Least Recently Used) algorithm or other strategies.
[0062] In some specific examples, the memory management tool can obtain detailed memory usage through the kernel interface (such as the sysinfo system call), and obtain the memory and page cache information using system tools (such as / proc / meminfo, vmstat, etc.).
[0063] In step S1012, the importance indicator of the file stored in the cache page is evaluated based on the file access frequency of the file, the file size, the file type, and the importance of the process to which the file belongs.
[0064] In the present embodiment, the file access frequency is one of the importance indicators of the file, and the importance of the file with high frequency access is usually higher. The file size can indicate the memory capacity occupied by the file, and the large file may occupy more memory, which is one of the importance indicators of the file, but the importance of the file size depends more on the access frequency. The file type is also one of the importance indicators of the file, and the importance of the system file and the high-priority user file is usually higher. The process importance is also one of the importance indicators of the file, and the importance of the file belonging to the critical process is usually higher.
[0065] In the embodiment, the memory management tool can evaluate the importance index of the file according to the file access frequency, file size, file type, and importance of the process to which the file belongs. For example, the file system monitoring tool (e.g., inotify) is used to record the file access frequency, and the process management tool (e.g., ps, top, or other process management tools) is used to evaluate the importance of the process to which the file belongs.
[0066] In step S1013, the memory fragmentation degree index of the cache page is evaluated based on the memory allocator information provided by the kernel of the operating system.
[0067] In the embodiment, the memory fragmentation includes external fragmentation (scattered unallocated memory blocks) and internal fragmentation (insufficient utilization of allocated memory blocks). The memory management tool can evaluate the memory fragmentation degree based on the memory allocator information provided by the kernel of the operating system to obtain the memory fragmentation degree index. For example, the memory fragmentation information is obtained using the kernel interface (e.g., / proc / buddyinfo), and the memory fragmentation degree is evaluated using the memory analysis tool (e.g., vmstat) to obtain the memory fragmentation degree index.
[0068] In step S1014, the recyclable cache page information is determined based on the total system memory information, available memory information, and used memory information, active page cache information, inactive page cache information, dirty page cache information, recently unused cache pages, file importance index, and memory fragmentation degree index.
[0069] In the embodiment, the recyclable cache page can include the size and location of the recyclable clean page cache, the size and location of the recyclable dirty page cache, and the estimated memory size to be released after recycling. The recyclable clean page cache, i.e., the unmodified page cache, can be directly recycled. The recyclable dirty page cache, i.e., the modified page cache, needs to be written back to the disk before recycling. The estimated memory size to be released after recycling can estimate the memory resources that can be released after recycling.
[0070] The memory management tool can determine the recyclable cache page information based on the total system memory information, available memory information, and used memory information, active page cache information, inactive page cache information, dirty page cache information, recently unused cache pages, file importance index, and memory fragmentation degree index.
[0071] The memory information of the operating system, the page cache information, the importance index of the file, the memory fragmentation degree index and the recently unused cache page in the embodiment are used to determine the recyclable page cache information. The memory utilization rate is improved by comprehensively analyzing the memory and the page cache information, releasing the memory resource, and comprehensively considering the system total memory information, the available memory information and the used memory information, the active page cache information, the inactive page cache information, the dirty page cache information, the recently unused cache page, the importance index of the file and the memory fragmentation degree index to determine the recyclable page cache information. The effectiveness of the determined recyclable page cache is optimized, the memory allocation efficiency is improved, and the system performance is avoided from being affected.
[0072] Another page cache recycling method is provided in the embodiment, which can be used in the computer system described above. Figure 3 The flowchart of another page cache recycling method according to the embodiment of the disclosure is shown in FIG. 2, which includes the following steps: Figure 3 The flowchart of another page cache recycling method according to the embodiment of the disclosure is shown in FIG. 2, which includes the following steps:
[0073] In step S201, the memory information of the operating system, the page cache information, the importance index of the file, the memory fragmentation degree index and the recently unused cache page are used to determine the recyclable page cache information.
[0074] In the embodiment, the memory information can include the used memory information, the free memory information and the cache memory information. The page cache information can include the size, distribution and usage of the cache. The importance index of the file can be evaluated according to the access frequency and type (such as system file and user file) of the file. The memory fragmentation degree index can affect the efficiency of memory allocation and the difficulty of recycling. The recently unused cache page can be identified by the least recently used (LRU) algorithm or other strategies.
[0075] The memory management tool (such as kernel thread, daemon process, tool or command) of the operating system of the computer system described above can determine the recyclable page cache information by comprehensively analyzing the memory information of the operating system, the page cache information, the importance index of the file, the memory fragmentation degree index and the recently unused cache page, that is, determining which page cache can be recycled. For example, the system tool (such as vmstat, / proc / meminfo) can be used to obtain the memory information and the page cache information, the monitoring tool (such as Prometheus) can be used to collect and analyze the memory usage in real time, and the LRU algorithm or similar strategies can be used to identify the unused cache page.
[0076] For details, please refer to Figure 1Step S101 of the illustrated embodiment will not be described here again.
[0077] Step S202, when the recyclable page cache information indicates that the cache occupancy ratio is greater than the dynamic threshold, determining the recycling strategy of the page cache according to the load information and the memory pressure information of the operating system.
[0078] In the present embodiment, the dynamic threshold can be determined based on the sliding average of the memory usage information within the target period. The load information can include CPU load, I / O load, etc. The memory pressure information can include memory usage, swap partition usage, etc.
[0079] The above-mentioned memory management tool can determine the appropriate recycling strategy based on the pre-set rules or agents according to the load information and the memory pressure information. For example, preferentially recycling clean pages, delaying recycling dirty pages, etc.
[0080] For details, please refer to Figure 1 Step S102 of the illustrated embodiment will not be described here again.
[0081] Step S203, recycling the recyclable page cache using the recycling strategy of the page cache.
[0082] In the present embodiment, the above-mentioned memory management tool can recycle the recyclable page cache using the above-mentioned recycling strategy of the page cache, thereby releasing memory resources. For example, recycling the page cache using system commands (such as sync, echo 3> / proc / sys / vm / drop_caches); and / or recycling using the automatic memory management mechanism dependent on the operating system (such as kswapd of Linux).
[0083] For details, please refer to Figure 1 Step S103 of the illustrated embodiment will not be described here again.
[0084] Step S204, monitoring the memory load, the page cache size and the disk input / output load.
[0085] In the present embodiment, the memory load is the pressure degree of the system memory resources being used, which is a dynamic evaluation of the memory state. It can be measured by the following indicators: calculated by key indicators in the memory information (such as used memory occupancy ratio, swap space usage, page fault exception rate), reflecting whether the memory meets the current workload demand. The page cache size can include active page cache size, inactive page cache size, dirty page cache size, etc. The disk input / output (I / O) load can include disk read / write rate, I / O latency, etc.
[0086] The memory management tool can monitor the memory load, the page cache size, and the disk I / O load in real time after the page cache is recycled to evaluate the effect of the recycling strategy. In some examples, the memory load, the page cache size, and the disk I / O load data can be collected and displayed in real time by using a monitoring tool such as Prometheus and Grafana, and detailed monitoring data can be obtained by using a system tool such as vmstat and iostat.
[0087] In step S205, the effectiveness index of the recycling strategy of the page cache is evaluated according to the memory load, the page cache size, and the disk I / O load.
[0088] In this embodiment, the memory load can indicate the memory usage. If the memory usage rate (used memory / total available memory x 100%, also referred to as used memory ratio) is continuously higher than a certain threshold, it indicates that the system memory resource is insufficient, and the recycling strategy needs to be optimized to release part of the memory. If the memory usage rate is low, but the system performance is still not ideal, the recycling strategy needs to be adjusted to improve the memory resource utilization efficiency. Monitoring the page cache size can indicate the system page cache condition. If the page cache continuously increases or decreases, the recycling strategy needs to be adjusted to balance the memory resource utilization. High disk I / O load usually indicates that the system frequently performs disk read / write operations, which may be related to insufficient memory resources, and the recycling strategy needs to be optimized to reduce the disk I / O operation and improve the system performance.
[0089] In this embodiment, the memory utilization rate can indicate whether the memory usage after recycling is reasonable. The page cache hit rate can indicate whether the page cache hit rate after recycling is improved. The disk I / O load can indicate whether the disk I / O load after recycling is reduced.
[0090] The memory management tool can evaluate the effectiveness index of the recycling strategy of the page cache according to the monitored memory load, the page cache size, and the disk I / O load. For example, a visual effectiveness index report can be generated by using a data analysis tool such as Kibana and Tableau to combine historical data and baseline data to determine the effect of the recycling strategy.
[0091] Another page cache recycling method in this embodiment is compared with the page cache recycling method shown in Figure 1 Compared with the page cache recycling method shown in
[0092] In some optional embodiments, the memory management tool can monitor the memory load, the page cache size, and the disk I / O load in real time after the page cache is recycled to evaluate the effect of the recycling strategy. In some examples, the memory load, the page cache size, and the disk I / O load data can be collected and displayed in real time by using a monitoring tool such as Prometheus and Grafana, and detailed monitoring data can be obtained by using a system tool such as vmstat and iostat. Figure 2The other page cache recovery method shown can also include:
[0093] In step S206, the dynamic threshold is dynamically adjusted according to the effectiveness index of the page cache recovery strategy.
[0094] In this embodiment, the trends of changes in memory usage, page cache size, and disk I / O load indicators can be continuously monitored. Then, based on the changes in the monitoring data, it can be evaluated whether the current weight coefficient and recovery threshold are appropriate. Then, based on the evaluation results, the weight coefficient is dynamically adjusted, and / or the size of the dynamic threshold for page cache recovery is adjusted to optimize the recovery strategy.
[0095] Specifically, in optimizing the recovery strategy, the following methods can be used: 1) adjusting parameters according to monitoring data: dynamically adjusting the weight coefficient or parameter of the calculation formula of the dynamic threshold according to the real-time monitoring data to adapt to the changes in system memory resource utilization. 2) Regular evaluation and optimization: regularly evaluate the memory usage, page cache size, and disk I / O load of the system, find problems and adjust the recovery strategy in time to maintain the stability and efficiency of the system performance. 3) Combined with actual scenarios: optimize the recovery strategy according to the specific system requirements and application scenarios to ensure efficient operation of the system under different workloads.
[0096] In this embodiment, the memory management tool described above can dynamically adjust the dynamic threshold according to the effectiveness index of the page cache recovery strategy to ensure the rationality and flexibility of the recovery strategy. For example, the sliding window algorithm can be used to calculate the sliding average of the memory usage information, and then the calculation formula or parameter of the dynamic threshold is adjusted according to the effectiveness index.
[0097] In some specific examples, when the fluctuation of the memory usage information within the target period indicated by the effectiveness index is greater than the fluctuation threshold, the data collection window of the memory usage information within the target period can be expanded; when the response speed of the operating system to the memory pressure information indicated by the effectiveness index is less than the response threshold, the data collection window of the memory usage information within the target period can be reduced.
[0098] The memory usage information is a set of raw data collected by the operating system or monitoring tools about the memory state, covering multiple dimensions and details of the memory. For example, it can include: basic information: total memory information, available memory information, used memory information, free memory information; usage type: process memory occupation, cache (Cache), buffer (Buffers), shared memory, swap usage; performance indicators: memory access delay, page fault rate, memory bandwidth utilization; hardware status: memory health status (such as ECC error count), NUMA node distribution (multi-core CPU scenario).
[0099] In this example, the memory usage information fluctuates greatly, indicating that the system memory usage may be unstable or abnormal. Expanding the data collection window can capture more memory usage data, providing a more comprehensive understanding of memory usage trends and patterns, which helps identify and solve memory usage problems. If the operating system responds slowly to memory pressure information, the system needs to monitor memory usage more frequently to make timely adjustments. Reducing the data collection window can shorten the time interval for data collection, allowing the system to quickly respond to memory pressure information.
[0100] In this embodiment, the dynamic threshold is dynamically adjusted according to the effectiveness index of the page cache recovery strategy, which can ensure that the recovery strategy can adapt to changes in memory usage, improve the rationality and flexibility of the recovery strategy, and thus optimize the performance of the operating system.
[0101] In summary, the system page cache recovery method in the present disclosure can reduce unnecessary cache occupation by more finely evaluating which page cache can be recovered, thereby releasing more available memory to other processes. In particular, by evaluating file importance, the recovery of frequently accessed important data is avoided, maximizing memory utilization. At the same time, considering the memory fragmentation degree index, the recovery strategy can be guided to avoid excessive memory fragmentation, thereby improving memory allocation efficiency and reducing memory allocation failures. More efficient memory management reduces the number of page cache replacements and reduces disk I / O load, thereby improving overall system performance and response speed.
[0102] Further, by dynamically adjusting the threshold value and selecting a suitable recovery strategy, the system can adapt to changing system load and memory pressure information, and the system does not rely too much on static configuration, but adjusts according to real-time conditions, ensuring the stability and flexibility of the system. By comprehensively considering multiple factors including memory usage, LRU information, file importance, and memory fragmentation, the method is more accurate than the related art recovery strategy based on a simple LRU algorithm, and more effectively manages the page cache, thereby improving cache hit rate, and solves the problem in the related art that a simple addition and subtraction method is used to calculate the recoverable page cache value, and many important factors are ignored, for example, the importance indicator of the file, the LRU list of the page cache, memory fragmentation, the write-back speed of dirty pages, and the source of memory pressure information, and a single threshold value is used to determine whether to trigger page cache recovery, which is too simple, and lacks monitoring and adjustment of the recovery process, and also ignores the types of page cache and does not distinguish between them, which may cause excessive recovery of some types of page cache.
[0103] In the present embodiment, a page cache recovery device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and contemplated.
[0104] The present embodiment provides a page cache recovery device, as shown in Figure 4 The device comprises:
[0105] The page cache determination module 401 is configured to determine the recoverable page cache information based on the memory information of the operating system, the page cache information, the importance indicator of the file, the memory fragmentation degree indicator, and the recently unused cache page.
[0106] The recovery strategy determination module 402 is configured to determine the recovery strategy of the page cache according to the load information and the memory pressure information of the operating system when the recoverable page cache information indicates that the cache occupancy ratio is greater than the dynamic threshold value; wherein the dynamic threshold value is determined based on the sliding average value of the memory usage information in the target period; and the memory pressure information is determined based on the memory information of the operating system.
[0107] The page cache recovery module 403 recovers the recoverable page cache using the recovery strategy of the page cache.
[0108] In some optional embodiments, the page cache determination module 401 comprises (not shown in the figure):
[0109] The information obtaining submodule is configured to obtain system total memory information, available memory information, and used memory information, active page cache information, inactive page cache information, dirty page cache information, and recently unused cache pages based on an interface provided by a kernel of an operating system.
[0110] The importance evaluating submodule is configured to evaluate an importance indicator of a file stored in a cache page based on a file access frequency of the file, a file size of the file, a file type of the file, and an importance of a process to which the file belongs.
[0111] The fragmentation evaluating submodule is configured to evaluate a memory fragmentation degree indicator of a cache page based on memory allocator information provided by a kernel of an operating system.
[0112] The page cache determining submodule is configured to determine recyclable page cache information based on the system total memory information, the available memory information, and the used memory information, the active page cache information, the inactive page cache information, the dirty page cache information, the recently unused cache pages, the importance indicator of the file, and the memory fragmentation degree indicator. The recyclable page cache information includes a size and a location of recyclable clean page cache, a size and a location of recyclable dirty page cache, and a size of memory to be released after recycling.
[0113] In some optional embodiments, the recycling strategy determining module 402 is further configured to:
[0114] According to the load information of the operating system and the memory pressure information, at least one of the following is performed:
[0115] Recycling a first number of the recently unused cache pages starting from a tail page of the recently unused cache pages;
[0116] Recycling a second number of the cache pages starting from a minimum value of the importance indicator of the file; and
[0117] Recycling a third number of the cache pages starting from a maximum value of the memory fragmentation degree indicator.
[0118] In some optional embodiments, the apparatus further includes:
[0119] The effective data monitoring module 404 is configured to monitor memory load, page cache size, and disk I / O load.
[0120] The effective indicator evaluating module 405 is configured to evaluate an effectiveness indicator of the recycling strategy of the page cache according to the memory load, the page cache size, and the disk I / O load.
[0121] In some optional embodiments, the apparatus further includes:
[0122] The dynamic threshold adjusting module 406 is configured to dynamically adjust the dynamic threshold according to the effectiveness index of the reclaiming strategy of the page cache.
[0123] In some optional embodiments, dynamically adjusting the dynamic threshold according to the effectiveness index of the reclaiming strategy of the page cache comprises:
[0124] The window expanding submodule is configured to expand the data collection window of the memory usage information in the target period when the effectiveness index indicates that the fluctuation of the memory usage information in the target period is greater than the fluctuation threshold.
[0125] The window reducing submodule is configured to reduce the data collection window of the memory usage information in the target period when the effectiveness index indicates that the response speed of the operating system to the memory pressure information is less than the response threshold.
[0126] In some optional embodiments, the dynamic threshold in the reclaiming strategy determining module 402 is determined based on a sliding average value of the memory usage information in the target period, comprising:
[0127] The dynamic threshold is determined based on a percentage of the sliding average value of the memory usage information in the target period and a fixed offset.
[0128] The percentage and the fixed offset decrease in value when the load information of the operating system increases, and increase in value when the load information of the operating system decreases.
[0129] Further function descriptions of the above-mentioned modules and units are the same as those of the corresponding embodiments, and will not be repeated here.
[0130] The page cache reclaiming device in the embodiment is presented in the form of functional modules. The modules herein can be software units, fixed programs, ASIC (Application Specific Integrated Circuit) circuits, processors and memories of ASIC circuits executing one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.
[0131] Please refer to Figure 5 , Figure 5Fig. 1 is a structural schematic diagram of a computer device provided by an optional embodiment of the present disclosure. The computer device includes one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other using different buses, and can be mounted on a common mainboard or mounted in other manners as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or graphical information of a GUI stored in the memory for displaying on an external input / output device, such as a display device coupled to the interface. In some optional embodiments, multiple processors and / or multiple buses can be used together with multiple memories and multiple memory banks, if needed. Also, multiple computer devices can be connected, each device providing part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).
[0132] The processor 10 can be a central processor, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.
[0133] The memory 20 stores instructions executable by the at least one processor 10, so that the at least one processor 10 performs the method shown in the above embodiments.
[0134] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some optional embodiments, the memory 20 can optionally include a memory remotely arranged with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.
[0135] The memory 20 can include a volatile memory, such as a random access memory; the memory can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state disk; and the memory 20 can further include a combination of the above kinds of memories.
[0136] The embodiments of the present disclosure further provide a computer readable storage medium, and the method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or recorded in a storage medium, or be implemented as computer code originally stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded through a network and stored in a local storage medium, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk or a solid state disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that the computer, processor, microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor or hardware, the method shown in the above embodiments is implemented.
[0137] Part of the present disclosure can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present disclosure can be called or provided. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source file, executable file, installation package file, etc., accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.
[0138] Although the embodiments of the present disclosure are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present disclosure, and such modifications and changes fall within the scope defined by the appended claims.
Claims
1. A page cache recovery method, characterized in that: The method comprises: Determine reclaimable page cache information based on the operating system's memory information, page cache information, file importance index, memory fragmentation index, and recently unused cache pages; When the reclaimable page cache information indicates that the cache occupancy ratio is greater than a dynamic threshold, determining a page cache reclaim strategy based on load information and memory pressure information of the operating system; wherein the dynamic threshold is determined based on a sliding average of memory usage information within a target period; and the memory pressure information is determined based on memory information of the operating system; The reclaimable page cache is reclaimed using the page cache reclaim policy.
2. The method according to claim 1, characterized in that Determining recyclable page cache information based on operating system memory information, page cache information, file importance index, memory fragmentation index, and recently unused cache pages includes: Based on the interface provided by the kernel of the operating system, obtain system total memory information, available memory information and used memory information, active page cache information, inactive page cache information, dirty page cache information, and cache pages that have not been used recently; Evaluate the importance of files stored in the cache page based on file access frequency, file size, file type, and the process to which the files belong; Evaluating a memory fragmentation index of a cache page based on memory allocator information provided by a kernel of the operating system; Based on the system total memory information, available memory information and used memory information, active page cache information, inactive page cache information, dirty page cache information, cache pages that have not been used recently, the importance index of the file, and the memory fragmentation level index, the reclaimable page cache information is determined; wherein the reclaimable page cache information includes: the size and location of the reclaimable clean page cache, the size and location of the reclaimable dirty page cache, and the memory size expected to be released after recycling.
3. The method according to claim 2, characterized in that Determining a page cache reclaim strategy based on operating system load information and memory pressure information includes: Based on the load and memory pressure information of the operating system, perform at least one of the following: Starting from the last page of the cached pages that have not been used recently, reclaiming a first number of cached pages that have not been used recently; Starting from the minimum value of the importance index of the file, reclaiming a second number of cache pages; and Starting from the maximum value of the memory fragmentation level indicator, a third number of cache pages are recycled.
4. The method according to claim 1, wherein The method further comprises: Monitor memory load, page cache size, and disk I / O load; An effectiveness index of the page cache reclaim strategy is evaluated based on the memory load, page cache size, and disk input / output load.
5. The method according to claim 4, characterized in that The method further comprises: The dynamic threshold is dynamically adjusted according to an effectiveness index of the reclaiming strategy of the page cache.
6. The method according to claim 5, characterized in that The dynamically adjusting the dynamic threshold according to the effectiveness index of the page cache recycling strategy includes: When the fluctuation of the memory usage information within the target period indicated by the effectiveness indicator is greater than a fluctuation threshold, expanding the data collection window of the memory usage information within the target period; When the response speed of the operating system to the memory pressure information indicated by the effectiveness indicator is less than a response threshold, the data collection window for the memory usage information within the target cycle is reduced.
7. The method according to any one of claims 1 to 6, characterized in that The dynamic threshold is determined based on a sliding average of memory usage information within a target period, including: The dynamic threshold is determined based on a ratio of a sliding average of memory usage information within a target period and a fixed offset; The ratio of the sliding average value and the fixed offset decreases when the load of the operating system increases, and increases when the load of the operating system decreases.
8. A page cache recovery device, characterized in that: The device comprises: A page cache determination module is used to determine reclaimable page cache information based on the operating system's memory information, page cache information, a file importance index, a memory fragmentation index, and cache pages that have not been used recently; a reclaim strategy determination module, configured to determine a page cache reclaim strategy based on operating system load information and memory pressure information when the reclaimable page cache information indicates that the cache occupancy ratio is greater than a dynamic threshold; wherein the dynamic threshold is determined based on a sliding average of memory usage information within a target period; and the memory pressure information is determined based on memory information of the operating system; The page cache recycling module is configured to recycle the recyclable page cache using the page cache recycling policy.
9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the page cache reclaiming method according to any one of claims 1 to 7 by executing the computer instructions.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the page cache reclaiming method according to any one of claims 1 to 7.