Cache trimming method and device, equipment, storage medium and program product

Abstract

The invention provides a cache trimming method and device, equipment, a storage medium and a program product, and relates to the field of cache space management. The method comprises the following steps: initializing a cache space to be trimmed; obtaining a trimming task, and obtaining space information of the cache space to be trimmed according to the trimming task; traversing all files in the cache space to be pruned through a depth-first search algorithm according to the file information; performing first verification according to the file information and a timestamp verification strategy to obtain a non-hot file set; checking the non-hot file set for the second time according to the file information and a state checking strategy to obtain a file set in a normal state; and trimming the file set in the normal state according to the file information and the trimming strategy to obtain a trimmed cache space. The technical problem that in the prior art, due to the fact that a trimming strategy cannot accurately recognize a hot file which is being accessed or modified by a service process, valuable data may be deleted by mistake, and the cache trimming efficiency is low is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cache space management, and in particular to a cache pruning method and device, equipment, storage medium and program product. BACKGROUND

[0002] With the rapid development of Internet technology, in the field of distributed storage, cloud computing and big data analysis, cache space management is the key link to ensure system performance and cost-effectiveness. Local cache disk, mainly through solid state disk or non-volatile memory express (NVMe) storage device, has a fixed physical capacity. This means that as data is continuously written and accumulated, the cache space will eventually be saturated. In order to continue to accommodate new data and maintain the efficient operation of the system, it is necessary to regularly recycle the space occupied by data that is no longer needed or has a low access frequency.

[0003] Currently, in the traditional cache pruning strategy, it is usually only based on the modification time or access time of the file to determine whether the file is a "cold file", i.e. data that is no longer needed.

[0004] However, the existing pruning strategy cannot accurately identify hot files that are being accessed or modified by business processes, and cannot accurately identify and preferentially recycle those data that are truly no longer needed, and thus may mistakenly delete valuable data, thereby resulting in low cache pruning efficiency. SUMMARY

[0005] The present application provides a cache pruning method, device, equipment, storage medium and program product to solve the problem that the existing pruning strategy cannot accurately identify hot files that are being accessed or modified by business processes, and cannot accurately identify and preferentially recycle those data that are truly no longer needed, and thus may mistakenly delete valuable data, thereby resulting in low cache pruning efficiency.

[0006] In a first aspect, the present application provides a cache pruning method, comprising:

[0007] initializing a cache space to be pruned;

[0008] obtaining a pruning task and obtaining space information of the cache space to be pruned according to the pruning task; wherein the space information includes file information;

[0009] According to the file information, traversing all files in the cache space to be pruned by using a depth-first search algorithm;

[0010] According to the file information and a timestamp verification strategy, performing a first verification to obtain a non-hot file set;

[0011] According to the file information and a state verification strategy, performing a second verification on the non-hot file set to obtain a state normal file set;

[0012] According to the file information and the pruning strategy, the state normal file set is pruned to obtain a pruned cache space.

[0013] In a possible design, the file information further includes metadata information, and the metadata information includes last access time stamps and last modification time stamps corresponding to the plurality of files.

[0014] The timestamp verification strategy includes a configuration threshold.

[0015] According to the file information and the timestamp verification strategy, first verification is performed to obtain a non-hot file set, including:

[0016] According to the file information, metadata information is called to obtain the last access time stamp and the last modification time stamp corresponding to each file.

[0017] It is determined whether the last access time stamp or the last modification time stamp corresponding to the first current file is less than the configuration parameter.

[0018] If the last access time stamp or the last modification time stamp corresponding to the first current file is less than the configuration parameter, the first current file is marked as a hot file, and protection is triggered.

[0019] If the last access time stamp or the last modification time stamp corresponding to the first current file is greater than or equal to the configuration parameter, the first current file is included in the non-hot file set.

[0020] Then, first verification is performed on the next file, until first verification is completed on all files in the cache space to be pruned, and the non-hot file set is obtained.

[0021] In a possible design, the metadata information further includes state data corresponding to the plurality of files, and the state data includes reference count data and a file lock.

[0022] According to the file information and the state verification strategy, second verification is performed on the non-hot file set to obtain a state normal file set, including:

[0023] According to the non-hot file set, metadata information is called to obtain the reference count data and the file lock corresponding to each file in the non-hot file set.

[0024] It is determined whether the reference count data corresponding to the second current file is greater than zero.

[0025] If the reference count data corresponding to the second current file is greater than zero, the second current file is in an abnormal state, and protection is triggered.

[0026] If the reference count data corresponding to the second current file is not greater than zero, the second current file is in a normal state, and the second current file is included in the state normal file set.

[0027] determining whether the file lock corresponding to the second current file is released;

[0028] if the file lock corresponding to the second current file is not released, the second current file is in an abnormal state, triggering protection;

[0029] if the file lock corresponding to the second current file is released, the second current file is in a normal state, and the second current file is included in the set of files in a normal state;

[0030] then, the second current file is checked again, until all files in the set of non-hot files are checked, and the set of files in a normal state is obtained.

[0031] In a possible design, the file information further includes physical occupation values corresponding to the plurality of files;

[0032] The pruning strategy includes a preset control amount;

[0033] The metadata includes an actual release amount;

[0034] The set of files in a normal state is pruned according to the file information and the pruning strategy, including:

[0035] All files in the set of files in a normal state are arranged in descending order of the corresponding physical occupation values, to obtain a sorted file set;

[0036] The first file in the sorted file set is pruned;

[0037] The actual release amount of the metadata information is queried, and the actual release amount is compared with the preset control amount;

[0038] If the actual release amount is less than the control amount, the control amount is updated, and the first file in the sorted file set is continuously pruned;

[0039] If the actual release amount is equal to the preset control amount, the pruning is ended;

[0040] If the actual release amount is greater than the preset control amount, the pruning is ended, and an alarm is issued.

[0041] In a possible design, the to-be-pruned cache space further includes a mutex, used to control all related data of the pruning task;

[0042] After all files in the to-be-pruned cache space are traversed according to the file information by using the depth-first search algorithm, the following is further included:

[0043] When it is detected that the to-be-pruned cache space is acquired by other tasks, a waiting instruction is triggered, and the pruning task enters an intelligent waiting state through the waiting instruction, and a waiting period is started;

[0044] Releasing the target files involved by the other task through the mutex, and executing the other task.

[0045] In a possible design, after releasing the target files involved by the other task through the mutex and executing the other task, the method further includes:

[0046] When the waiting period ends, checking whether a stop instruction is received;

[0047] If the stop instruction is received, immediately terminating the pruning task, and releasing all data related to the pruning task through the mutex;

[0048] If the stop instruction is not received, automatically resuming the pruning task.

[0049] In a second aspect, the present application provides a cache pruning apparatus, including:

[0050] An initialization module, configured to initialize a to-be-pruned cache space;

[0051] An obtaining module, configured to obtain a pruning task, and obtain space information of the to-be-pruned cache space according to the pruning task; wherein the space information includes file information;

[0052] A first processing module, configured to traverse all files in the to-be-pruned cache space according to the file information through a depth-first search algorithm;

[0053] A first checking module, configured to perform a first check according to the file information and a timestamp checking strategy, to obtain a non-hot file set;

[0054] A second checking module, configured to perform a second check on the non-hot file set according to the file information and a state checking strategy, to obtain a state-normal file set;

[0055] A second processing module, configured to perform pruning processing on the state-normal file set according to the file information and a pruning strategy, to obtain a pruned cache space.

[0056] In a third aspect, the present application provides an electronic device, including a memory and a processor.

[0057] The memory stores computer-executed instructions;

[0058] The processor executes the computer-executed instructions stored in the memory, so that the processor executes the cache pruning method according to the first aspect.

[0059] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer-executed instructions, and the computer-executed instructions are executed by the processor to implement the cache pruning method according to the first aspect.

[0060] In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the cache pruning method of the first aspect.

[0061] The cache pruning method, device, equipment, storage medium and program product provided by the present application comprise: initializing a to-be-pruned cache space; obtaining a pruning task, and obtaining space information of the to-be-pruned cache space according to the pruning task; wherein the space information comprises file information; traversing all files in the to-be-pruned cache space according to a depth-first search algorithm according to the file information; performing a first verification according to the file information and a timestamp verification strategy to obtain a non-hot file set; performing a second verification on the non-hot file set according to the file information and a state verification strategy to obtain a state-normal file set; and performing pruning processing on the state-normal file set according to the file information and a pruning strategy to obtain a pruned cache space. Compared with the prior art, the pruning strategy cannot accurately identify hot files that are being accessed or modified by a business process, cannot accurately identify and preferentially recycle data that is truly no longer needed, and thus may mistakenly delete valuable data, thereby resulting in low cache pruning efficiency. The present application accurately protects key business files through hot file identification, avoids mistaken deletion, maximizes space release efficiency through efficient traversal, reduces fragmentation, and ensures the dynamic balance between pruning operation and business load through a cooperative control mechanism, thereby improving cache pruning efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0063] Figure 1 A system architecture of a cache pruning method provided by an embodiment of the present application Figure 1

[0064] Figure 2 A cache pruning method flow provided by an embodiment of the present application Figure 1

[0065] Figure 3 A cache pruning method flow provided by an embodiment of the present application Figure 2

[0066] Figure 4 A cache pruning method flow provided by an embodiment of the present application Figure 3

[0067] Figure 4 ​​​​A cache pruning method flowchart provided by an embodiment of the present application Figure 5 ;

[0068] Figure 6 A cache pruning method flowchart provided by an embodiment of the present application Figure 5 ;

[0069] Figure 6 A structure diagram of a cache pruning device provided by an embodiment of the present application

[0070] Figure 1 A structure diagram of an electronic device provided by an embodiment of the present application DETAILED DESCRIPTION

[0071] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements in the several figures. The following description of exemplary embodiments is not representative of all possible embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0072] In the embodiments of the present application, the same items or similar items with basically same functions and effects are distinguished by using "first", "second", and the like. Those skilled in the art can understand that "first", "second", and the like do not limit the quantity and execution order, and "first", "second", and the like do not necessarily mean different. It should be noted that in the embodiments of the present application, "exemplary" or "for example" is used to indicate an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, "exemplary" or "for example" is used to present the relevant concept in a specific manner. In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more.

[0073] It should be noted that "at" in the embodiments of the present application can be the moment when a certain condition occurs, or a period of time after a certain condition occurs, which is not specifically limited in the embodiments of the present application. In addition, the cache pruning method provided by the embodiments of the present application is only an example, and the cache pruning method can include more or less content.

[0074] With the rapid development of Internet technology, in the field of distributed storage, cloud computing and big data analysis, cache space management is the key link to ensure system performance and cost-effectiveness. It is mainly managed through local cache disks such as solid state drives (SSD) or non-volatile memory express (NVMe) storage devices, which all have fixed physical capacities.

[0075] This means that as data is continuously written and accumulated, the cache space will eventually become saturated. In order to continue to accommodate new data and maintain the efficient operation of the system, it is necessary to regularly recycle the space occupied by data that is no longer needed or has low access frequency. However, traditional space recycling strategies often lack intelligence and may not accurately identify and prioritize recycling of data that is truly no longer needed, resulting in low cache space utilization and even the accidental deletion of valuable data.

[0076] In traditional cache pruning strategies, it is usually only based on the modification time or access time of a file to determine whether the file is a "cold file", i.e. data that is no longer needed. However, this strategy has significant limitations.

[0077] On the one hand, pruning strategies based on a single timestamp (such as st_mtime) or static rules cannot accurately identify "hot files" that are being accessed or modified by business processes. These hot files may be in use by database transactions, frequently read by applications, or in other critical operations. This can easily lead to data corruption or business interruption.

[0078] On the other hand, traditional traversal algorithms (such as breadth-first search) do not prioritize large files, resulting in severe fragmentation and low storage utilization.

[0079] On the other hand, the pruning process competes with business processes for lock resources or I / O bandwidth, causing business delays to skyrocket.

[0080] On the other hand, traditional strategies may not be able to adapt to the timestamp update mechanism of the file system, such as some file systems that may use a lazy update strategy, resulting in inaccurate timestamp information.

[0081] To solve the above problems, the inventors found, in the process of researching the low efficiency of cache pruning, that the traditional cache pruning method cannot accurately identify hot files being accessed or modified by business processes, cannot accurately identify and preferentially recover data that is truly no longer needed, and thus may mistakenly delete valuable data. Accordingly, the inventors considered accurately protecting key business files through hot file identification, avoiding mistaken deletion; maximizing space release efficiency through efficient traversal, reducing fragmentation; ensuring dynamic balance between pruning operations and business loads through a cooperative control mechanism. Through dynamic time window configuration and file state secondary verification, the application achieves differentiated protection of online transaction processing and analytical business; through depth-first traversal and large file priority strategy, conditional variable intelligent waiting and dynamic wake-up interface, the application achieves dynamic balance between pruning operations and business loads, avoiding performance jitter. Based on this, the embodiments of the application provide a cache pruning method, device, equipment, storage medium and program product, which can be used in the field of cache space management and aims to solve the problem of low cache pruning efficiency in the prior art.

[0082] The technical solutions of the application and how the technical solutions solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the application will be described below with reference to the drawings.

[0083] Figure 1 A system architecture diagram of a cache pruning method provided by the embodiments of the application is shown in the figure, and the cache pruning system is a computer device. Figure 1 In the above architecture, the above architecture includes at least one of a data acquisition device 101, a processing device 102 and a display device 103.

[0084] It can be understood that the structure shown in the embodiments of the application does not constitute a specific limitation on the processing system architecture of the cache pruning method. In other possible embodiments of the application, the above architecture can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements, which can be determined according to actual application scenarios, and are not limited herein. Figure 2 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0085] In the specific implementation process, the data acquisition device 101 can include an input / output interface and can also include a communication interface. The data acquisition device 101 can be connected to the processing device through the input / output interface or the communication interface.

[0086] The processing device 102 can prune the cache space to be pruned.

[0087] The display device 103 can also be a touch display screen or a screen of a terminal device for receiving user instructions while displaying the above-mentioned content to realize interaction with the user.

[0088] It should be understood that the above processing device can be implemented by a processor reading instructions in a memory and executing the instructions, or by a chip circuit.

[0089] In addition, the network architecture and service scenarios described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of network architecture and the appearance of new service scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0090] The technical solutions of the present application will be described in detail below in combination with specific embodiments:

[0091] Figure 1 A cache pruning method flow provided by an embodiment of the present application Figure 2 As shown in the method, the method comprises: Figure 3

[0092] S201, initializing a to-be-pruned cache space.

[0093] S202, obtaining a pruning task and acquiring space information of the to-be-pruned cache space according to the pruning task.

[0094] The space information comprises file information.

[0095] S203, traversing all files in the to-be-pruned cache space according to the file information by using a depth-first search algorithm.

[0096] For example, a depth-first search strategy is used to realize recursive traversal of a directory tree by using opendir / readdir system calls.

[0097] Optionally, directory expansion: a directory is opened by using opendir, and entries (dirent structure) are read one by one by using readdir.

[0098] Optionally, type filtering: the d_type field is parsed, special directories are skipped, and files and subdirectories are distinguished.

[0099] Optionally, recursive call: the _do_trim function is recursively called for a subdirectory to form a tree-shaped pruning topology, and it is ensured that all levels of files are covered.

[0100] The traversal order adopts a "file first, subdirectory postorder" strategy, that is, files in the current directory are processed first, and then subdirectories are recursively processed.​

[0101] In the path management, a complete path (such as / cache / data / file.txt) is generated by string splicing to ensure the accuracy of cross-directory operations.

[0102] S204, first verification according to the file information and the timestamp verification policy, to obtain a non-hot file set.

[0103] S205, second verification of the non-hot file set according to the file information and the state verification policy, to obtain a state normal file set.

[0104] S206, pruning processing of the state normal file set according to the file information and the pruning policy, to obtain a pruned cache space.

[0105] The cache pruning method provided in the embodiment includes: initializing a to-be-pruned cache space; obtaining a pruning task, and acquiring space information of the to-be-pruned cache space according to the pruning task; wherein the space information includes file information; traversing all files in the to-be-pruned cache space according to the file information by using a depth-first search algorithm; first verification according to the file information and a timestamp verification policy, to obtain a non-hot file set; second verification of the non-hot file set according to the file information and a state verification policy, to obtain a state normal file set; and pruning processing of the state normal file set according to the file information and a pruning policy, to obtain a pruned cache space. Compared with the prior art, the pruning policy cannot accurately identify hot files that are being accessed or modified by a business process, cannot accurately identify and preferentially recycle data that is truly no longer needed, and thus valuable data may be mistakenly deleted, thereby resulting in low cache pruning efficiency. The application accurately protects key business files through hot file identification, avoids mistaken deletion, maximizes space release efficiency through efficient traversal, reduces fragmentation, and ensures the dynamic balance between pruning operations and business loads through a cooperative control mechanism, thereby improving cache pruning efficiency.

[0106] Figure 2 A cache pruning method flowchart provided in the embodiment of the application Figure 3 As shown in Figure 4 The file information also includes metadata information, the metadata information includes last access timestamps and last modification timestamps corresponding to a plurality of files, and the timestamp verification policy includes a configuration threshold. The specific implementation steps of S204 include:

[0107] S301, metadata information is called according to the file information, to obtain last access timestamps and last modification timestamps corresponding to each file.

[0108] For example, the metadata of a file is obtained through a stat system call, and the following two timestamps are analyzed in detail:

[0109] Optional, st_atime (last access time): records the time when the file was last read or executed.

[0110] Optional, st_mtime (last modification time): records the time when the content of the file was last modified.

[0111] S302, determine whether the last access timestamp or the last modification timestamp corresponding to the first current file is less than the configuration parameter.

[0112] The configuration parameter is client_trim_interval, and the default is 300 seconds.

[0113] S303, if the last access timestamp or the last modification timestamp corresponding to the first current file is less than the configuration parameter, mark the first current file as a hot file and trigger protection.

[0114] S304, if the last access timestamp or the last modification timestamp corresponding to the first current file is greater than or equal to the configuration parameter, the first current file is classified into a non-hot file set.

[0115] S305, then perform the first check on the next file, until all files in the to-be-trimmed cache space complete the first check, and obtain a non-hot file set.

[0116] For example, two timestamps combined with a configuration parameter implement the first check:

[0117] Optionally, if the file is accessed within the last 300 seconds (now-st_atime<300 seconds), it is marked as a hot file.

[0118] Optionally, if the file is modified within the last 300 seconds (now-st_mtime<300 seconds), it is marked as a hot file.

[0119] The first check uses logical OR (||) judgment, and any condition that meets the trigger protection. It can cover the following scenarios:

[0120] Optionally, the file is frequently read but not modified (such as a configuration file).

[0121] Optionally, the file is modified but not read (such as a temporary log).

[0122] Optionally, the file system updates st_atime lazily (such as the relatime option of ext4), and checks st_mtime as a backup.

[0123] It also needs to be explained that the client_trim_interval configuration parameter can be dynamically adjusted through the configuration file to adapt to different business scenarios:

[0124] Optionally, in online transaction processing (OLTP) business, it is recommended to set it to 60 seconds, and high-frequency trading scenarios need to shorten the protection window to reduce the hot file misprotection rate.

[0125] Optionally, in analytical business, it is recommended to set it to 300 seconds, and batch processing scenarios can extend the protection window to reduce the impact of frequent verification on performance.

[0126] Specifically, the client_trim_interval parameter is managed through the configuration file to dynamically adjust the hot file protection time window:

[0127] Among them, the parameter path is: / etc / cache_manager / config.ini.

[0128] Among them, the parameter format is:

[0129]

[0130] Among them, the unit is: second.

[0131] In addition, hot reloading of configuration is also supported, which can take effect without restarting the service:

[0132]

[0133] It needs to be explained that through multi-dimensional timestamp analysis, a hot file immunity mechanism is built to avoid the impact of pruning operations on business performance. To prevent the misdeletion of "hot files" that are being accessed or modified by business processes, resulting in data corruption or business interruption. Through configurable time window and state verification rules, adapt to different business scenarios (such as OLTP, analytical business) for hot file protection needs. A balance mechanism is established between cache space recycling and business performance to ensure that critical business files are not affected by pruning operations.

[0134] It also needs to be explained that the client_trim_interval parameter can be dynamically adjusted through the configuration file to adapt to OLTP (short window) and analytical business (long window). Realize the dynamic matching of hot file protection strategy and business load, reduce manual intervention. Configuration parameter hot reloading mechanism (such as systemctl reload) and scenario-based adaptation rules.

[0135] In the embodiment, based on the simultaneous detection of st_atime (last access time) and st_mtime (last modification time), the range of hot file protection is expanded by logical OR (||) judgment. The single timestamp misjudgment problem (such as file system lazy update st_atime) is solved, the "only access without modification" or "only modification without access" scenarios are covered, and the cache pruning efficiency is improved.

[0136] Figure 3 A cache pruning method flowchart provided for the embodiment of the application Figure 4 As shown in Figure 5 The metadata information further includes state data corresponding to a plurality of files, and the state data includes reference count data and file locks. The specific implementation steps of S205 include:

[0137] S401, retrieve metadata information according to the non-hot file set, and obtain reference count data and file locks corresponding to each file in the non-hot file set.

[0138] Among them, for the file passing the first verification, further check its kernel-level metadata Inode state.

[0139] S402, judge whether the reference count data corresponding to the second current file is greater than zero.

[0140] S403, if the reference count data corresponding to the second current file is greater than zero, the second current file state is abnormal, triggering protection.

[0141] S404, if the reference count data corresponding to the second current file is not greater than zero, the second current file state is normal, and the second current file is classified into the state normal file set.

[0142] S405, judge whether the file lock corresponding to the second current file is released.

[0143] S406, if the file lock corresponding to the second current file is not released, the second current file state is abnormal, triggering protection.

[0144] S407, if the file lock corresponding to the second current file is released, the second current file state is normal, and the second current file is classified into the state normal file set.

[0145] S408, then the next file is checked for the second time, until all files in the non-hot file set are checked for the second time, and the state normal file set is obtained.

[0146] For example, for the file passing the first verification, further check its kernel-level metadata (Inode state) to confirm whether the file is in the following state:

[0147] Optionally, determine whether in open state: determine whether the file is held by a process (e.g. being written) through reference count (i_count).

[0148] Optionally, determine whether in lock state: detect whether the file is exclusively accessed through file lock (e.g. F_WRLCK write lock).

[0149] Among them, if any of the following states is detected, the pruning operation is forcibly skipped:

[0150] Optionally, if the reference count > 0, it means that the file is opened by a process (e.g. the log file is being appended to write).

[0151] Optionally, if the file lock is not released, it means that the file is exclusively locked (e.g. database checkpoint operation).

[0152] It should be noted that the priority of file state verification (second check) is higher than that of timestamp check (first check). Even if the file passes the time check, if the state is abnormal, it is still preferred to protect.

[0153] It should be noted that through file state verification, a hot file immune mechanism is constructed to avoid the impact of pruning operation on business performance. Preventing the deletion of "hot files" being accessed or modified by a business process, resulting in data damage or business interruption. Through the configurable time window and state verification rule, adapt to the needs of hot file protection in different business scenarios (such as OLTP, analytical business). A balance mechanism is established between cache space recycling and business performance to ensure that critical business files are not affected by pruning operations.

[0154] In this embodiment, on the basis of time check, further through inode reference count (i_count) and file lock (such as F_WRLCK) to detect whether the file is held or locked by a process. Prevent pruning interruption or data damage caused by exclusive access to the file (such as database checkpoint). The priority logic of state verification (higher than time check) and specific detection means (such as reference count threshold, lock type judgment), thereby improving the efficiency of cache pruning.

[0155] Figure 4 A cache pruning method flow provided by the embodiment of the application Figure 5 As shown in Figure 6 The file information further includes physical occupation values corresponding to a plurality of files; the pruning strategy includes a preset control amount; and the metadata includes an actual release amount. The specific implementation steps of S206 include:

[0156] S501, arrange all files in the state normal file set in descending order according to the corresponding physical occupation values to obtain a sorted file set.

[0157] S502, pruning the file ranked first in the file set.

[0158] S503, querying the actual release amount of the metadata information, and comparing the actual release amount with the preset control amount.

[0159] S504, if the actual release amount is less than the control amount, updating the control amount, and continuing to prune the file ranked first in the file set.

[0160] S505, if the actual release amount is equal to the preset control amount, ending the pruning.

[0161] S506, if the actual release amount is greater than the preset control amount, ending the pruning and issuing an alarm.

[0162] For example, in descending order of file size, large files occupying space exceeding a threshold (FILE_BLK_SIZE*1024, default 4MB) are processed preferentially.

[0163] Optionally, physical space calculation: the actual physical occupation of the file (real_size = st_blocks*FILE_BLK_SIZE) is calculated through the st_blocks field.

[0164] Optionally, sorting rule: in descending order of real_size, to ensure that large files are preferentially pruned.

[0165] Optionally, trim_size is used as a control parameter, and only a specified size of space is released each time.

[0166] Optionally, parameter passing: the total space to be released (trim_size) is calculated by an upper decision module (such as a double-threshold dynamic decision).

[0167] Optionally, accurate deduction: the remaining amount to be released (trim_size -= real_size) is updated after each pruning, until the target is reached or the traversal is completed.

[0168] Wherein, pruning deficiency caused by sparse files or hollow files (Logical Size>Physical Size) needs to be avoided.

[0169] Wherein, it is necessary to ensure that the released space is an integer multiple of the file system block size (FILE_BLK_SIZE), to reduce fragmentation.

[0170] Wherein, a "pruning-verification-feedback" loop mechanism is adopted:

[0171] Optionally, real-time verification: the actual release amount (through the file system metadata) is checked after each pruning.

[0172] Optionally, dynamic feedback: if the release amount is insufficient, continue to traverse; if excessive, terminate pruning and alarm.

[0173] In this embodiment, by combining the depth-first directory traversal and the large file priority strategy with the file system block-level calculation, the efficiency of space release is maximized. The physical space occupied by large files is preferentially recovered, the fragmentation is reduced, and the storage utilization is improved; by recursive traversal, it is ensured that all directory levels of files are processed, pruning omissions are avoided, and the cache pruning efficiency is improved.

[0174] Figure 5 A cache pruning method flow provided by the embodiment of the present application Figure 6 As shown in Figure 7 The to-be-pruned cache space further includes a mutex, which is used to control all related data of the pruning task. After S203, the above-mentioned process further includes:

[0175] S601, when detecting that the to-be-pruned cache space acquires other tasks, a waiting instruction is triggered, and the pruning task enters an intelligent waiting state through the waiting instruction, and a waiting period is started.

[0176] S602, the target file involved in the other task is released through the mutex, and the other task is executed.

[0177] S603, when the waiting period ends, it is checked whether a stop instruction is received.

[0178] S604, if the stop instruction is received, the pruning task is immediately terminated, and the mutex releases all related data of the pruning task.

[0179] S605, if the stop instruction is not received, the pruning task is automatically resumed.

[0180] Among them, the mutex (Mutex) is used to protect the shared resources, and the exclusive access to the key variables (such as the stopped flag and the trim_size) in the multi-threaded environment is ensured.

[0181] Specifically, automatic locking / release: the lock operation is encapsulated through the resource acquisition initialization (Resource Acquisition Is Initialization, RAII) mode, the lock is automatically locked at the entrance of the scope, and the lock is automatically released at the exit, so as to avoid deadlocks caused by exceptions or omissions.

[0182] Optionally, the stopped flag is used to control the life cycle of the pruning thread.

[0183] Optionally, the trim_size variable is used to record the amount of cache space to be released at present.

[0184] Need to explain, RAII model, even if the code path occurs exception, lock resources can be reliable release; RAII model reduces the explicit locking / unlocking call, reduces the risk of human error.

[0185] Where, when detecting any of the following conditions, pruning process through condition variable (client_cond) into intelligent waiting state.

[0186] Optional, system shutdown, stopped flag is set to true (such as the administrator trigger stop command).

[0187] Optional, resource competition, business process is accessing shared resources (such as disk I / O, file lock).

[0188] Further, the call client_cond.wait_interval into 60 seconds waiting period, during which the automatic release of lock resources.

[0189] Specifically, during the waiting period, other threads or processes can normally get the lock and execute the critical operation, avoid business process due to pruning operation and block.

[0190] Need to explain, 60 seconds timing waiting ensures the business process I / O delay reduction, avoid due to pruning performance jitter, at the same time through the automatic yield mechanism, reduce the thread competition overhead.

[0191] Further, after the waiting period, the system automatically performs the following operations:

[0192] Optional, check stopped flag: if still true, then enter the termination process, release the remaining resources.

[0193] Optional, detect resource competition: if the competition is removed (such as business process complete I / O operation), then automatically restore pruning operation.

[0194] Optional, provide client_admin_wakeup command, allow the administrator to force trigger state check.

[0195] Optional, after waking up to reevaluate the system load (such as CPU / I / O usage) and resource competition state, ensure that pruning operation and business load dynamic balance.

[0196] In this embodiment, through the mutex ensures the atomic access to shared resources (such as system state flag, pruning progress variable), in resource competition or system shutdown, condition variable intelligent waiting mechanism automatically release lock resources, allow business process priority execution, state reinspection and manual wake up interface, ensure that the system load is reduced after the automatic recovery of pruning operation, improve resource utilization and thus improve the cache pruning efficiency.

[0197] Figure 7 A structure diagram of the cache pruning device provided by the embodiment of the present application is shown in the figure, which comprises an initialization module 71, an acquisition module 72, a first processing module 73, a first checking module 74, a second checking module 75 and a second processing module 76. Figure 8

[0198] The initialization module 71 is configured to initialize a cache space to be pruned.

[0199] The acquisition module 72 is configured to acquire a pruning task and acquire space information of the cache space to be pruned according to the pruning task, wherein the space information comprises file information.

[0200] The first processing module 73 is configured to traverse all files in the cache space to be pruned according to the file information by using a depth-first search algorithm.

[0201] The first checking module 74 is configured to perform a first check according to the file information and a timestamp checking strategy to obtain a non-hot file set.

[0202] The second checking module 75 is configured to perform a second check on the non-hot file set according to the file information and a state checking strategy to obtain a state normal file set.

[0203] The second processing module 76 is configured to perform pruning processing on the state normal file set according to the file information and a pruning strategy to obtain a pruned cache space.

[0204] In a possible design, the file information further comprises metadata information, and the metadata information comprises last access timestamps and last modification timestamps corresponding to the plurality of files.

[0205] The timestamp checking strategy comprises a configuration threshold.

[0206] The first check according to the file information and the timestamp checking strategy to obtain the non-hot file set comprises:

[0207] The first checking module 74 is further configured to acquire the last access timestamps and the last modification timestamps corresponding to each file according to the file information.

[0208] It is determined whether the last access timestamp or the last modification timestamp corresponding to the first current file is less than the configuration parameter.

[0209] If the last access timestamp or the last modification timestamp corresponding to the first current file is less than the configuration parameter, the first current file is marked as a hot file, and protection is triggered.

[0210] ​If the last access timestamp or the last modification timestamp corresponding to the first current file is greater than or equal to the configuration parameter, the first current file is classified into the non-hot file set;

[0211] Then, the first time verification is performed on the next file, until the first time verification is completed on all the files in the to-be-pruned cache space, and the non-hot file set is obtained.

[0212] In a possible design, the metadata information further includes state data corresponding to the plurality of files, and the state data includes reference count data and a file lock;

[0213] According to the file information and the state verification strategy, the second time verification is performed on the non-hot file set, and a state normal file set is obtained, including:

[0214] The second verification module 75 is further configured to call the metadata information according to the non-hot file set, and obtain the reference count data and the file lock corresponding to each file in the non-hot file set;

[0215] It is determined whether the reference count data corresponding to the second current file is greater than zero;

[0216] If the reference count data corresponding to the second current file is greater than zero, the second current file is in an abnormal state, and protection is triggered;

[0217] If the reference count data corresponding to the second current file is not greater than zero, the second current file is in a normal state, and the second current file is classified into the state normal file set;

[0218] It is determined whether the file lock corresponding to the second current file is released;

[0219] If the file lock corresponding to the second current file is not released, the second current file is in an abnormal state, and protection is triggered;

[0220] If the file lock corresponding to the second current file is released, the second current file is in a normal state, and the second current file is classified into the state normal file set;

[0221] Then, the second time verification is performed on the next file, until the second time verification is completed on all the files in the non-hot file set, and the state normal file set is obtained.

[0222] In a possible design, the file information further includes a physical occupation value corresponding to the plurality of files;

[0223] The pruning strategy includes a preset control amount;

[0224] The metadata includes an actual release amount;

[0225] According to the file information and the pruning strategy, pruning processing is performed on the state normal file set, including:

[0226] The second processing module 76 is further configured to arrange all the files in the state normal file set in descending order of the corresponding physical occupation values to obtain a sorted file set;

[0227] trimming the file with the first order in the sorted file set;

[0228] querying an actual release amount of the metadata information, comparing the actual release amount with a preset control amount;

[0229] if the actual release amount is less than the control amount, updating the control amount, and continuing to trim the file with the first order in the sorted file set;

[0230] if the actual release amount is equal to the preset control amount, ending the trimming;

[0231] if the actual release amount is greater than the preset control amount, ending the trimming and issuing an alarm.

[0232] In a possible design, the cache space to be trimmed further includes a mutex, which is used to control all the related data of the trimming task;

[0233] after the file information is obtained and all the files in the cache space to be trimmed are traversed by using the depth-first search algorithm, the method further includes:

[0234] when it is detected that the cache space to be trimmed is acquired by other tasks, a waiting instruction is triggered, and the trimming task enters an intelligent waiting state through the waiting instruction and starts a waiting period;

[0235] the target file involved in the other task is released through the mutex, and the other task is executed.

[0236] In a possible design, after the target file involved in the other task is released through the mutex and the other task is executed, the method further includes:

[0237] when the waiting period ends, it is checked whether a stop instruction is received;

[0238] if the stop instruction is received, the trimming task is immediately terminated, and the mutex releases all the related data of the trimming task;

[0239] if the stop instruction is not received, the trimming task is automatically resumed.

[0240] The cache trimming apparatus provided in this embodiment can execute the cache trimming method in the above-described embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0241] In the specific implementation of the cache trimming method, each module can be implemented as a processor, and the processor can execute computer execution instructions stored in a memory, so that the processor executes the cache trimming method.

[0242] Figure 8 A structural schematic diagram of an electronic device apparatus is provided in the embodiments of the present application. As shown in the figure, the electronic device 80 comprises at least one processor 81 and a memory 82. The electronic device 80 further comprises a communication component 83. The processor 81, the memory 82 and the communication component 83 are connected through a second bus 84. ​

[0243] In the specific implementation process, the at least one processor 81 executes the computer execution instructions stored in the memory 82, so that the at least one processor 81 executes a method in the field of cache space management as executed on the electronic device side.

[0244] The specific implementation process of the processor 11 can refer to the above-mentioned method embodiments, which have similar implementation principles and technical effects. Therefore, the specific implementation process of the processor 11 will not be described here again.

[0245] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU for short), and can also be other general-purpose processors, digital signal processors (DSP for short), application-specific integrated circuits (ASIC for short) and the like. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0246] The memory can contain a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory.

[0247] The second bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit only one bus or one type of bus.

[0248] ​The functions implemented by the electronic device and the host device are described above, and the scheme provided by the embodiments of the present application is introduced. It can be understood that the electronic device or the host device includes a hardware structure and / or a software module corresponding to each function to implement the above functions. The units and algorithm steps of each example described in combination with the embodiments disclosed in the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present application.

[0249] The present application also provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, when the processor executes the computer-executable instructions, the computer-executable instructions are used to implement the above-mentioned method in the field of cache space management.

[0250] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0251] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium, and can write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the electronic device or the host device.

[0252] The present application also provides a computer program product, the computer program product includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of the electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the scheme provided by any one of the above-mentioned embodiments.

[0253] Those skilled in the art can understand that all or part of the steps of the foregoing method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the foregoing method embodiments; and the foregoing storage medium includes various storage media that can store program codes, such as ROM, RAM, magnetic disk, or optical disk.

[0254] So far, the technical solutions of the present application have been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments, and the above embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A cache pruning method, characterized in that, include: Initialize the cache space to be pruned; Obtain a trimming task, and obtain the space information of the cache space to be trimmed based on the trimming task; wherein, the space information includes file information; Based on the file information, all files in the cache space to be pruned are traversed using a depth-first search algorithm. The first verification is performed based on the file information and the timestamp verification strategy to obtain a set of non-hot files; Based on the file information and status verification strategy, the set of non-hot files is verified a second time to obtain a set of files with normal status. The set of files in normal condition is pruned according to the file information and pruning strategy to obtain the pruned cache space.

2. The method according to claim 1, characterized in that, The file information also includes metadata information, which includes the last access timestamp and the last modification timestamp corresponding to multiple files; The timestamp verification strategy includes configuring thresholds; The first verification based on the file information and the timestamp verification strategy yields a set of non-hot files, including: Based on the file information, retrieve the metadata information to obtain the last access timestamp and the last modification timestamp corresponding to each file; Determine whether the last access timestamp or the last modification timestamp corresponding to the first current file is less than the configuration parameter; If the last access timestamp or the last modification timestamp corresponding to the first current file is less than the configuration parameter, then the first current file is marked as a hot file, and protection is triggered. If the last access timestamp or the last modification timestamp corresponding to the first current file is greater than or equal to the configuration parameter, then the first current file is included in the non-hot file set. Then, the first verification is performed on the next file until all files in the cache space to be pruned have completed the first verification, thus obtaining the set of non-hot files.

3. The method according to claim 2, characterized in that, The metadata information also includes status data corresponding to multiple files, and the status data includes reference count data and file locks; The second verification of the non-hot file set based on the file information and status verification strategy yields a set of files with a normal status, including: Based on the set of non-hot files, retrieve metadata information to obtain the reference count data and the file lock corresponding to each file in the set of non-hot files; Determine whether the reference count data corresponding to the second current file is greater than zero; If the reference count data corresponding to the second current file is greater than zero, then the state of the second current file is abnormal, triggering protection. If the reference count data corresponding to the second current file is not greater than zero, then the second current file is in a normal state and is included in the set of files in a normal state. Determine whether the file lock corresponding to the second current file has been released; If the file lock corresponding to the second current file is not released, the second current file is in an abnormal state, triggering protection. If the file lock corresponding to the second current file is released, the second current file is in a normal state and is included in the set of files in a normal state. Then, the second verification is performed on the next file until all files in the non-hot file set have completed the second verification, thus obtaining the normal file set.

4. The method according to claim 3, characterized in that, The file information also includes multiple physical occupancy values ​​corresponding to the files; The pruning strategy includes preset control values; The metadata includes the actual amount released; The step of trimming the set of files in normal condition according to the file information and trimming strategy includes: Arrange all the files in the set of normal files in descending order of their corresponding physical occupancy values ​​to obtain a sorted file set; Trim the first file in the sorted file set; Query the actual release amount of the metadata information and compare the actual release amount with the preset control amount; If the actual release amount is less than the control amount, update the control amount and continue to prune the first-ranked file in the sorted file set; If the actual release amount is equal to the preset control amount, the trimming ends; If the actual release amount is greater than the preset control amount, the trimming will end and an alarm will be issued.

5. The method according to any one of claims 1 to 4, characterized in that, The cache space to be pruned also includes a mutex lock, used to control all data related to the pruning task; After traversing all files in the cache space to be pruned using a depth-first search algorithm based on the file information, the process further includes: When the cache space to be pruned is detected to acquire other tasks, a wait instruction is triggered. The pruning task enters a smart wait state through the wait instruction and starts a wait cycle. Release the target files involved in the other tasks using a mutex lock, and then execute the other tasks.

6. The method according to claim 5, characterized in that, After releasing the target file involved in the other tasks via a mutex lock and executing the other tasks, the process further includes: When the waiting period ends, check if a stop command has been received; If a stop command is received, the pruning task is terminated immediately, and the mutex lock releases all data related to the pruning task. If no stop command is received, the pruning task will resume automatically.

7. A cache trimming device, characterized in that, include: The initialization module is used to initialize the cache space to be pruned; The acquisition module is used to acquire a trimming task and acquire the space information of the cache space to be trimmed based on the trimming task; wherein, the space information includes file information; The first processing module is used to traverse all files in the cache space to be pruned using a depth-first search algorithm based on the file information. The first verification module is used to perform the first verification based on the file information and the timestamp verification strategy to obtain a set of non-hot files. The second verification module is used to perform a second verification on the set of non-hot files based on the file information and the status verification strategy to obtain a set of files with normal status. The second processing module is used to perform trimming processing on the set of files in normal state according to the file information and trimming strategy, so as to obtain the trimmed cache space.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images