File processing method, device, equipment, medium and product
By mounting eBPF functions in the kernel and dynamically adjusting the page cache strategy, the performance problems of existing page cache algorithms under diversified loads are solved, and efficient cache resource management and performance optimization are achieved.
Patent Information
- Application Number
- CN202411355308.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2044-09-27
AI Technical Summary
Existing page cache replacement algorithms fail to achieve optimal performance in the face of diversified, dynamically changing workloads, and lack universality and user-mode configuration capabilities, which may lead to cache pollution and premature elimination of important data.
By mounting eBPF functions in the kernel, determining the decision mode based on the type of file processing requests, dynamically adjusting the allocation strategy of the page cache mechanism, and dynamically compute and adjust the threshold and replacement strategy of the page cache using the real-time state and application requirements of the virtual file system. .
It realizes custom cache configuration based on actual scenarios, avoiding low-hot files occupying system resources, and improving system performance and optimized allocation of cache resources.
Smart Images
Figure CN118897654B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a file processing method, device, equipment, medium and product. Background Art
[0002] With the rapid development of computer technology, data storage and processing capabilities have been greatly improved. However, the performance gap between memory (RAM) and disk (HDD / SSD) is still large, which has led to data access speed becoming a performance bottleneck in many application scenarios, such as big data processing, cloud computing, and real-time systems. To alleviate this problem, modern operating systems generally use a page cache mechanism, which uses part of the physical memory to store frequently accessed data to reduce disk I / O operations and thus increase data access speed.
[0003] Although the page cache mechanism has greatly improved system performance, its traditional implementation still has limitations. For example, the commonly used least recently used (LRU) algorithm eliminates pages based on their recent usage, which can achieve good results in many cases, but this simple strategy may not achieve optimal performance when faced with diverse and dynamically changing workloads. In addition, the LRU algorithm does not take into account factors such as the actual access frequency of the page and data locality, which may lead to cache pollution and premature elimination of important data.
[0004] At present, in order to solve the above problems, cache replacement algorithms are generally improved, such as LFU algorithm, ARC algorithm, etc. However, these algorithms usually need to modify the operating system kernel, which not only increases the complexity of implementation, but also may introduce stability and security issues. In addition, these algorithms are often optimized for specific types of application scenarios, lack sufficient versatility, and cannot support user-mode self-configuration. Summary of the invention
[0005] The purpose of the embodiments of the present application is to provide a file processing method, device, equipment, medium and product. The specific technical solutions are as follows:
[0006] In a first aspect of the implementation of the present application, a file processing method is first provided, the method comprising:
[0007] In the case of detecting that a file processing request initiated by a user process is received, triggering a target eBPF function mounted on a kernel function based on the file processing request, determining a target decision mode based on the type of the target eBPF function, and obtaining current file processing data of the virtual file system;
[0008] Based on the target decision mode and the file processing data, it is determined whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request, wherein the page cache processing mechanism processes the to-be-processed file data from a memory.
[0009] Optionally, the target decision mode includes a first decision mode and a second decision mode, the file processing request includes a target decision configuration identifier, and triggering a target eBPF function mounted on a kernel function based on the file processing request and determining the target decision mode based on a type of the target eBPF function includes:
[0010] triggering a target eBPF function mounted on a kernel function based on the file processing request, wherein the target eBPF function includes pre-set user-mode configuration information corresponding to the first decision mode, or historical system performance data corresponding to the second decision mode;
[0011] Determine a target eBPF function based on the target decision configuration identifier carried by the file processing request;
[0012] A target decision mode is determined based on the type of the target eBPF function.
[0013] Optionally, the first decision mode includes a preset strategy mode, and the determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data includes:
[0014] If the target decision mode is the preset strategy mode, obtaining preset user state configuration information based on the preset strategy mode, wherein the preset user state configuration information includes target file processing data for page cache processing corresponding to each first file processing data;
[0015] Based on the current first file processing data of the virtual file system and the target file processing data, it is determined whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request.
[0016] Optionally, the second decision mode includes a self-learning mode, and the determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data includes:
[0017] If the target decision mode is the self-learning mode, determining the second file processing data corresponding to the self-learning mode;
[0018] Based on the current second file processing data of the virtual file system, the preset scoring function corresponding to each second file processing data, and the weight corresponding to each second file processing data, it is determined whether to enable the page cache processing mechanism for the to-be-processed file data corresponding to the file processing request.
[0019] Optionally, the determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the current second file processing data of the virtual file system, a preset scoring function corresponding to each second file processing data, and a weight corresponding to each second file processing data includes:
[0020] generating a target score corresponding to each second file processing data based on the current second file processing data of the virtual file system and a preset scoring function corresponding to each second file processing data;
[0021] Generating a target total score by summing up the target scores corresponding to the respective second file processing data and the weights corresponding to the respective second file processing data;
[0022] Based on the target total score and a preset threshold, it is determined whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request.
[0023] Optionally, the determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target total score and a preset threshold value includes:
[0024] When it is detected that the target total score is greater than a preset threshold, page cache processing is performed on the to-be-processed file data corresponding to the file processing request.
[0025] Optionally, the second file processing data includes file path, file type, file extension, file descriptor, file capacity, file update time, and file source information.
[0026] Optionally, when the second file processing data is a file path, the preset scoring function is a file path scoring function, and the file path scoring function is set based on a matching degree between a file path depth and a target directory.
[0027] Optionally, when the second file processing data is a file type, the preset scoring function is a file type scoring function, and the file type scoring function is set based on the preset priorities and file access frequencies corresponding to each file type.
[0028] Optionally, when the second file processing data is a file extension, the preset scoring function is a file extension scoring function, and the file extension scoring function is set based on the popularity of the file extension and the relevance of the file extension to the current context.
[0029] Optionally, when the second file processing data is file source information, the preset scoring function is a file source information scoring function, and the file source information scoring function is set based on a user role score and a user authority score.
[0030] Optionally, when the second file processing data is file capacity, the preset scoring function is a file capacity function, and the file capacity function is set based on the file capacity and a preset maximum file capacity value.
[0031] Optionally, when the second file processing data is file update time, the preset scoring function is a file update time scoring function, and the file update time scoring function is set based on current time, file download time and a preset decay rate.
[0032] Optionally, after the step of determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data, the method includes:
[0033] In the case where it is detected that a page cache processing mechanism is enabled for the to-be-processed file data corresponding to the file processing request, the to-be-processed file data is read from a memory based on the to-be-processed file data corresponding to the file processing request.
[0034] Optionally, after the step of determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data, the method includes:
[0035] In the case where it is detected that the page cache processing mechanism is closed for the to-be-processed file data corresponding to the file processing request, the to-be-processed file data is read from the target disk based on the to-be-processed file data corresponding to the file processing request.
[0036] Optionally, reading the to-be-processed file data from the target disk based on the to-be-processed file data corresponding to the file processing request includes:
[0037] The to-be-processed file data is read from the target disk in a preset reading manner.
[0038] In a second aspect of the implementation of the present application, a file processing device is also provided, which is applied to the blockchain server system described in the first aspect, and the device includes:
[0039] A determination module, configured to, upon detecting a file processing request initiated by a user process, trigger a target eBPF function mounted on a kernel function based on the file processing request, determine a target decision mode based on a type of the target eBPF function, and obtain current file processing data of the virtual file system;
[0040] A processing module is used to determine whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data, wherein the page cache processing mechanism processes the to-be-processed file data from a memory.
[0041] In a third aspect of the implementation of the present application, there is also provided a communication device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor;
[0042] The processor is used to read the program in the memory to implement the memory resource integrated machine power backup method as described in any one of the first aspects.
[0043] In the fourth aspect of the implementation of the present application, there is also provided a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer implements the memory resource integrated machine power backup method as described in any one of the first aspects.
[0044] In a fifth aspect of the implementation of the present application, a computer program product is also provided, including a computer program / instruction, which, when executed by a processor, implements the memory resource integrated machine power backup method as described in any one of the first aspects.
[0045] The file processing method provided by the embodiment of the present application, by detecting and receiving a file processing request initiated by a user process, triggering a target eBPF function mounted on a kernel function based on the file processing request, and determining a target decision mode based on the type of the target eBPF function, and obtaining the current file processing data of the virtual file system; based on the target decision mode and the file processing data, determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request, wherein the page cache processing mechanism processes the to-be-processed file data from the memory. The embodiment of the present application dynamically adjusts the allocation strategy of the page cache mechanism according to the real-time status and application requirements of the virtual file system. Through the eBPF program loaded into the kernel, the present application can monitor and obtain the current file processing data of the virtual file system in real time. Based on this information, the eBPF program can determine the target decision mode, and based on different target decision modes, it can determine how to process the file processing request initiated by the user process. The eBPF program can dynamically calculate and adjust the threshold of the page cache, the replacement strategy and other parameters to achieve the optimal allocation of cache resources. This application can use the eBPF function to perform dynamic judgment on cache reading before reading the file cache in kernel mode, and supports defining the cache in a customized configuration according to actual scenarios, thus avoiding a large number of low-temperature files occupying system resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0047] Figure 1 A flowchart of a file processing method provided in an embodiment of the present application;
[0048] Figure 2 is a device block diagram of a file processing device provided in an embodiment of the present application;
[0049] Figure 3 It is a schematic diagram of a communication device provided in an embodiment of the present application;
[0050] Figure 4 This is a flow chart of a dynamic page cache mechanism provided by an embodiment of the present application;
[0051] Figure 5 This is a modular schematic diagram of a dynamic page caching mechanism provided in an embodiment of the present application. DETAILED DESCRIPTION
[0052] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in each embodiment of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not constitute any limitation, and the various embodiments can be combined with each other and referenced to each other without contradiction.
[0053] Reference Figure 1 , shows a flowchart of the steps of a file processing method provided by an embodiment of the present application, the method may include:
[0054] Step 101, when detecting and receiving a file processing request initiated by a user process, triggering a target eBPF function mounted on a kernel function based on the file processing request, determining a target decision mode based on the type of the target eBPF function, and obtaining the current file processing data of the virtual file system.
[0055] It should be noted that in the embodiments of the present application, reference can be made to Figure 5 , Figure 5 It is a flow chart of a dynamic page cache mechanism. It can be seen that in user mode, a user process can initiate a file processing request, wherein the file processing request may include a file read request, a file write request, and the like.
[0056] In order to facilitate those skilled in the art to understand the technical solution of the present application, the following is an explanation using file reading as an example.
[0057] First of all, it should be noted that the filter is eBPF (Extended Berkeley Packet Filter), a powerful programming framework designed to run sandboxed programs securely in the Linux kernel without modifying the kernel code. eBPF was originally developed for Linux and is still the most mature and widely used platform in this field. eBPF is a revolutionary technology that can run sandboxed programs in the Linux kernel without changing the kernel source code or loading kernel modules. eBPF is a register-based virtual machine that uses a custom 64-bit RISC instruction set. It can run just-in-time natively compiled BPF programs in the Linux kernel and can access a subset of kernel functions and memory.
[0058] Therefore, in the embodiment of the present application, the overall system performance information can be monitored through the eBPF program code. For example, the eBPF program loaded into the kernel can monitor the usage of the page cache, the system load, and the application access mode in real time.
[0059] In addition, eBPF programs can be regarded as eBPF functions, which can run in the Linux kernel and are used to perform specific tasks or operations. In other words, eBPF programs are essentially a set of functions executed in the kernel, which interact with the kernel through specific hooks to perform various tasks. The security and performance of these programs are guaranteed by the kernel's verifier and JIT compiler.
[0060] Therefore, in an embodiment of the present application, when a file processing request initiated by a user process is detected, a target eBPF function mounted on a kernel function is triggered based on the file processing request.
[0061] Specifically, a user process initiates a file read request. For example, the user process initiates a file read request through a system call (such as read() or pread()), and triggers the file read operation of the operating system kernel based on the file read request. Therefore, when the kernel receives the file read request, it triggers the eBPF function mounted on the kernel function, and then determines the target decision mode based on the type of the target eBPF function, and obtains the current file processing data of the virtual file system.
[0062] Further, the target decision module includes a first decision mode and a second decision mode, the file processing request includes a target decision configuration identifier, the target eBPF function mounted on the kernel function is triggered based on the file processing request, and the target decision mode is determined based on the type of the target eBPF function, including:
[0063] triggering a target eBPF function mounted on a kernel function based on the file processing request, wherein the target eBPF function includes pre-set user-mode configuration information corresponding to the first decision mode, or historical system performance data corresponding to the second decision mode;
[0064] Determine a target eBPF function based on the target decision configuration identifier carried by the file processing request;
[0065] A target decision mode is determined based on the type of the target eBPF function.
[0066] It should be noted that in the embodiments of the present application, reference Figure 5 , it can be seen that Figure 5It includes an eBPF management module, which manages the eBPF mount points according to different modes, and different modes mount different eBPF functions.
[0067] Among them, different modes include a first decision mode and a second decision mode, wherein the first decision mode is a preset strategy mode, and the second decision mode is a self-learning mode. In the preset strategy mode, the eBPF function of the file operation function is mounted, and in the self-learning mode, the eBPF function of the kernel allocation and release of memory page core functions is mounted.
[0068] Further, the first decision mode includes a preset policy mode, and determining whether to enable a page cache processing mechanism for the pending file data corresponding to the file processing request based on the target decision mode and the file processing data includes: if the target decision mode is the preset policy mode, obtaining preset user state configuration information based on the preset policy mode, wherein the preset user state configuration information includes target file processing data for page cache processing corresponding to each first file processing data; and determining whether to enable a page cache processing mechanism for the pending file data corresponding to the file processing request based on the current first file processing data of the virtual file system and the target file processing data.
[0069] Further, the second decision mode includes a self-learning mode, and determining whether to enable a page cache processing mechanism for the pending file data corresponding to the file processing request based on the target decision mode and the file processing data includes: if the target decision mode is the self-learning mode, determining the second file processing data corresponding to the self-learning mode; and determining whether to enable a page cache processing mechanism for the pending file data corresponding to the file processing request based on the current second file processing data of the virtual file system, a preset scoring function corresponding to each second file processing data, and a weight corresponding to each second file processing data.
[0070] It should be noted that, in the embodiment of the present application, the eBPF function map stores the policies read by the policy module or the collected system performance data, and different eBPF functions may be included for different target decision modes.
[0071] Among them, the policy read by the policy module is used in the preset policy mode. The policy content supports configuring whether to read PageCache for system file path, file type, file extension, specific file descriptor, file size threshold, file update time, file owner, etc.
[0072] The collected system performance data, that is, the historical system performance data is used in the self-learning mode, including the current memory usage of the system, the recent read frequency and reference count of the currently read file.
[0073] Step 102, based on the target decision mode and the file processing data, determine whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request, wherein the page cache processing mechanism processes the to-be-processed file data from a memory.
[0074] It should be noted that, when the target decision mode has been determined and the file processing data has been obtained in step 101, whether to enable the page cache processing mechanism for the to-be-processed file data corresponding to the file processing request can be determined based on different target decision modes.
[0075] Specifically, based on the above strategies and data, the eBPF function makes a decision on whether to read the page cache mechanism.
[0076] It should be noted that in the embodiment of the present application, the page cache mechanism stores data directly in the memory without reading from the disk, thereby reducing frequent access to the disk and improving the system read and write performance.
[0077] Specifically, the disk is a long-term storage medium responsible for data persistence and eventual consistency; the memory is a cache responsible for improving the performance of file reading and writing. The page cache mechanism reduces the frequent access of the disk by caching disk data in memory, thereby significantly improving the overall performance of the system.
[0078] In addition, in the embodiment of the present application, the eBPF function is mounted on the VFS (virtual file system) to dynamically determine whether to enable the page cache mechanism for the file processing request. For details, please refer to Figure 4 ,It can be seen that there is a rule-based dynamic cache judgment mounted on the VFS, where the Virtual File System (VFS) is an abstract layer in the operating system kernel, which provides a unified interface for different file systems. VFS allows applications to access different types of file systems in a consistent way without having to care about the specific implementation of the underlying file system.
[0079] Further, the determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the current second file processing data of the virtual file system, a preset scoring function corresponding to each second file processing data, and a weight corresponding to each second file processing data includes:
[0080] generating a target score corresponding to each second file processing data based on the current second file processing data of the virtual file system and a preset scoring function corresponding to each second file processing data;
[0081] Generating a target total score by summing up the target scores corresponding to the respective second file processing data and the weights corresponding to the respective second file processing data;
[0082] Based on the target total score and a preset threshold, it is determined whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request.
[0083] Furthermore, before elaborating on the processing of file processing requests under the above-mentioned second decision-making mode, it needs to be confirmed that in the second decision-making mode, that is, in the self-learning mode, the key factors affecting the decision include: file path (FilePath), that is, the path of the system file; file type (FileType), that is, the type of file, such as text, picture, video, etc.; file extension (FileExt), that is, the extension of the file, such as .txt, .jpg, .mp4, etc.; specific file descriptor (FileDescriptor), that is, the identifier of certain specific files; file size threshold (FileSizeThreshold), that is, the threshold of file size; file update time (FileUpdateTime), that is, the time when the file was last updated; file owner (FileOwner), that is, the owner of the file. Therefore, it can be confirmed that the second file processing data that needs to be obtained includes file path, file type, file extension, file descriptor, file capacity, file update time, and file source information.
[0084] Therefore, firstly, a weight is defined for each of the second file processing data, and these weights can be adjusted according to actual conditions.
[0085] Specifically, it may include: WFP: file path weight; WFT: file type weight; WFE: file extension weight; WFD: specific file descriptor weight; WFS: file size weight; WFU: file update time weight; WFO: file owner weight.
[0086] Furthermore, for each factor, a scoring function needs to be defined. For example, the scoring range can be set to 0 to 1. The final score is the sum of the scores of all factors multiplied by the corresponding weights. Whether the total score is greater than the threshold is used to decide whether to read the file based on the page cache mechanism.
[0087] Furthermore, the determining whether to enable a page cache processing mechanism for the pending file data corresponding to the file processing request based on the target total score and a preset threshold includes: when it is detected that the target total score is greater than the preset threshold, performing page cache processing on the pending file data corresponding to the file processing request.
[0088] Further, when the second file processing data is a file path, the preset scoring function is a file path scoring function, and the file path scoring function is set based on the matching degree of the file path depth and the target directory.
[0089] Further, when the second file processing data is a file type, the preset scoring function is a file type scoring function, and the file type scoring function is set based on the preset priorities and file access frequencies corresponding to each file type.
[0090] Further, when the second file processing data is a file extension, the preset scoring function is a file extension scoring function, and the file extension scoring function is set based on the popularity of the file extension and the relevance of the file extension to the current context.
[0091] Further, when the second file processing data is file source information, the preset scoring function is a file source information scoring function, and the file source information scoring function is set based on a user role score and a user authority score.
[0092] Further, when the second file processing data is file capacity, the preset scoring function is a file capacity function, and the file capacity function is set based on the file capacity and a preset maximum file capacity value.
[0093] Further, when the second file processing data is file update time, the preset scoring function is a file update time scoring function, and the file update time scoring function is set based on the current time, the file download time and a preset decay rate.
[0094] Specifically, referring to the above description, it can be seen that the scoring functions can include the following:
[0095] First, the file path scoring function (S_FP)
[0096] Considering that a file path may contain multiple subdirectories, we can design a scoring function using the path depth and the degree of matching of a specific directory:
[0097] (Formula 1)
[0098] In formula 1, depth(FilePath) represents the depth of the path, and matchScore is a function used to calculate the file path. With important directories degree of match.
[0099] Second, the file type scoring function (S_FT)
[0100] File types have different priorities, and the scoring function can be designed based on the purpose and access frequency of the file:
[0101] (Formula 2)
[0102] In the above formula 2, freq(FileType_i) represents the access frequency of file type i, and importance(FileType_i) represents the importance of file type i, and specifically, it may be the preset priority corresponding to each file type.
[0103] Third, the file extension scoring function (S_FE)
[0104] File extensions can be scored based on the popularity and relevance of the extension:
[0105] (Formula 3)
[0106] Among them, popularity(FileExt) represents the popularity of the file extension, and relevance(FileExt,currentContext) represents the file extension With the current context of relevance.
[0107] Fourth, specific file descriptor scoring function (S_FD)
[0108] It is possible to set the score for a specific file descriptor.
[0109] Fifth, file capacity scoring function (S_FS)
[0110] The file capacity is the file size, and its score can be set by setting a nonlinear scoring function to distinguish the impact of files of different sizes on pageCache. Specifically, the following formula 4 can be referred to, wherein, is the file capacity, The maximum file capacity.
[0111] (Formula 4)
[0112] Sixth, file update time scoring function (S_FU)
[0113] The update time of a file can use an exponential decay model to reflect the freshness of the file.
[0114] (Formula 5)
[0115] in, is a preset attenuation rate, which can be adjusted according to the needs of the system. is the current time, The time the file was downloaded.
[0116] Seventh, file source information scoring function
[0117] The file source information, that is, the file owner, therefore, its score can be based on the user's role and permissions. Specifically, it can be constructed with reference to the following formula 6.
[0118] (Formula 6)
[0119] Among them, roleScore and permissionScore are the scoring functions of user roles and permissions respectively.
[0120] Optionally, after the step of determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data, the method includes:
[0121] If so, the to-be-processed file data is read from the page cache (memory) based on the to-be-processed file data corresponding to the file processing request.
[0122] Optionally, after the step of determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data, the method includes:
[0123] If not, the to-be-processed file data is read from the target disk based on the to-be-processed file data corresponding to the file processing request.
[0124] Optionally, reading the to-be-processed file data from the target disk based on the to-be-processed file data corresponding to the file processing request includes:
[0125] The to-be-processed file data is read from the target disk in a preset reading manner.
[0126] It should be noted that in the embodiment of the present application, after determining whether to process the file processing request through the page cache mechanism, the kernel processes the file read request.
[0127] Specifically, based on the decision of the dynamic cache eBPF function, the read scheduling module triggers the actual read operation. When the PageCache decision is yes, the data of PageCache is read first according to the original read process of the system. When the PageCache decision is no, the O_DIRECT flag is added to the flags parameter of open(), and the Direct I / O method is used to bypass PageCache and directly read the data from the disk.
[0128] The file processing method provided by the embodiment of the present application, by detecting and receiving a file processing request initiated by a user process, triggering a target eBPF function mounted on a kernel function based on the file processing request, and determining a target decision mode based on the type of the target eBPF function, and obtaining the current file processing data of the virtual file system; based on the target decision mode and the file processing data, determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request, wherein the page cache processing mechanism processes the to-be-processed file data from the memory. The embodiment of the present application dynamically adjusts the allocation strategy of the page cache mechanism according to the real-time status and application requirements of the virtual file system. Through the eBPF program loaded into the kernel, the present application can monitor and obtain the current file processing data of the virtual file system in real time. Based on this information, the eBPF program can determine the target decision mode, and based on different target decision modes, it can determine how to process the file processing request initiated by the user process. The eBPF program can dynamically calculate and adjust the threshold of the page cache, the replacement strategy and other parameters to achieve the optimal allocation of cache resources. This application can use the eBPF function to perform dynamic judgment on cache reading before reading the file cache in kernel mode, and supports defining the cache in a customized configuration according to actual scenarios, thus avoiding a large number of low-temperature files occupying system resources.
[0129] In another embodiment, a file processing system, specifically, referring to Figure 5 The system may include an eBPF management module, a configuration parsing module, a policy management module in user mode, a memory monitoring module and a file monitoring module based on the filter eBPF function in kernel mode, and a policy matching and reading scheduling module in kernel mode. Specifically, each module may refer to the following content.
[0130] Configuration parsing module: used in preset policy mode, responsible for loading user-state configuration. The configuration content supports configuring whether to read PageCache for system file path, file type, file extension, specific file descriptor, file size threshold, file update time, file owner, etc.
[0131] Policy management module: responsible for processing the update actions of user-mode configuration and notifying the kernel-mode eBPF function in real time based on the configuration refresh.
[0132] Filter management module: manages eBPF mount points according to different modes (preset policy mode or self-learning mode). Different modes mount different filter eBPF functions. In the preset policy mode, eBPF functions of file operation functions are mounted. In the self-learning mode, eBPF functions of kernel allocation and release of memory page core functions are mounted.
[0133] Memory monitoring module: By monitoring the kernel allocation and release of memory page functions, the system's memory usage is obtained in real time, which serves as an important reference for the system's current performance status and the system's operating status. Based on this information, the system's resource requirements can be more accurately evaluated.
[0134] File monitoring module: counts the read frequency of a specific file, records the popularity of the file and its access pattern. By real-time monitoring and analyzing this data, combined with the current performance status of the system, reasonable decisions can be made dynamically.
[0135] Strategy matching: Based on pre-loaded configuration files or monitoring data collected through self-learning mode, the system can intelligently identify the current environment and needs. By analyzing and comparing these data, the system can match the most suitable decision results to achieve optimal configuration and efficient use of resources.
[0136] Read scheduling: Responsible for intelligently choosing to read data from the cache or I / O device based on the decision result. When the PageCache decision is yes, the data in the PageCache is read first according to the original reading process of the system. When the PageCache decision is no, the O_DIRECT flag is added to the flags parameter of open(), and the Direct I / O method is used to bypass PageCache and directly read the data from the disk.
[0137] Reference Figure 2 , Figure 2 : is a device block diagram of a file processing device provided in an embodiment of the present application, the device comprising:
[0138] The determination module 201 is used to trigger a target eBPF function mounted on a kernel function based on the file processing request when a file processing request initiated by a user process is detected, determine a target decision mode based on the type of the target eBPF function, and obtain current file processing data of the virtual file system;
[0139] The processing module 202 is used to determine whether to enable a page cache processing mechanism for the pending file data corresponding to the file processing request based on the target decision mode and the file processing data, wherein the page cache processing mechanism processes the pending file data from the memory.
[0140] The present application also provides a communication device, such as Figure 3 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.
[0141] Memory 703, used for storing computer programs;
[0142] The processor 701, when used to execute the program stored in the memory 703, can implement the following steps:
[0143] In the case of detecting that a file processing request initiated by a user process is received, triggering a target eBPF function mounted on a kernel function based on the file processing request, determining a target decision mode based on the type of the target eBPF function, and obtaining current file processing data of the virtual file system;
[0144] Based on the target decision mode and the file processing data, it is determined whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request, wherein the page cache processing mechanism processes the to-be-processed file data from a memory.
[0145] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together, which are all well known in the art, so this article will not further describe them. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor can be transmitted on a wired medium or on a wireless medium through an antenna. Further, the antenna also receives data and transmits the data to the processor. The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor when performing operations.
[0146] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0147] The communication interface is used for communication between the above terminal and other devices.
[0148] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0149] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0150] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes the memory resource integrated machine power backup method described in any of the above embodiments.
[0151] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the memory resource integrated machine power backup method described in any of the above embodiments.
[0152] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0153] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0154] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0155] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A file processing method, characterized in that: The method comprises: In the case of detecting that a file processing request initiated by a user process is received, triggering a target eBPF function mounted on a kernel function based on the file processing request, determining a target decision mode based on the type of the target eBPF function, and obtaining current file processing data of the virtual file system, wherein the target decision mode includes a first decision mode and a second decision mode, and the file processing request includes a target decision configuration identifier; Determine whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data, wherein the page cache processing mechanism processes the to-be-processed file data from a memory; The triggering of the target eBPF function mounted on the kernel function based on the file processing request and determining the target decision mode based on the type of the target eBPF function includes: triggering a target eBPF function mounted on a kernel function based on the file processing request, wherein the target eBPF function includes pre-set user-mode configuration information corresponding to the first decision mode, or historical system performance data corresponding to the second decision mode; Determine a target eBPF function based on the target decision configuration identifier carried by the file processing request; A target decision mode is determined based on the type of the target eBPF function.
2. The method according to claim 1, characterized in that The first decision mode includes a preset strategy mode, and the determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data includes: If the target decision mode is the preset strategy mode, obtaining preset user state configuration information based on the preset strategy mode, wherein the preset user state configuration information includes target file processing data for page cache processing corresponding to each first file processing data; Based on the current first file processing data of the virtual file system and the target file processing data, it is determined whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request.
3. The method according to claim 1, characterized in that The second decision mode includes a self-learning mode, and the determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data includes: If the target decision mode is the self-learning mode, determining the second file processing data corresponding to the self-learning mode; Based on the current second file processing data of the virtual file system, the preset scoring function corresponding to each second file processing data, and the weight corresponding to each second file processing data, it is determined whether to enable the page cache processing mechanism for the to-be-processed file data corresponding to the file processing request.
4. The method according to claim 3, characterized in that The determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the current second file processing data of the virtual file system, a preset scoring function corresponding to each second file processing data, and a weight corresponding to each second file processing data includes: generating a target score corresponding to each second file processing data based on the current second file processing data of the virtual file system and a preset scoring function corresponding to each second file processing data; Generating a target total score by summing up the target scores corresponding to the respective second file processing data and the weights corresponding to the respective second file processing data; Based on the target total score and a preset threshold, it is determined whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request.
5. The method according to claim 4, characterized in that The determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target total score and a preset threshold value includes: When it is detected that the target total score is greater than a preset threshold, page cache processing is performed on the to-be-processed file data corresponding to the file processing request.
6. The method according to claim 4 or 5, characterized in that: The second file processing data includes file path, file type, file extension, file descriptor, file capacity, file update time, and file source information.
7. The method according to claim 6, characterized in that When the second file processing data is a file path, the preset scoring function is a file path scoring function, and the file path scoring function is set based on the matching degree of the file path depth and the target directory.
8. The method according to claim 6, characterized in that When the second file processing data is a file type, the preset scoring function is a file type scoring function, and the file type scoring function is set based on the preset priorities and file access frequencies corresponding to each file type.
9. The method according to claim 6, characterized in that When the second file processing data is a file extension, the preset scoring function is a file extension scoring function, and the file extension scoring function is set based on the popularity of the file extension and the relevance of the file extension to the current context.
10. The method according to claim 6, characterized in that When the second file processing data is file source information, the preset scoring function is a file source information scoring function, and the file source information scoring function is set based on a user role score and a user authority score.
11. The method according to claim 6, characterized in that When the second file processing data is file capacity, the preset scoring function is a file capacity function, and the file capacity function is set based on the file capacity and a preset maximum file capacity value.
12. The method according to claim 6, characterized in that When the second file processing data is file update time, the preset scoring function is a file update time scoring function, and the file update time scoring function is set based on the current time, the file download time and a preset decay rate.
13. The method according to claim 1, characterized in that After the step of determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data, the method includes: In the case where it is detected that a page cache processing mechanism is enabled for the to-be-processed file data corresponding to the file processing request, the to-be-processed file data is read from a memory based on the to-be-processed file data corresponding to the file processing request.
14. The method according to claim 1, characterized in that After the step of determining whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data, the method includes: In the case where it is detected that the page cache processing mechanism is closed for the to-be-processed file data corresponding to the file processing request, the to-be-processed file data is read from the target disk based on the to-be-processed file data corresponding to the file processing request.
15. The method according to claim 14, characterized in that The reading the to-be-processed file data from the target disk based on the to-be-processed file data corresponding to the file processing request includes: The to-be-processed file data is read from the target disk in a preset reading manner.
16. A file processing device, characterized in that: The device comprises: A determination module, configured to, upon detecting that a file processing request initiated by a user process is received, trigger a target eBPF function mounted on a kernel function based on the file processing request, determine a target decision mode based on the type of the target eBPF function, and obtain current file processing data of the virtual file system, wherein the target decision mode includes a first decision mode and a second decision mode, and the file processing request includes a target decision configuration identifier; a processing module, configured to determine whether to enable a page cache processing mechanism for the to-be-processed file data corresponding to the file processing request based on the target decision mode and the file processing data, wherein the page cache processing mechanism processes the to-be-processed file data from a memory; The determination module is specifically used for: triggering a target eBPF function mounted on a kernel function based on the file processing request, wherein the target eBPF function includes pre-set user-mode configuration information corresponding to the first decision mode, or historical system performance data corresponding to the second decision mode; Determine a target eBPF function based on the target decision configuration identifier carried by the file processing request; A target decision mode is determined based on the type of the target eBPF function.
17. A communication device, characterized in that: include: A transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; The processor is used to read the program in the memory to implement the file processing method as described in any one of claims 1-15.
18. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the file processing method as described in any one of claims 1 to 15 is implemented.
19. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the file processing method as described in any one of claims 1 to 15 is implemented.