Data storage and cache optimization method and system based on Alluxio

Abstract

The invention discloses a data storage and cache optimization method and system based on Alluxio, and the method comprises the steps: obtaining a to-be-written file, and calculating the priority of the file according to the four attributes of the to-be-written file; the four attributes of the file to be written comprise file access frequency, file capacity, file importance level and file freshness; according to a file transmission strategy corresponding to the file priority, storing the file to be written into an Alluxio cache and a bottom storage; obtaining data to be read, judging whether the data to be read is in the cache or not, and if yes, returning the data in the cache to the user; if not, returning the data in the bottom layer storage to the user; and according to the association rule, performing prefetching operation on the to-be-read data and the data blocks associated with the to-be-read data, and if the cache utilization rate exceeds a set threshold value, executing cache replacement operation.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage and cache optimization, and in particular to a data storage and cache optimization method and system based on Alluxio. Background Art

[0002] In the current architecture of big data and cloud computing, data centers are usually distributed in multiple geographic locations. However, data access efficiency in a multi-data center environment is often affected by geographic distribution, network latency, and bandwidth limitations between data centers. Especially in cross-data center scenarios, due to the long physical distance between different data centers, network latency becomes an important bottleneck affecting system I / O performance. The synchronization and data transmission speed between data centers is far slower than that within the same data center. This latency and bandwidth limitation significantly reduces the data access speed, which in turn affects the performance of the overall system. Summary of the invention

[0003] In order to solve the deficiencies of the prior art, the present invention provides a data storage and cache optimization method and system based on Alluxio;

[0004] On the one hand, it provides data storage and cache optimization methods based on Alluxio, including:

[0005] Obtain a file to be written, and calculate the priority of the file according to four attributes of the file to be written; the four attributes of the file to be written include: file access frequency, file capacity, file importance level, and file freshness;

[0006] Store the files to be written into the Alluxio cache and underlying storage according to the file transfer policy corresponding to the file priority.

[0007] Get the data to be read and determine whether the data to be read is in the cache. If yes, return the data in the cache to the user; if not, return the data in the underlying storage to the user;

[0008] According to the association rule, a pre-fetch operation is performed on the data to be read and the data blocks associated with the data to be read. If the cache usage rate exceeds a set threshold, a cache replacement operation is performed.

[0009] On the other hand, it provides a data storage and cache optimization system based on Alluxio, including:

[0010] The acquisition module is configured to: acquire the file to be written, and calculate the priority of the file according to four attributes of the file to be written; the four attributes of the file to be written include: file access frequency, file capacity, file importance level and file freshness;

[0011] The write module is configured to store the files to be written into the Alluxio cache and underlying storage according to the file transfer policy corresponding to the file priority;

[0012] A reading module is configured to: obtain the data to be read, determine whether the data to be read is in the cache, and if so, return the data in the cache to the user; if not, return the data in the underlying storage to the user;

[0013] The pre-fetch module is configured to: perform pre-fetch operations on the data to be read and the data blocks associated with the data to be read according to the association rules, and perform a cache replacement operation if the cache usage rate exceeds a set threshold.

[0014] The above technical solution has the following advantages or beneficial effects:

[0015] The present invention introduces a file priority allocation mechanism, which can dynamically adjust data transmission and storage strategies according to indicators such as file importance, access frequency, size, and freshness, ensuring that high-priority files receive higher resource priority during asynchronous storage. The combined use of synchronous writing and asynchronous writing not only improves data writing efficiency, but also effectively reduces the risk of data loss caused by failures before asynchronous writing to the underlying storage system, thereby ensuring high availability and persistence of data.

[0016] The present invention adopts an intelligent prefetch and replacement strategy based on the correlation between data blocks, which can actively prefetch data blocks associated with the currently accessed data during the data access process, reduce the frequency of remote calls in cross-data center scenarios, and reduce data access latency. Especially when the system load is low, the intelligent prefetch mechanism can make full use of idle resources, improve the cache hit rate and the system's I / O performance, and further enhance the availability of Alluxio in a multi-data center environment.

[0017] By designing a dynamic load balancing mechanism, the present invention can monitor the resource usage of each node in real time, and dynamically adjust task allocation and weight according to the load status, avoiding resource overload while maximizing the use of system resources. This optimization strategy ensures that the system can still maintain efficient operation in high-concurrency scenarios, and reduces performance bottlenecks caused by load imbalance, thereby improving the overall performance and response speed of the system.

[0018] The present invention will comprehensively consider the file access frequency, size, importance and freshness to realize the allocation of file priority, formulate different transmission strategies for files of different priorities, reduce the risk of data loss, and improve the overall performance of the system.

[0019] By considering the access frequency and timing correlation between data blocks, combined with the network conditions and system load conditions in cross-data center scenarios, related data blocks can be intelligently pre-fetched to reduce the frequency of remote calls and lower data access latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0021] Figure 1 is a flow chart of the method of embodiment 1;

[0022] Figure 2 This is a data transmission strategy diagram of embodiment 1;

[0023] Figure 3 This is a dynamic load balancing flow chart of embodiment 1;

[0024] Figure 4 This is the parallel mining of association rules of embodiment 1;

[0025] Figure 5 This is a flow chart of intelligent cache replacement according to Embodiment 1. DETAILED DESCRIPTION

[0026] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0027] As an open source distributed file system, Alluxio can provide an efficient data access layer to alleviate these problems. It caches hot data in the local data center and uses asynchronous write mechanisms to reduce the frequency of data transmission across data centers. It supports multiple ways to write data, which can be specified when writing data. Alluxio supports four ways to write data, namely, data is only kept in Alluxio (MUST_CACHE), data is written to Alluxio and the underlying persistent storage at the same time (CACHE_THROUGH), data is asynchronously written to the underlying storage (ASYNC_THROUGH), and data is only written to the underlying storage (THROUGH). Although the synchronous method can ensure data persistence, the speed of writing to the underlying storage is usually much slower than writing to local storage, so the synchronous write speed will be equivalent to the speed of writing to the underlying storage. Asynchronous storage is a way to separate the data write operation from the data persistence operation, so that data can be written to a high-speed cache layer first, and then asynchronously written to a low-speed persistent layer in the background, thereby improving data write performance and efficiency. This method can provide data writes at a memory speed close to MUST_CACHE and can complete data persistence.

[0028] In order to persist data while writing data at memory speed, the asynchronous storage strategy provided by Alluxio is proposed to write data synchronously to the worker and asynchronously to the underlying storage system. However, if a failure occurs before the data is asynchronously written to the underlying storage system, data loss may occur. And because data transmission and storage operations are performed in the background, they may compete with foreground applications for system resources (such as CPU, memory, and bandwidth), affecting overall performance. Therefore, an asynchronous storage optimization strategy is studied to reduce the risk of data loss.

[0029] In the era of big data and cloud computing, the efficiency of data processing and storage has a crucial impact on the overall performance of the system. As an open source distributed file system, Alluxio aims to provide an efficient data access layer between the computing framework and the storage system, thereby improving the performance of data-intensive applications. However, in cross-domain scenarios, network latency becomes an important bottleneck affecting the system I / O speed, especially when the computing speed is significantly improved. This mismatch is more obvious. The bottleneck of I / O speed not only limits the efficiency of data processing, but also directly affects the quality of external services. Using pre-fetching and replacement technologies to keep hot data in Alluxio persistently avoids frequent remote calls, reduces the number of accesses to the underlying storage, and reduces data access latency. Therefore, a cache pre-fetching and replacement strategy based on Alluxio in cross-domain scenarios is studied to improve the availability and performance of Alluxio.

[0030] Embodiment 1

[0031] This embodiment provides a data storage and cache optimization method based on Alluxio;

[0032] like Figure 1 As shown in the figure, the data storage and cache optimization method based on Alluxio includes:

[0033] S101: Obtain a file to be written, and calculate the priority of the file according to four attributes of the file to be written; the four attributes of the file to be written include: file access frequency, file capacity, file importance level, and file freshness;

[0034] S102: Store the file to be written into the Alluxio cache and underlying storage according to the file transfer policy corresponding to the file priority.

[0035] S103: Obtain the data to be read, and determine whether the data to be read is in the cache. If yes, return the data in the cache to the user; if no, return the data in the underlying storage to the user;

[0036] S104: Pre-fetching the data to be read and the data blocks associated with the data to be read according to the association rule, and performing a cache replacement operation if the cache usage rate exceeds a set threshold.

[0037] It should be understood that S101: Obtain the file to be written, and the data packet of each file contains detailed information to ensure effective management and tracking of data transmission. Each file data packet includes: file ID, priority, source node and target node, data block information (data block size and storage location), transmission status, checksum, copy information and timestamp.

[0038] File ID is a unique identifier of a file, which is used to distinguish different files. File priorities are divided into three levels: high, medium, and low. The priority level determines the different transmission strategies of the file. High-priority files need to be transmitted immediately and ensure data persistence, medium-priority files allow a certain delay, and low-priority files can be transmitted when resources are idle. The transmission progress records the current transmission status of the data packet, such as "waiting for transmission", "transmitting", "transmission completed", "transmission failed", etc. The transmission status is updated in real time to monitor the transmission progress and handle exceptions in time. Each data block calculates the checksum before transmission and verifies it after the transmission is completed to ensure that the data has not been tampered with or damaged during the transmission process. For high-priority data, the number and storage location of the data block replicas are recorded to quickly restore the data in the event of a node failure. The replica information ensures high data availability and reliability. The timestamp records the timestamp information such as the creation time, transmission start time, and transmission completion time of each data packet.

[0039] Furthermore, the file access frequency refers to:

[0040]

[0041] Among them, F i represents the access frequency of file i, n represents the number of time periods, and w j represents the weight of the jth time period, A j Indicates the number of times file i is accessed in the jth time period.

[0042] For example, divide the time into days and record the number of file accesses every day. Assign a weight w to each time period j The weight of the more recent time period is larger, and the weight of the more distant time period is smaller.

[0043] It should be understood that the access frequency of a file is measured by real-time monitoring and tracking the usage of each file, taking into account time factors, access times and weights. The higher the access frequency, the higher the priority.

[0044] Furthermore, the file capacity is obtained from the metadata of the file. A file with a large file capacity will require more transmission and storage resources.

[0045] Furthermore, the importance level of the file is determined by user selection. When uploading a file, the user can determine the importance of the file by selecting a level from 1 to 10. The larger the number, the more important the file is and the higher the priority.

[0046] Furthermore, the file freshness is determined according to the last modification time of the file. The file freshness is obtained by subtracting the last modification time from the current time. The smaller the value obtained, the fresher the data.

[0047] Further, the calculating the priority of the file according to the four attributes of the file to be written includes:

[0048] Assume that for file i, the original indicators are: access frequency F i 、File size S i 、The Importance of Documents i and the freshness of the data N i , and standardize each indicator:

[0049]

[0050] Combine the standardized indicators and calculate the priority P of each file according to the weight ratio. i for:

[0051] P i =aF i+bS i +cI i +dN i ,

[0052] Among them, a, b, c and d are weights, a+b+c+d=1.

[0053] Furthermore, if Figure 2 As shown, the S102: storing the file to be written in the Alluxio cache and the underlying storage according to the file transfer policy corresponding to the file priority includes:

[0054] If the file priority is high, the synchronous write mechanism and the dual copy mechanism are used;

[0055] The synchronous writing mechanism means that the files to be written are written to the Alluxio cache and the underlying storage at the same time.

[0056] The dual copy mechanism refers to copying each data block to another working node;

[0057] After the data is written to the underlying storage, data verification begins. The data verification is used to determine whether the written file is damaged. If the verification succeeds, it means that the file is written; if the verification fails, it means that the file needs to be rewritten.

[0058] It should be understood that Alluxio cache refers to a distributed memory storage layer. Alluxio can cache hot data (i.e. frequently accessed data) in memory, making data reading and processing more efficient. Through Alluxio cache, applications can quickly obtain data from memory without directly accessing the underlying storage, greatly reducing I / O latency and improving data processing performance.

[0059] It should be understood that the underlying storage refers to the persistent storage system or device that actually stores the data. It is responsible for long-term data preservation and ensures the security, persistence, and reliability of the data. The underlying storage usually includes various forms of storage systems, such as distributed file systems (such as HDFS), object storage (such as Amazon S3, Azure Blob Storage), local hard disks, or network storage devices. The underlying storage is more stable and persistent than the cache, and the data will not disappear even if the system is restarted or the memory is lost. Alluxio will eventually write the file or data to the underlying storage after processing to ensure that the data can be stored for a long time and accessed later.

[0060] It should be understood that because the priority of a file reflects the importance and urgency of the file, a synchronous write mechanism is considered for high-priority files. When writing to the Alluxio cache, it is immediately written to the underlying storage. To ensure high availability and reliability of the data, a dual copy mechanism is maintained before the write is completed, that is, each data block is copied to another worker node. After the data is written to the underlying storage, the system automatically triggers the data verification process. The data block is verified using a checksum (such as MD5 or SHA-256) to ensure that no data corruption or tampering occurs during the transmission process. Once the verification is successful, the system generates a confirmation message and updates the file status to "Write Completed". If the verification fails, the data retransmission mechanism is triggered and the write operation is performed again until the data verification is successful.

[0061] Furthermore, the S102: storing the file to be written into the Alluxio cache and the underlying storage according to the file transfer policy corresponding to the file priority, further includes:

[0062] If the file priority is medium, the asynchronous writing mechanism and load balancing mechanism are used;

[0063] The asynchronous writing mechanism means that the file to be written is first written to the Alluxio cache, and then the file to be written is written to the underlying storage; during the writing process, the intermediate state of the saved data is periodically checked through the checkpoint mechanism.

[0064] It should be understood that the checkpoint mechanism is a mechanism that ensures that data processing can be recovered even if a failure occurs. It saves the current data processing state as a "checkpoint" periodically during the task processing process, so that when a failure occurs, the system can restart processing from the most recent checkpoint instead of starting from the beginning, thereby reducing data recovery time and resource consumption.

[0065] Furthermore, the step of periodically checking the intermediate state of the saved data through the checkpoint mechanism specifically includes:

[0066] (11) Set the checkpoint interval: Configure a set time interval or data volume (such as every minute or every time a certain size of data is processed) as the frequency for creating checkpoints;

[0067] This ensures that checkpoints are not created too often to affect system performance, but are frequent enough to effectively save intermediate states;

[0068] (12) Save checkpoint: When the set time interval or data volume is reached, the progress of the current file writing operation is saved as a checkpoint;

[0069] The progress includes: data offset, write status, file metadata and system environment information; data offset is the current location where the file is written; write status is the completed data block or fragment; file metadata is the location information of the file in storage; system environment information is the node status of the current processing, resource usage, etc.

[0070] (13)Store checkpoint information;

[0071] By storing checkpoints, the system can accurately locate the progress of writes during failure recovery.

[0072] (14) Recovery when failure occurs: If a failure occurs during the asynchronous write process, restart the write operation from the most recent checkpoint.

[0073] like Figure 3 As shown, the load balancing mechanism includes:

[0074] (21) Based on the real-time load situation, a weighted round-robin algorithm is used to allocate transmission tasks: the weighted round-robin algorithm assigns an initial weight to each working node;

[0075] (22) During the polling process of the transmission task, the monitoring agent subnode of the working node collects the current load data of the current working node at the set time interval and uploads the load data to the central monitoring server;

[0076] The central monitoring server determines whether the load of the current working node exceeds the set first threshold. If it exceeds the set first threshold, the weight of the current working node is reduced; determines whether the load of the current working node is lower than the set second threshold, and increases the weight of the current working node; if the load of the current working node is lower than the first threshold and higher than the second threshold, the weight of the current working node is kept unchanged;

[0077] (23) Allocate tasks according to weights.

[0078] It should be understood that a working node refers to a machine or server responsible for storing data and processing file transfers. Each working node is equipped with a monitoring agent subnode, which regularly collects the node's resource usage data and sends this data to the central monitoring server to help achieve load balancing and resource optimization scheduling.

[0079] The monitoring agent subnode is a lightweight monitoring component or service deployed on each working node. It is responsible for real-time collection and monitoring of the resource usage and operation status of the working node. Its main function is to regularly collect key resource indicators of the working node and report these data to the central monitoring server to support the system's load balancing and resource optimization decisions.

[0080] The central monitoring server is the core component or server responsible for centralized management and monitoring of all working nodes in a distributed system. It receives and analyzes resource usage data uploaded by each working node, evaluates the overall load of the system in real time, and makes intelligent scheduling decisions based on this data, such as load balancing and task allocation.

[0081] Furthermore, the weighted polling algorithm (21) assigns an initial weight to each working node, where the initial weight W i The calculation formula is:

[0082] W i =αC i +βM i +γB i +δI i +εD i ;

[0083] Among them, C i represents the CPU capacity of worker node i, M i represents the memory capacity of worker node i, B i represents the network bandwidth of node i, I i represents the input and output I / O performance of worker node i, D i It represents the distance between working node i and the central node. α, β, γ, δ and ε are weight coefficients used to adjust the impact of each resource capacity and distance on the total weight.

[0084] Furthermore, in the polling process of the transmission task (22), the monitoring agent subnode of the working node itself periodically collects the current load data of the current working node, including:

[0085] During the polling process of the transmission task, the current load data of the working node is collected every minute. The current total load L of the working node i i The calculation formula is:

[0086] L i =αL cpu,i +βL mem,i +γL bw,i +δL io,i

[0087] Among them, L cpu,i represents the CPU load of worker node i, L mem,i represents the memory load of worker node i, L bw,i represents the network bandwidth load of worker node i, L io,i represents the input and output I / O load of working node i, according to the current load L i Adjustment right W i .

[0088] Further, the (22) central monitoring server determines whether the load of the current working node exceeds a set first threshold, and if so, reduces the weight of the current working node; determines whether the load of the current working node is lower than a set second threshold, and increases the weight of the current working node, specifically including:

[0089] If the load L of worker node i i If the load exceeds the preset high threshold T1, the weight of the working node i is reduced to W' i ;

[0090] W i '=W i -λ(L i -T1), ifL i >T1

[0091] If the load L of worker node i i If it is lower than the set low threshold T2, the weight of the working node i is increased to W′ i , the adjusted weight W′ i The calculation formula is:

[0092] W′ i =W i +η(T2-L i ), ifL i <T2

[0093] Among them, λ and η are coefficients for adjusting weights.

[0094] It should be understood that (23) assigning tasks according to weights specifically includes:

[0095] W i is the weight of the ith node; in each polling cycle, all working nodes are traversed and tasks are assigned in order according to the weight. Nodes with larger weights are assigned more tasks in each round, while nodes with smaller weights are assigned fewer tasks.

[0096] Exemplarily, (23) assigning tasks according to weights specifically includes:

[0097] If there are three working nodes A, B, and C, their weights are W respectively. A =5, W B =3, W C =2, the tasks will be allocated according to the following pattern:

[0098] Polling 1: Worker node A receives 5 tasks, B receives 3 tasks, and C receives 2 tasks.

[0099] Poll 2: If the load on worker node A becomes high, the weight may be adjusted to W A =4, then in the next task polling, A will receive 4 tasks, and the amount of tasks assigned to B and C will also be dynamically adjusted according to the load.

[0100] It should be understood that the medium-priority files are written asynchronously. The data is first written to the Alluxio cache, which then triggers asynchronous writing to the underlying storage. During the writing process, the system periodically saves the intermediate state of the data through the checkpoint mechanism. This mechanism ensures that in the event of a failure, the writing process can be restarted from the most recent checkpoint without having to start from scratch, reducing data recovery time and system resource consumption.

[0101] To avoid resource competition, the system dynamically monitors the load of each working node and distributes the transfer tasks of medium-priority files to nodes with lower loads. Based on the real-time load situation, the system uses a weighted polling algorithm to allocate transfer tasks. The weighted polling algorithm assigns a weight to each node, which is calculated based on the node's resource capabilities (such as CPU, memory, network bandwidth, and I / O) and the distance between nodes. First, set an initial weight for each node, which reflects the node's maximum resource processing capacity. The initial weight of each node is calculated based on its maximum resource processing capacity.

[0102] This dynamic adjustment mechanism ensures that the system remains balanced when the load changes, avoiding overloading of certain nodes and causing system performance degradation. The system analyzes the load of each node in real time. The monitoring agent sends the collected resource usage data to the central monitoring server, which analyzes the data through algorithms and identifies nodes with lower current loads. These nodes will receive new transmission tasks first, thus achieving load balancing.

[0103] After the transmission task is assigned, the system will continue to monitor the execution of the task. If a node is detected to be faulty, the system will immediately stop assigning new tasks to the node and reallocate the unfinished tasks to other nodes with lower load and closer distance. This ensures that the transmission task can be quickly restored in the event of a failure, reducing the interruption time of data transmission.

[0104] Furthermore, the S102: storing the file to be written into the Alluxio cache and the underlying storage according to the file transfer policy corresponding to the file priority, further includes:

[0105] If the file priority is low, the asynchronous writing mechanism and data compression mechanism are used;

[0106] The asynchronous writing mechanism means that the file to be written is first written into the Alluxio cache, and then the file to be written is written into the underlying storage when the system resources are idle.

[0107] The data compression mechanism refers to:

[0108] When the network bandwidth utilization is higher than the set threshold, while the CPU utilization and the input / output interface utilization are lower than the set threshold, the LZ4 compression algorithm is selected to compress the file to be written;

[0109] When the network bandwidth utilization is lower than the set threshold, and the CPU utilization and input / output interface utilization are higher than the set threshold, the Gzip compression algorithm is selected to compress the file to be written;

[0110] When the network bandwidth utilization, CPU utilization, and input / output interface utilization are all within the set threshold range, the Zstandard compression algorithm is selected to compress the files to be written.

[0111] It should be understood that the LZ4 compression algorithm is a lossless data compression algorithm, which is known for its high compression and decompression speed. It can compress and decompress data in a very short time.

[0112] It should be understood that the Gzip compression algorithm is a commonly used lossless data compression algorithm. It can significantly compress the file size while ensuring data integrity. Gzip is often used to reduce file size in order to save space and bandwidth in file storage and file transfer. When the network bandwidth utilization is low and the CPU and I / O utilization is high, the purpose of selecting the Gzip compression algorithm is to use its high compression rate to reduce the file size, thereby reducing the pressure on the CPU and I / O and making efficient use of network resources.

[0113] It should be understood that the Zstandard compression algorithm is a lossless data compression algorithm. It is designed to provide the best balance between compression speed, decompression speed and compression ratio. It can provide extremely fast processing speed while maintaining an efficient compression ratio.

[0114] It should be understood that the data of low-priority files is first written to the cache, and asynchronous writing to the underlying storage is triggered when system resources are idle. After the file is written to the Alluxio cache, the system will continue to monitor the current resource usage. The utilization of network bandwidth, CPU, and I / 0 is comprehensively considered to decide whether to trigger asynchronous writing. When the utilization is lower than the respective set thresholds, the asynchronous writing process is triggered to transfer the data of low-priority files from the cache to the underlying storage, avoiding the competition between low-priority tasks for high-priority task resources, thereby optimizing overall performance.

[0115] To reduce the amount of data transmitted and bandwidth usage, the system determines the file size when transmitting low-priority files. If the file size is larger than the set threshold, the system will compress the file before transmitting it. In terms of the selection of compression algorithms, the type of compression algorithm to be used is dynamically determined based on real-time resource utilization and task requirements.

[0116] When the system detects that the network bandwidth utilization is high, while the CPU and I / O utilization is low, this usually indicates that the transmission bandwidth is the current main bottleneck. In this case, the system will select the LZ4 algorithm because LZ4 provides very fast compression and decompression speeds, which can quickly reduce the amount of data and relieve bandwidth pressure. LZ4 is suitable for scenarios where large amounts of data need to be transmitted quickly.

[0117] When the system detects that the CPU and I / O utilization are high, but the network bandwidth utilization is low, this indicates that the system resources are relatively sufficient and suitable for more efficient compression. In this case, the system selects the Gzip algorithm. Gzip has a better compression ratio than LZ4. Although the compression and decompression speeds are slightly slower, it can significantly reduce the amount of transmitted data and is suitable for scenarios that do not require high transmission latency but require a higher compression ratio.

[0118] When the system detects that the utilization of CPU, I / O, and network bandwidth are all at a medium level and a balance needs to be achieved between compression ratio and compression speed, the Zstandard (Zstd) algorithm will be selected. Zstd provides a high compression ratio while also providing fast compression and decompression speeds, making it suitable for situations that require comprehensive performance.

[0119] Furthermore, the step S103: obtaining the data to be read, determining whether the data to be read is in the cache, and if so, returning the data in the cache to the user; if not, returning the data in the underlying storage to the user, specifically includes:

[0120] When Alluxio receives a data read request from a user, it first checks whether the requested data is already in the Alluxio cache.

[0121] If the data is in the cache, Alluxio reads it directly from the cache and returns it to the user;

[0122] If the data is not in the cache, Alluxio retrieves the data from the underlying storage (such as HDFS, S3, etc.) and returns it to the user;

[0123] At the same time, Alluxio caches the data obtained from the underlying storage locally and updates the metadata for the next access.

[0124] Furthermore, if Figure 5As shown, the S104: according to the association rule, pre-fetching operation is performed on the data to be read and the data blocks associated with the data to be read. If the cache usage rate exceeds the set threshold, a cache replacement operation is performed, which specifically includes:

[0125] S104-1: Setting the size of the sliding window and the step length of the sliding window;

[0126] S104-2: Calculate closed frequent itemsets for the data in the sliding window;

[0127] S104-3: Construct a Trie tree to store closed frequent itemsets, merge the same prefixes, and obtain the association relationship between data blocks based on the Trie tree;

[0128] S104-4: Determine whether the data accessed by the user is in the cache, if yes, proceed to S104-5; if not, proceed to S104-7;

[0129] S104-5: Determine whether the data block associated with the current data block is in the cache. If yes, return to S104-1; if not, mark the data block associated with the current data block as a "pre-fetch candidate" and enter S104-6; the associated data block is obtained based on the Trie tree. If data block A and data block B belong to the same closed frequent item set, it means that data block A and data block B are associated, otherwise, it means that data block A and data block B are not associated; the pre-fetch candidate refers to the candidate data block waiting to be selected;

[0130] S104-6: Determine whether the load is lower than a set threshold, if not, return to S104-1; if yes, store the data block marked as "prefetch candidate" into the cache, and return to S104-1;

[0131] S104-7: Determine whether the cache usage rate exceeds the set threshold. If not, store the data block accessed by the user in the cache and proceed to S104-5; if yes, remove the data block based on the LRU strategy and mark the associated data block; proceed to S104-8;

[0132] S104-8: Determine whether the difference between the most recent access time of the associated data block and the current time exceeds the set time threshold. If yes, remove the current data block and return to S104-7; if no, return to S104-7.

[0133] Furthermore, the S104-1 sets the size of the sliding window and the step size of the sliding window, including: the size of the sliding window is the amount of data contained in the sliding window, and the step size of the sliding window is the amount of new data added each time the window slides; the data in the window is regarded as a queue, and if the amount of data in the queue exceeds the set window size W, the earliest data added to the queue is removed to ensure that there are only the latest W data in the queue; the removed data is marked as "old data"; when new data arrives, the new data is added to the current window and marked as "new data". The "new data" mark of the data previously marked as "new data" is removed after the new round of data is added. In other words, only the data added in the latest round is "new data".

[0134] It should be understood that due to the huge amount of data to be mined, full data mining may be a heavy task, so a sliding window strategy is adopted. First, the window size and sliding step size need to be set. The window size determines the amount of data contained in each window, and the sliding step size determines the amount of new data added each time the window slides.

[0135] In order to manage the data in the window, a queue needs to be maintained to store the data in the current window. Whenever new data arrives, it is added to the end of the queue and the size of the queue is checked. If the amount of data in the queue exceeds the set window size W, the earliest added data is removed from the front of the queue to ensure that there are always only the latest W data in the queue.

[0136] During the data update process, when new data arrives, it is first added to the current window and marked as "new data" to distinguish which data is newly added and which data is already existing, thereby realizing incremental update.

[0137] The size of the queue is checked periodically. When the amount of data in the queue exceeds the window size W, the earliest data is removed from the queue and marked as "old data".

[0138] Furthermore, the S104-2: calculating closed frequent itemsets for the data in the sliding window includes:

[0139] S104-21: Each time the window slides, only the data marked as "new data" is processed, and the closed frequent itemsets of the "new data" in the window are calculated; the closed frequent itemsets of the "new data" in the window are merged with the closed frequent itemsets of all the data in the window, and the closed frequent itemset list is updated;

[0140] S104-22: Divide the entire window into several subsets; and assign the subsets to different working nodes;

[0141] S104-23: Based on the closed frequent itemset list, each working node calculates a local closed frequent itemset and performs pruning processing in a local range;

[0142] S104-24: After each working node completes the local closed frequent itemset mining, it first performs local merging on some nodes, and then sends the merged results to the central monitoring server for final merging;

[0143] S104-25: De-duplicate the closed frequent itemsets generated by each node, merge the same closed frequent itemsets, and recalculate the support of the merged closed frequent itemsets to ensure that their support is correct in the global scope.

[0144] S104-26: Perform final pruning on the merged global closed frequent itemsets to remove itemsets that do not meet the support threshold, that is, itemsets that are locally closed frequent itemsets but not globally closed frequent itemsets.

[0145] Furthermore, the closed frequent itemset means that if the support of an itemset (ie, the proportion of transactions containing the itemset to the total transactions) meets a set threshold, then the itemset is called a frequent itemset.

[0146] The support of an itemset is defined as:

[0147]

[0148] A transaction represents a set of data records processed in a sliding window, and each transaction usually consists of multiple data items. For example, in a data stream processing, each transaction may correspond to a purchase behavior, a set of user operation records, or a specific event. In frequent item set mining, the system calculates the support of the item set based on these transactions and analyzes which data item combinations often appear together. The total number of transactions in the sliding window is the total number of data records currently covered by the window, which serves as the basis for calculating frequent item sets.

[0149] Furthermore, the item set refers to a set of data composed of multiple data items in the transaction within the sliding window. It is usually used to describe the combination of items that appear together in a transaction database or data set. Item sets can be used to represent several elements that appear simultaneously in a transaction or record, and are widely used in tasks such as association rule mining and frequent item set mining. Frequent item sets are item sets that appear above a certain support threshold. It reflects the combination of items that appear frequently in the data.

[0150] Itemset I refers to the combination of several data items that appear simultaneously in the transactions within the sliding window. It can be used to represent data items that often appear together in a transaction database or data set.

[0151] Let's take an example to illustrate the meaning of item set I:

[0152] Assume that the current sliding window contains the following transaction data (each transaction is a set of events or data items that occur simultaneously):

[0153] Transaction 1: {apple, banana, pear};

[0154] Transaction 2: {apple, pear};

[0155] Transaction 3: {banana, grape};

[0156] Transaction 4: {apple, banana};

[0157] In this case, an itemset I can be any combination of items that appear in a transaction at the same time. For example:

[0158] The item set {apple, pear} means that apple and pear appear in the same transaction (appear in transaction 1 and transaction 2).

[0159] The item set {banana, grape} indicates that banana and grape appear at the same time in the same transaction (appearing in transaction 3).

[0160] The item set {apple, banana} means that apple and banana appear in the same transaction (appear in transaction 1 and transaction 4).

[0161] If the support of the item set I (i.e., the frequency of the item set appearing in transactions within the sliding window) meets the set threshold, then it is a frequent item set. For example, if the item set {apple, pear} appears in 2 transactions and the threshold is 50%, then it is a frequent item set.

[0162] Regarding the processing of "new data":

[0163] Each time the window slides, only the data marked as "new data" will be processed. For example, if transaction 4 is the latest (that is, newly slid into the window), then the current sliding window will only process the data items {apple, banana} in transaction 4 to update the frequent itemset.

[0164] Furthermore, the closed frequent itemset is a special type of frequent itemset, which not only meets the conditions of frequent itemsets (i.e., its support is higher than the set threshold), but also has closure. The definition of closure is: if the superset of a frequent itemset (i.e., the larger itemset containing the itemset) has the same support as it, then the itemset is no longer a closed frequent itemset, and the superset is considered a closed frequent itemset. Therefore, closed frequent itemsets represent those itemsets that cannot be covered by larger itemsets.

[0165] A closed frequent itemset is not only a frequent itemset, but also has no other larger itemset containing the itemset with the same support in the data.

[0166] It should be understood that each time the window is slid, only those transactions marked as "new data" are processed and their closed frequent itemsets are calculated. For the removal of old data, when data is removed from the window, the system needs to update the current frequent itemsets and remove those affected itemsets, that is, those itemsets whose support no longer meets the threshold due to the removal of data. Finally, the mining results of the newly added data are merged with the closed frequent itemset results of the current window, and the closed frequent itemset list is updated. This requires recalculating the support of all affected itemsets to ensure their support in the current window.

[0167] The idea of ​​using mining algorithm when mining closed frequent itemsets is the parallelized CHARM algorithm, which divides the entire data window into several subsets, each of which can be processed independently.

[0168] The CHARM algorithm comes from: Zaki, Mohammed J., and Ching-Jui Hsiao. "CHARM: An efficient algorithm for closed itemset mining." Proceedings of the 2002SIAMinternational conference on data mining. Society for Industrial and AppliedMathematics, 2002.

[0169] The partitioning strategy uses transaction partitioning to divide the data window into equal parts by quantity, ensuring that the amount of data in each subset is balanced and avoiding excessive concentration of calculations in certain subsets.

[0170] In the parallel computing stage, the distributed computing framework Apache Hadoop is used to assign tasks to different computing nodes, and a hash table is used to record local frequent itemsets and their support.

[0171] Each node runs the CHARM algorithm independently, performs local closed frequent item set mining on its own subset, and performs pruning in a local range.

[0172] After each computing node completes the mining of local closed frequent itemsets, it first performs local merging in some nodes, and then sends the merged results to the central node for final merging.

[0173] The closed frequent itemsets generated by each node are deduplicated, the same closed frequent itemsets are merged, and the support of the merged closed frequent itemsets is recalculated to ensure that their support is correct globally.

[0174] The final pruning is performed on the merged global closed frequent itemsets to remove itemsets that do not meet the support threshold, that is, itemsets that are locally closed frequent itemsets but not globally closed frequent itemsets.

[0175] Furthermore, the S104-21: merging the closed frequent itemsets of the “new data” in the window with the closed frequent itemsets of all the data in the window specifically includes:

[0176] In the sliding window strategy, when "new data" enters the window and its closed frequent itemsets are calculated, the closed frequent itemsets of the "new data" are merged with the closed frequent itemsets of all data in the current window.

[0177] First, for each closed frequent itemset of the "new data", check whether it already exists in the frequent itemset of the current window. If it does, update the support of the itemset; if it does not exist, add it as a new itemset;

[0178] Next, recalculate the support of all item sets in the entire window, and remove the item sets whose support is lower than the set threshold from the frequent item set;

[0179] Finally, during the merging process, if a superset of an itemset has the same support, the itemset is removed.

[0180] This merging process can dynamically update frequent itemsets to reflect incremental changes in data while maintaining the integrity and correctness of frequent itemsets.

[0181] A superset is a larger set formed by adding other data items based on a certain item set.

[0182] For example:

[0183] Suppose we have an itemset A = {apple, banana}, then the itemset {apple, banana, pear} is a superset of A because it contains all the data items of A and has an additional data item "pear".

[0184] Itemset A = {apple, banana}

[0185] Itemset B = {apple, banana, pear}

[0186] Itemset B is a superset of itemset A because B contains all the elements of A and has more elements.

[0187] In the definition of closed frequent itemsets, if the superset of a frequent itemset has the same support as it (that is, they appear simultaneously in the same number of transactions), then the frequent itemset is not a closed frequent itemset, and the superset is a closed frequent itemset.

[0188] Example: Suppose the item set {apple, banana} appears in 3 transactions, and its superset {apple, banana, pear} also appears in the same 3 transactions, then {apple, banana} is not a closed frequent item set, only {apple, banana, pear} is a closed frequent item set, because it contains more items, and no larger item set appears in the same transaction.

[0189] Furthermore, the S104-21: updating the closed frequent itemset list includes:

[0190] The closed frequent itemset list is a structure that stores all closed frequent itemsets and their corresponding supports in the current window. It is used to track frequent patterns in a dataset and maintain its accuracy through an incremental update mechanism. In a sliding window, when "new data" enters, the new closed frequent itemsets need to be merged and updated with the current closed frequent itemset list.

[0191] Closed frequent itemset list = [{itemset:{A,B},support:0.8},{itemset:{A,C},support:0.3}].

[0192] Furthermore, the S104-22: dividing the entire window into a number of subsets, including:

[0193] Dividing the entire window into several subsets means dividing the data within the window.

[0194] In the process of data mining, in order to improve computing efficiency and take advantage of parallel processing, large data sets (such as all data in a sliding window) are usually split into multiple subsets, and these subsets are assigned to different computing nodes for parallel processing.

[0195] Furthermore, the S104-22: allocating the subsets to different computing nodes includes:

[0196] In the parallel computing stage, the distributed computing framework Apache Hadoop is used to assign tasks to different computing nodes.

[0197] Furthermore, the S104-23: based on the closed frequent itemset list, each computing node calculates a local closed frequent itemset, including:

[0198] The mining process of local closed frequent itemsets is similar to that of global closed frequent itemsets, but it only works on the data of each subset. Specifically, local closed frequent itemsets refer to frequent itemsets calculated within a certain data subset and meet the definition of closed frequent itemsets.

[0199] Furthermore, if Figure 4 As shown, the S104-23: and performing pruning processing in a local range, including:

[0200] In local calculation, if the superset of an itemset has the same support, the itemset is no longer a closed frequent itemset and needs to be removed. In addition, if the support of an itemset is lower than the set minimum support threshold, the itemset will also be pruned to prevent it from entering the subsequent calculation stage.

[0201] After local pruning is completed, the node merges the calculation results and further performs pruning operations on a global scale. Global pruning removes itemsets that are closed frequent itemsets in the local scope, but no longer meet the closed frequent itemset conditions in the global scope due to reduced support. Pruning operations reduce computational complexity and improve the overall efficiency of the system by removing redundant itemsets.

[0202] A superset is a larger item set formed by adding one or more data items to the original item set. In other words, the superset of item set A is a set that contains all the data items of item set A and at least one more data item.

[0203] Furthermore, the S104-24: after each computing node completes the local closed frequent item set mining, firstly performs local merging on some nodes, specifically includes:

[0204] When each computing node completes the calculation of the local closed frequent itemsets, the local closed frequent itemsets generated by each computing node are first merged. The local merging process refers to: preliminarily integrating the local closed frequent itemsets generated in the same group of nodes to reduce redundant and duplicate itemsets.

[0205] When merging, compare the local frequent itemsets of each node, find the same itemsets, and add their supports.

[0206] If it is found during the local merging process that the support of some item sets does not reach the set minimum threshold, pruning is performed directly in the local range to remove these item sets.

[0207] By performing local merging on some nodes first, the amount of frequent item set data transmitted to the central node can be effectively reduced, the computing and transmission overhead can be reduced, and the efficiency of the entire distributed system can be improved.

[0208] Furthermore, the S104-24: then sending the merged results to the central monitoring server for final merging specifically includes:

[0209] After each computing node completes the local closed frequent itemset mining and performs preliminary local merging, the merged results are sent to the central monitoring server through the communication mechanism built into the distributed computing framework Hadoop.

[0210] Each node packages the merged local frequent itemsets into a result set and sends the result set to the central monitoring server via network transmission.

[0211] After receiving the local results from all computing nodes, the central monitoring server is responsible for merging and recalculating the support to ensure that the final closed frequent itemsets have correct support globally.

[0212] At the same time, the central node will also deduplicate and prune during the merging process to remove redundant itemsets or itemsets that do not meet the support threshold, thereby generating a complete list of global closed frequent itemsets.

[0213] Furthermore, the S104-25: removing duplicates from the closed frequent item set results generated by each node includes:

[0214] After each node generates local closed frequent itemsets, the central node will first deduplicate these results.

[0215] This process is to compare the frequent itemsets uploaded by different nodes, identify the same itemsets, and merge them.

[0216] For example, if the itemsets generated by different nodes have the same content, they are considered to be duplicate itemsets.

[0217] When merging, the central node will accumulate the support of these duplicate itemsets to ensure that the support reflects the true frequency of the itemsets in the global scope. Through this deduplication and merging operation, the system can avoid redundant storage and processing and ensure the correctness and accuracy of the global closed frequent itemset list.

[0218] Furthermore, the S104-25: merging identical closed frequent item sets includes:

[0219] Merging the same closed frequent itemsets means that the central node integrates the itemsets with the same content in the local closed frequent itemset results generated by each computing node.

[0220] If multiple nodes independently calculate the same closed frequent itemsets, these itemsets are duplicated globally, so they need to be merged.

[0221] During the merging process, the central node adds the support of the same itemsets to reflect the frequency of the itemsets in the entire dataset.

[0222] This process ensures that each item set in the global closed frequent item set list appears only once, and its support accurately represents the frequency of the item set in all data.

[0223] Furthermore, the S104-26: performing final pruning in the merged global closed frequent itemsets to remove itemsets that do not meet the support threshold, that is, itemsets that are locally closed frequent itemsets but not globally closed frequent itemsets, includes:

[0224] The final pruning process in the merged global closed frequent itemsets is to ensure that only itemsets that meet the global support threshold are retained.

[0225] Although some itemsets satisfy the condition of closed frequent itemsets in local nodes, they no longer meet the criteria of frequent itemsets globally because their support may decrease.

[0226] Therefore, the system will remove those item sets whose support is lower than the set threshold by comparing the global support of each item set.

[0227] In addition, if a superset of an itemset has the same support, the itemset will also be pruned because it is no longer a closed itemset in the global scope.

[0228] Through this pruning operation, it is ensured that the final retained itemset is both frequent and closed globally.

[0229] An item set that does not meet the support threshold means that the support of an item set is lower than the pre-set minimum support threshold within the global data range. Support measures the frequency of an item set in the data set. If the support of an item set does not meet the threshold requirement, it is considered not frequent enough in the global scope and cannot be considered a frequent item set.

[0230] Furthermore, the S104-3: constructing a Trie tree to store closed frequent itemsets and merging identical prefixes includes:

[0231] S104-31: Use Trie tree to store closed frequent item sets; each node represents an item, and each path represents a closed frequent item set;

[0232] S104-32: During the storage process, child nodes with the same prefix are merged to reduce repeated storage; when inserting frequent item sets, check whether there is a common prefix, for example, for frequent item sets "ABC" and "ABD", the prefix "AB" is common;

[0233] S104-33: If the prefix node does not exist, create a prefix node, if the "AB" node does not exist, create node "A", and then create node "B" under node "A";

[0234] S104-34: Based on the prefix node, create branches for different child nodes. For "ABC", create node "C" under node "B"; for "ABD", create node "D" under node "B";

[0235] S104-35: At each terminal node (ie, leaf node), the support of the frequent itemset is stored.

[0236] It should be understood that the final constructed Trie tree contains the organizational structure of all closed frequent itemsets: each node represents an item, a path represents a frequent itemset, itemsets with a common prefix share the path, and leaf nodes store the support of the corresponding frequent itemsets.

[0237] Furthermore, the S104-32: merging sub-nodes with the same prefix during storage includes:

[0238] In the process of storing frequent item sets, child nodes with the same prefix are merged by constructing a Trie tree: when inserting a new frequent item set, first start from the root node and check layer by layer whether the prefix node of the item set already exists in the tree. If it exists, continue to insert subsequent items along the path; if it does not exist, create a new node. In this way, if different frequent item sets share the same prefix (such as "ABC" and "ABD" share the prefix "AB"), their common prefix path will only be stored once, thereby reducing the storage of duplicate nodes and improving storage efficiency.

[0239] Further, the S104-34: based on the prefix node, creating branches for different child nodes, for "ABC", creating a node "C" under the "B" node; for "ABD", creating a node "D" under the "B" node; includes:

[0240] In the process of storing frequent item sets, child nodes with the same prefix are merged by constructing a Trie tree: when inserting a new frequent item set, first start from the root node and check layer by layer whether the prefix node of the item set already exists in the tree. If it exists, continue to insert subsequent items along the path; if it does not exist, create a new node. In this way, if different frequent item sets share the same prefix (such as "ABC" and "ABD" share the prefix "AB"), their common prefix path will only be stored once, thereby reducing the storage of duplicate nodes and improving storage efficiency. Among them, ABC and ABD have a common prefix AB, and C and D after AB are different branches.

[0241] In this way, all item sets with the same prefix share the same prefix path. This merging method not only reduces the number of nodes stored, but also improves query efficiency and optimizes storage and search performance.

[0242] Proper organization and storage of rule sets can help find association rules more quickly and effectively, thereby speeding up searches and improving efficiency.

[0243] Furthermore, the S104-7, based on the LRU strategy, removes the data block, including:

[0244] Alluxio itself provides an LRU strategy. When the cache usage exceeds the set threshold (for example, 80%), the system will start the LRU strategy to find and remove the data blocks that have not been accessed the longest. Considering the association between data blocks, the associated data blocks of the removed data blocks will also be removed.

[0245] The LRU (Least Recently Used) strategy is a cache replacement algorithm that is mainly used to manage data in the cache. When the cache space is limited and needs to make room for new data, the LRU strategy selects the cache items that have not been used for the longest time to be eliminated. This algorithm assumes that recently used data is likely to be used again, while data that has not been used for a long time is unlikely to be used.

[0246] Furthermore, the step S104-7 of marking the associated data blocks includes:

[0247] Each data block can have a status field in the metadata, indicating its current marking status (such as "prefetch candidate" or "to be removed"). This metadata is stored in the cache with the data block, and once the status changes, the metadata can be updated to reflect the change in marking.

[0248] In cross-domain scenarios where Alluxio cache is combined with underlying storage, network latency may prevent I / O speed from syncing with computing speed, thus affecting the performance of external services. By using prefetch and replace techniques, hot data can be persistently retained in the Alluxio cache, reducing the number of remote calls and accesses to the underlying storage, and reducing data access latency and communication frequency under high load. Therefore, the present invention will adopt a cache strategy based on the association between data blocks, use the parallelized CHARM algorithm to obtain the association rules between data blocks, adjust the cache content according to the obtained association rules, implement cache prefetch and replacement operations, and thus improve the availability of Alluxio.

[0249] Currently, Alluxio’s prefetching strategy is that when the data requested by the user is not in the Alluxio cache, Alluxio will obtain the required data from the underlying storage system (such as HDFS, S3, etc.) and store it in the Alluxio cache.

[0250] Based on the current Alluxio prefetching strategy, a more intelligent prefetching strategy is designed taking into account the relevance of data blocks and system load.

[0251] When a user accesses data block a, if data block a is not in the cache, Alluxio obtains data block a from the underlying storage system and stores it in the cache, and then looks for several data blocks associated with a to check whether these data blocks are in the cache. If these associated data blocks are not in the cache, they are marked as prefetch candidates.

[0252] When the system load is lower than the set threshold, idle prefetching is triggered. Data blocks marked as prefetch candidates are selected from the prefetch candidate list, prefetched in batches from the underlying storage system, and stored in the cache.

[0253] When the cache usage rate exceeds the set upper limit (such as 50%), based on the LRU strategy, the least recently used data block is removed, and the data block associated with the replaced data block a is marked as a candidate for removal;

[0254] If the last access time and current time of the associated data block exceed the set time threshold, it means that the data block has rarely been accessed recently, and consider removing them together to free up more cache space. If the last access time and current time of the associated data block are less than the set time threshold, it means that the data block has been accessed recently, and they are retained in the cache.

[0255] Embodiment 2

[0256] This embodiment provides a data storage and cache optimization system based on Alluxio, including:

[0257] The acquisition module is configured to: acquire the file to be written, and calculate the priority of the file according to four attributes of the file to be written; the four attributes of the file to be written include: file access frequency, file capacity, file importance level and file freshness;

[0258] The write module is configured to store the files to be written into the Alluxio cache and underlying storage according to the file transfer policy corresponding to the file priority;

[0259] A reading module is configured to: obtain the data to be read, determine whether the data to be read is in the cache, and if so, return the data in the cache to the user; if not, return the data in the underlying storage to the user;

[0260] The pre-fetch module is configured to: perform pre-fetch operations on the data to be read and the data blocks associated with the data to be read according to the association rules, and perform a cache replacement operation if the cache usage rate exceeds a set threshold.

[0261] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The data storage and cache optimization method based on Alluxio is characterized by: include: Obtain a file to be written, and calculate the priority of the file according to four attributes of the file to be written; The four attributes of the file to be written include: file access frequency, file capacity, file importance level and file freshness; Store the files to be written into the Alluxio cache and underlying storage according to the file transfer policy corresponding to the file priority. Get the data to be read and determine whether the data to be read is in the cache. If yes, return the data in the cache to the user; if not, return the data in the underlying storage to the user; According to the association rule, a pre-fetch operation is performed on the data to be read and the data blocks associated with the data to be read. If the cache usage rate exceeds a set threshold, a cache replacement operation is performed.

2. The data storage and cache optimization method based on Alluxio as claimed in claim 1, characterized in that: The calculating the priority of the file according to the four attributes of the file to be written includes: Assume that for file i, the original indicators are: access frequency F i 、File size S i 、The Importance of Documents i and the freshness of the data N i , standardize each indicator; combine the standardized indicators and calculate the priority P of each file according to the weight ratio i For: P i =aF i +bS i +cI i +dN i , where a, b, c and d are weights, a+b+c+d=1.

3. The data storage and cache optimization method based on Alluxio as claimed in claim 1, characterized in that: According to the file transfer policy corresponding to the file priority, the files to be written are stored in the Alluxio cache and underlying storage, including: If the file priority is high, the synchronous write mechanism and the dual copy mechanism are used; The synchronous writing mechanism means writing the file to be written into the Alluxio cache and the underlying storage at the same time; the dual copy mechanism means copying each data block to another working node; After the data is written into the underlying storage, data verification begins. The data verification is used to determine whether the written file is damaged. If the verification succeeds, it means that the file is written. If the verification fails, it means that the file needs to be rewritten. According to the file transfer policy corresponding to the file priority, the file to be written is stored in the Alluxio cache and the underlying storage. If the file priority is medium, the asynchronous writing mechanism and load balancing mechanism are used. The asynchronous writing mechanism means that the file to be written is first written to the Alluxio cache, and then written to the underlying storage; during the writing process, the intermediate state of the saved data is regularly checked through the checkpoint mechanism; According to the file transfer policy corresponding to the file priority, the files to be written are stored in the Alluxio cache and the underlying storage, which also includes: If the file priority is low, the asynchronous writing mechanism and data compression mechanism are used; The asynchronous writing mechanism means that the file to be written is first written into the Alluxio cache, and then the file to be written is written into the underlying storage when the system resources are idle.

4. The data storage and cache optimization method based on Alluxio as claimed in claim 3 is characterized in that: The load balancing mechanism includes: Based on the real-time load situation, a weighted round-robin algorithm is used to allocate transmission tasks: the weighted round-robin algorithm assigns an initial weight to each working node; During the polling process of the transmission task, the monitoring agent subnode of the working node collects the current load data of the current working node at the set time interval and uploads the load data to the central monitoring server; The central monitoring server determines whether the load of the current working node exceeds the set first threshold. If it exceeds the set first threshold, the weight of the current working node is reduced; determines whether the load of the current working node is lower than the set second threshold, and increases the weight of the current working node; if the load of the current working node is lower than the first threshold and higher than the second threshold, the weight of the current working node is kept unchanged; Assign tasks based on weights.

5. The data storage and cache optimization method based on Alluxio as claimed in claim 4, characterized in that: The weighted polling algorithm assigns an initial weight to each working node, where the initial weight W i The calculation formula is: W i =αC i +βM i +γB i +δI i +εD i ; Among them, C i represents the CPU capacity of worker node i, M i represents the memory capacity of worker node i, B i represents the network bandwidth of node i, I i represents the input and output I / O performance of worker node i, D i represents the distance between working node i and the central node; α, β, γ, δ, and ε are weight coefficients used to adjust the impact of each resource capacity and distance on the total weight; During the polling process of the transmission task, the monitoring agent subnode of the working node itself regularly collects the current load data of the current working node, including: During the polling process of the transmission task, the current load data of the working node is collected every minute. The current total load L of the working node i i The calculation formula is: L i =αL cpu,i +βL mem,i +γL bw,i +δL io,i Among them, L cpu,i represents the CPU load of worker node i, L mem,i represents the memory load of worker node i, L bw,i represents the network bandwidth load of worker node i, L io,i represents the input and output I / O load of working node i, according to the current load L i Adjustment right W i ; The central monitoring server determines whether the load of the current working node exceeds a set first threshold, and if so, reduces the weight of the current working node; determines whether the load of the current working node is lower than a set second threshold, and increases the weight of the current working node, specifically including: If the load L of worker node i i If the load exceeds the preset high threshold T1, the weight of the working node i is reduced to W i ′; W i '=W i -λ(L i -T1), ifL i >T1 If the load L of worker node i i If the value is lower than the set low threshold T2, the weight of the working node i is increased to W. i ′, the adjusted weight W i The calculation formula is: W i ′=W i +η(T2-L i ),ifL i <T2 Among them, λ and η are coefficients for adjusting weights.

6. The data storage and cache optimization method based on Alluxio as claimed in claim 3, characterized in that: The data compression mechanism refers to: when the network bandwidth utilization is higher than a set threshold, while the CPU utilization and the input and output interface utilization are lower than the set threshold, the LZ4 compression algorithm is selected to compress the file to be written; When the network bandwidth utilization is lower than the set threshold, and the CPU utilization and input / output interface utilization are higher than the set threshold, the Gzip compression algorithm is selected to compress the file to be written; When the network bandwidth utilization, CPU utilization, and input / output interface utilization are all within the set threshold range, the Zstandard compression algorithm is selected to compress the files to be written.

7. The data storage and cache optimization method based on Alluxio as claimed in claim 1, characterized in that: According to the association rules, the data to be read and the data blocks associated with the data to be read are pre-fetched. If the cache usage exceeds the set threshold, a cache replacement operation is performed, which specifically includes: (1): Set the size of the sliding window and the step size of the sliding window; (2): Calculate closed frequent itemsets for the data in the sliding window; (3): Construct a Trie tree to store closed frequent itemsets, merge the same prefixes, and obtain the association relationship between data blocks based on the Trie tree; (4): Determine whether the data accessed by the user is in the cache. If yes, go to (5); if not, go to (7); (5): Determine whether the data block associated with the current data block is in the cache. If so, return to (1); if not, mark the data block associated with the current data block as a "pre-fetch candidate" and enter (6); the associated data blocks are obtained based on the Trie tree. If data block A and data block B belong to the same closed frequent item set, it means that data block A and data block B are associated, otherwise, it means that data block A and data block B are not associated; the pre-fetch candidate refers to the candidate data block waiting to be selected; (6): Determine whether the load is lower than the set threshold. If not, return to (1); if yes, store the data block marked as "prefetch candidate" in the cache and return to (1); (7): Determine whether the cache usage rate exceeds the set threshold. If not, store the data block accessed by the user in the cache and enter (5); if yes, remove the data block based on the LRU strategy and mark the associated data block; enter (8); (8): Determine whether the difference between the most recent access time of the associated data block and the current time exceeds the set time threshold. If so, remove the current data block and return to (7); if not, return to (7).

8. The data storage and cache optimization method based on Alluxio as claimed in claim 7, characterized in that: The size of the sliding window and the step size of the sliding window are set, including: the size of the sliding window is the amount of data contained in the sliding window, and the step size of the sliding window is the amount of new data added each time the window slides; the data in the window is regarded as a queue, if the amount of data in the queue exceeds the set window size W, the earliest data added to the queue is removed to ensure that there are only the latest W data in the queue; the removed data is marked as "old data"; when new data arrives, the new data is added to the current window and marked as "new data"; the data previously marked as "new data" has its "new data" mark removed after a new round of data is added.

9. The data storage and cache optimization method based on Alluxio as claimed in claim 7, characterized in that: For the data in the sliding window, calculate the closed frequent itemsets, including: Each time the window slides, only the data marked as "new data" is processed, and the closed frequent itemsets of the "new data" in the window are calculated; the closed frequent itemsets of the "new data" in the window are merged with the closed frequent itemsets of all the data in the window, and the closed frequent itemset list is updated; Divide the entire window into several subsets; assign the subsets to different working nodes; Based on the closed frequent itemset list, each working node calculates the local closed frequent itemset and performs pruning in the local range; After each working node completes the local closed frequent itemset mining, it first performs local merging on some nodes, and then sends the merged results to the central monitoring server for final merging; Remove duplicate closed frequent item sets generated by each node, merge identical closed frequent item sets, and recalculate the support of the merged closed frequent item sets to ensure that their support is correct globally. The final pruning is performed on the merged global closed frequent itemsets to remove itemsets that do not meet the support threshold.

10. The data storage and cache optimization system based on Alluxio is characterized by: include: An acquisition module is configured to: acquire a file to be written, and calculate a priority of the file according to four attributes of the file to be written; The four attributes of the file to be written include: file access frequency, file capacity, file importance level and file freshness; The write module is configured to store the files to be written into the Alluxio cache and underlying storage according to the file transfer policy corresponding to the file priority; A reading module is configured to: obtain the data to be read, determine whether the data to be read is in the cache, and if so, return the data in the cache to the user; if not, return the data in the underlying storage to the user; The pre-fetch module is configured to: perform pre-fetch operations on the data to be read and the data blocks associated with the data to be read according to the association rules, and perform a cache replacement operation if the cache usage rate exceeds a set threshold.

Images