Cache optimization method and device based on frequency and weight, equipment and medium

Abstract

The invention relates to the technical field of data processing, can be applied to business system platforms of medical health, financial science and technology and the like, and discloses a frequency and weight-based cache optimization method, which comprises the following steps of: acquiring access data items, and distributing the access data items into clusters according to access mode characteristics; calculating a Hash fingerprint of the access data item, determining a cluster position of the accessed data according to the Bloom filter and the Hash fingerprint, and updating a global counter of the cluster; obtaining the multi-dimensional features of the access data items and the time sequence of the access data items, and inputting the time sequence into the recurrent neural network to generate a weight three-dimensional vector; constructing an index according to the access data item, the subordinate cluster, the Hash fingerprint and the weight three-dimensional vector, and determining a comprehensive elimination strategy through the index; and determining a cache optimization strategy according to the global counter, the weight three-dimensional vector and a comprehensive elimination strategy. According to the method, the data is cached through the clustering hash, the time sequence neural network and the hierarchical index structure, so that the storage cost is remarkably reduced, and elastic expansion of resources is realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cache optimization method, device, equipment and storage medium based on frequency and weight. Background Art

[0002] LFU (Least Frequently Used) is a cache replacement algorithm whose core idea is to prioritize replacing the least frequently used objects. However, traditional LFU algorithms have many problems in practical applications.

[0003] First, the traditional LFU algorithm faces the linear expansion problem of frequency storage. This algorithm allocates a separate counter to each data item to record access frequency. When the data scale reaches hundreds of millions, frequency storage alone consumes a large amount of memory. For example, if each counter occupies 8 bytes, 100 million data items requires 80GB of memory, significantly reducing cache space utilization.

[0004] Secondly, traditional LFU algorithms suffer from the drawback of static frequency characteristics. They typically use a fixed window to count frequencies, which cannot effectively capture the temporal and spatial variations in access patterns. On the one hand, in scenarios with short bursts of access, such as flash sales, some low-frequency data may be mistakenly eliminated due to the small number of accesses within a short period of time. On the other hand, traditional LFU algorithms also have difficulty identifying the importance of data that has been low-frequency for a long time but has recently become high-frequency, such as periodic hotspot data, and thus cannot effectively retain it.

[0005] Finally, traditional LFU algorithms also suffer from the problem of fragmented processing of multi-dimensional features. They rely solely on the single dimension of access frequency to determine data eviction, ignoring other important characteristics of data access. For example, multi-dimensional features such as spatial locality of data access (such as continuous access to adjacent data blocks) and access latency sensitivity (such as prioritizing the retention of real-time data) are not fully considered. This results in a lack of flexibility in eviction strategies, making it difficult to meet complex and changing cache requirements.

[0006] In healthcare, systems like hospital information systems (HIS) and electronic medical records (EMRs) generate massive amounts of data. For example, large hospitals generate millions of patient data records daily, including medical histories, test results, and medical orders. Traditional LFU algorithms cache this data by assigning independent counters to each data item, leading to linear expansion of frequency storage. For example, in a large medical database containing tens of millions of patient records, each counter occupies 8 bytes, requiring tens or even hundreds of GB of memory for frequency storage alone. This reduces cache space utilization, impacts system performance, lengthens patient data query response times, and reduces medical efficiency. Access patterns for healthcare data are complex. During peak influenza season, access to influenza-related test items and medication information increases dramatically in the short term. Traditional LFU algorithms, due to their fixed-window frequency statistics, can easily eliminate low-frequency but important data. Chronic disease patient data, which has a long-term low frequency but recently high frequency access (such as regular imaging follow-up data), is also difficult for traditional LFU algorithms to effectively identify. This can prevent doctors from timely accessing this data, impacting the continuity of medical diagnosis and treatment. Furthermore, medical data access is not only frequency-dependent but also involves multi-dimensional characteristics such as spatial locality and access latency sensitivity. In radiology departments, doctors review patient imaging data by typically viewing a series of adjacent image slices continuously (spatial locality). Traditional LFU algorithms fail to account for this spatial locality, which can result in frequent replacement of adjacent image slices and constant waiting for data to load. In emergency scenarios, patient data is latency-sensitive and requires priority preservation. However, traditional LFU algorithms rely solely on access frequency and cannot effectively distinguish latency-sensitive data, potentially impacting emergency response efficiency.

[0007] Traditional LFU algorithms face numerous challenges in fintech businesses. First, assigning independent counters to each data item leads to linear expansion of frequency storage, consuming significant memory and reducing cache space utilization. This in turn slows transaction query and processing, impacting the real-time and efficiency of financial services. Second, fintech businesses have complex and volatile access patterns, including short-term bursts of access and long periods of low frequency followed by recent high frequency. Traditional LFU algorithms, due to their fixed-window statistical frequency limitations, struggle to effectively identify and process this data, potentially preventing investors from obtaining timely access to important information and impacting investment decisions. Furthermore, financial data access involves multidimensional characteristics, such as spatial locality in transaction risk monitoring systems and the latency sensitivity of high-frequency trading. Traditional LFU algorithms fail to account for these factors, potentially leading to frequent replacement of relevant data or failure to prioritize retention, impacting the continuity of risk monitoring and the efficiency and accuracy of high-frequency trading. Summary of the Invention

[0008] The main purpose of the present invention is to provide a cache optimization method, device, equipment and storage medium based on frequency and weight, aiming to solve the problems of low frequency storage efficiency, static frequency characteristics, fragmented multi-dimensional feature processing and insufficient flexibility of elimination strategy in traditional LFU cache algorithm.

[0009] To achieve the above object, the present invention provides a cache optimization method based on frequency and weight, comprising:

[0010] Obtaining access data items, and assigning the access data items to clusters according to access pattern characteristics;

[0011] Calculating a hash fingerprint of the accessed data item, determining a cluster location of the accessed data according to the Bloom filter and the hash fingerprint, and updating a global counter of the corresponding cluster according to the cluster location;

[0012] Obtaining multi-dimensional features of the access data item and its time series sequence, inputting the time series sequence into a recurrent neural network to generate a weighted three-dimensional vector;

[0013] Building an index based on the access data item, the cluster to which the access data item belongs, the hash fingerprint, and the weight three-dimensional vector, and determining a comprehensive elimination strategy through the index;

[0014] Determine a cache optimization strategy based on the global counter, the weight three-dimensional vector and the comprehensive elimination strategy;

[0015] Cache data according to the cache optimization strategy.

[0016] Furthermore, to achieve the above-mentioned purpose, the present invention provides a cache optimization device based on frequency and weight, comprising:

[0017] a data allocation module, configured to obtain access data items and allocate the access data items to clusters according to access pattern characteristics;

[0018] a data location module, configured to calculate a hash fingerprint of the accessed data item, determine a cluster location of the accessed data based on the Bloom filter and the hash fingerprint, and update a global counter of the corresponding cluster based on the cluster location;

[0019] A data weight module is used to obtain the multi-dimensional features of the access data item and its time series sequence, input the time series sequence into a recurrent neural network, and generate a weight three-dimensional vector;

[0020] A data indexing module is used to construct an index based on the access data item, the cluster to which the access data item belongs, the hash fingerprint and the weight three-dimensional vector, and determine a comprehensive elimination strategy through the index;

[0021] A cache optimization strategy module, configured to determine a cache optimization strategy based on the global counter, the weight three-dimensional vector, and the comprehensive elimination strategy;

[0022] The data cache module is used to cache data according to the cache optimization strategy.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a frequency and weight-based cache optimization program stored in the memory and runnable on the processor. When the frequency and weight-based cache optimization program is executed by the processor, the steps of the frequency and weight-based cache optimization method as described above are implemented.

[0024] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a frequency- and weight-based cache optimization program is stored. When the frequency- and weight-based cache optimization program is executed by a processor, the steps of the frequency- and weight-based cache optimization method as described above are implemented.

[0025] Beneficial effects: The present invention relates to the field of data processing technology and can be applied to business system platforms such as medical health and financial technology. It discloses a cache optimization method based on frequency and weight, including: obtaining access data items, and allocating the access data items to clusters according to access pattern characteristics; calculating the hash fingerprint of the access data item, determining the cluster position of the accessed data according to the Bloom filter and the hash fingerprint, and updating the global counter of the cluster to which it belongs according to the cluster position; obtaining the multi-dimensional characteristics of the access data item and its time series sequence, inputting the time series sequence into a recurrent neural network to generate a weighted three-dimensional vector; constructing an index according to the access data item, the cluster to which the access data item belongs, the hash fingerprint and the weighted three-dimensional vector, and determining a comprehensive elimination strategy through the index; determining a cache optimization strategy according to the global counter, the weighted three-dimensional vector and the comprehensive elimination strategy; caching data according to the cache optimization strategy. The present invention uses a frequency fingerprint compression technology based on cluster hashing, utilizes a clustering algorithm to cluster data items, generates a hash fingerprint and combines a Bloom filter to quickly locate clusters, thereby achieving efficient frequency update and storage, with a compression rate exceeding 90%. At the same time, a dynamic weight prediction model based on a time-series neural network is introduced to predict data item weights by integrating multi-dimensional features, improving hotspot identification and responsiveness. Furthermore, a hierarchical index structure is designed to integrate frequency and weight information to quickly determine elimination strategies, significantly improving cache performance and resource utilization efficiency, and adapting to various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0027] Figure 1 A schematic diagram of an application environment of a cache optimization method based on frequency and weight according to an embodiment of the present invention;

[0028] Figure 2 This is a flow chart of an embodiment of a cache optimization method based on frequency and weight according to the present invention;

[0029] Figure 3 Schematic diagram of functional modules of a preferred embodiment of a cache optimization device based on frequency and weight according to the present invention;

[0030] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0031] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] The cache optimization method based on frequency and weight provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can obtain access data items through the user terminal, and assign the access data items to clusters according to access pattern characteristics; calculate the hash fingerprint of the access data item, determine the cluster position of the accessed data according to the Bloom filter and the hash fingerprint, and update the global counter of the cluster to which it belongs according to the cluster position; obtain the multi-dimensional features of the access data item and its time series sequence, input the time series sequence into a recurrent neural network, and generate a weight three-dimensional vector; construct an index according to the access data item, the cluster to which the access data item belongs, the hash fingerprint and the weight three-dimensional vector, and determine the comprehensive elimination strategy through the index; determine the cache optimization strategy according to the global counter, the weight three-dimensional vector and the comprehensive elimination strategy; cache data according to the cache optimization strategy. The present invention uses a frequency fingerprint compression technology based on cluster hashing, clusters the data items using a clustering algorithm, generates a hash fingerprint and combines the Bloom filter to quickly locate the cluster, thereby achieving efficient frequency update and storage, and a compression rate of over 90%. At the same time, a dynamic weight prediction model based on a time series neural network is introduced to predict the weight of the data item based on multi-dimensional features, thereby improving the hotspot identification and response capabilities. Furthermore, a hierarchical index structure is designed to integrate frequency and weight information, rapidly determining culling strategies, significantly improving cache performance and resource utilization efficiency, and adapting to a variety of application scenarios. The user end includes, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server end can be implemented as a standalone server or a server cluster consisting of multiple servers. The present invention is described in detail below using specific embodiments.

[0034] See also Figure 2 , Figure 2This is a flow chart of an embodiment of a frequency and weight-based cache optimization method provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0035] like Figure 2 As shown, the cache optimization method based on frequency and weight proposed by the present invention includes the following steps:

[0036] S100, obtaining access data items, and assigning the access data items to clusters according to access pattern characteristics;

[0037] S200, calculating a hash fingerprint of the accessed data item, determining a cluster location of the accessed data according to the Bloom filter and the hash fingerprint, and updating a global counter of the corresponding cluster according to the cluster location;

[0038] S300, obtaining the multi-dimensional features of the access data item and its time series sequence, inputting the time series sequence into a recurrent neural network, and generating a weighted three-dimensional vector;

[0039] S400, constructing an index based on the access data item, the cluster to which the access data item belongs, the hash fingerprint, and the weight three-dimensional vector, and determining a comprehensive elimination strategy through the index;

[0040] S500, determining a cache optimization strategy based on the global counter, the weight three-dimensional vector, and the comprehensive elimination strategy;

[0041] S600: Cache data according to the cache optimization strategy.

[0042] In this embodiment, access data items are extracted from the cache system's access log. These data items record the user or system's access behavior to cached data. By analyzing access pattern characteristics, such as access time intervals and associated data blocks, the data items are divided into several clusters using the Density Peak Clustering (DPC) algorithm. During the clustering process, the local density and relative distance of each data item are calculated. Data items with high local density and large relative distance are selected as cluster centers, and other data items are assigned to the cluster with the nearest cluster center. Each cluster is assigned a global frequency counter to record the total access frequency of all data items within the cluster.

[0043] A hash fingerprint is calculated for each accessed data item, which contains information such as cluster ID and offset within the cluster. An efficient hash algorithm (such as MurmurHash or CityHash) is used to generate the hash fingerprint. Based on the expected number of data items and the allowed false positive rate, the size of the Bloom filter and the number of hash functions are determined, and the hash fingerprint of each accessed data item is inserted into the Bloom filter. When a data item is accessed, the system quickly determines whether the data item exists through the Bloom filter. If the Bloom filter returns a "presence" signal, the system determines the cluster to which the data item belongs based on the cluster ID in the hash fingerprint and only updates the global counter of the cluster. In addition, the individual frequency offsets within the cluster are periodically synchronized through asynchronous threads to reduce the overhead of real-time updates.

[0044] Multi-dimensional features of access data items are extracted from cache access logs, including historical access frequency, time interval, spatial proximity, and business priority. These features are then arranged in chronological order to construct a time series. An attention-enhanced long short-term memory network (AttLSTM) is constructed, and a training set is constructed using cache access logs. Model parameters are updated every 10 minutes using incremental learning algorithms (such as SGD Momentum). The time series is input into a recurrent neural network, which outputs a three-dimensional vector containing "recent frequency weights," "historical frequency weights," and "spatial association weights." A dynamic gating mechanism is used to calculate the comprehensive elimination priority in real time, enabling dynamic prediction and adaptive adjustment of frequency weights.

[0045] A first-level index (cluster index) is constructed, creating a hash table based on cluster IDs to store metadata such as the global frequency of the cluster and the number of data items within the cluster. A second-level index (fingerprint index) is constructed, creating a skip list based on the fingerprint of the data item. This skip list links the cluster index with the individual offset, supporting fast O(logN) queries. A third-level index (weight index) is constructed, creating a min-heap based on the predicted weights. During eviction, the top element is directly extracted, reducing eviction decision latency to microseconds. Using this three-level index structure, a comprehensive eviction priority is determined based on the three-dimensional weight vector, prioritizing eviction of the lowest-weighted data items while ensuring that higher-weighted data items remain in the cache. Combining a global counter with the three-dimensional weight vector, the comprehensive eviction priority of each data item is calculated in real time, and the cached data items are dynamically adjusted based on this priority. A training set is constructed using cache access logs, and model parameters are updated every 10 minutes using an incremental learning algorithm to adapt to real-time hotspot changes.

[0046] Based on the cache optimization strategy, cached data items are adjusted, prioritizing those with higher weights and eliminating those with the lowest weights. When a data item is accessed, the cached data is updated based on the item's hash fingerprint and the global counter of the cluster to which it belongs. Asynchronous threads regularly synchronize individual frequency offsets within the cluster to ensure the accuracy of frequency information. A hierarchical index structure and dynamic weight prediction model improve the performance and responsiveness of the cache system, ensuring efficient operation in high-concurrency scenarios while reducing query latency and update overhead.

[0047] Through the implementation details described above, the entire technical solution implements a complete process from access data item acquisition, cluster assignment, hash fingerprint calculation, Bloom filter application, multi-dimensional feature extraction, recurrent neural network prediction, hierarchical index construction, to cache optimization strategy. This process not only improves the storage and query efficiency of the cache system, but also enhances the intelligence and adaptability of the cache system through dynamic weight prediction and comprehensive elimination strategies.

[0048] In the healthcare sector, hospitals need to quickly access and update patients' electronic medical records to provide efficient medical services. This requires a high-performance cache system with high data consistency. To this end, we cluster and compress electronic medical record data to reduce storage space and improve data query efficiency. By predicting the access frequency and importance of medical record data, we dynamically adjust the cache strategy to ensure the real-time and accuracy of critical medical record data. In addition, by building a hierarchical index, we can quickly query and update medical record data, further improving the responsiveness of the electronic medical record system. This series of measures helps improve the overall performance and data consistency of the electronic medical record system, thereby enhancing the efficiency of medical services.

[0049] In the fintech sector, high-frequency trading requires systems to rapidly process large volumes of market data and trade requests, placing extremely high demands on the responsiveness and accuracy of cache systems. To address this, we employ a clustering hashing algorithm to allocate market data items into distinct clusters, reducing storage space and improving data query efficiency. We leverage a recurrent neural network (RNN) to predict the access frequency and importance of data items and dynamically adjust the cached data to ensure the real-time and accuracy of high-frequency trading data. By constructing a hierarchical index, we can quickly locate and update data items, reducing query latency and further improving the responsiveness of the trading system. These measures help increase the processing speed of high-frequency trading systems, reduce transaction latency, and ultimately improve transaction success rates.

[0050] In one embodiment, the step S100 includes:

[0051] S101, obtaining access data, and extracting access data items from the access data;

[0052] S102, obtaining access pattern characteristics of each access to a data item;

[0053] S103, performing a correlation analysis on the access data items according to the access pattern characteristics to obtain data item correlation;

[0054] S104, calculating the local density and relative distance of the access data items;

[0055] S105, determining a cluster center according to the local density and relative distance;

[0056] S106 : Allocate the access data items to clusters through cluster centers.

[0057] In this embodiment, the time difference between each access to a data item is first recorded. For example, for a video stream data item, the time interval between users viewing different segments is recorded to determine the periodicity or continuity of access. Next, the correlation between the data items is analyzed. In a file system, if multiple data items belong to different parts of the same file, or in a database, they belong to different records in the same table, then they are associated. This correlation analysis can reveal similarities in data item access patterns.

[0058] Next, the local density and relative distance of each data item are calculated. Local density is determined by counting the number of data items within a certain distance around the data item, while relative distance is the minimum distance between the data item and other high-density data items. For example, in two-dimensional space, if data item A is surrounded by five data items and the closest high-density data item B is 10 units away, then A's local density is 5 and its relative distance is 10.

[0059] Based on local density and relative distance, the cluster center is determined. Data items with high local density and large relative distance are usually selected as cluster centers. For example, if the local density of data item C is 10 and the relative distance is 20, while the local density of data item D is 8 and the relative distance is 15, then data item C is more likely to be selected as the cluster center. Afterwards, the data items are assigned to different clusters based on their distance from the cluster center. Each cluster is assigned a global frequency counter that records the sum of the access frequencies of all data items in the cluster. For example, if data item E is closest to cluster center C, then E is assigned to the cluster centered on C, and its access frequency is recorded in the global frequency counter of the cluster.

[0060] For example, in the fintech business sector, financial institutions need to quickly access and update customer information to provide personalized services. To this end, access data is obtained from the customer relationship management system (CRM), and access pattern characteristics of customer data are extracted, such as customer access time intervals, transaction frequency, and preferences. By performing correlation analysis on customer data, customer groups with similar behavior patterns are identified, for example, identifying customer groups that frequently conduct online banking transactions. Next, the local density and relative distance of each customer data are calculated to determine the cluster center. For example, customers with high transaction frequency and similar preferences are used as cluster centers. The customer data is then assigned to different clusters, each representing a customer behavior pattern. For example, one cluster may contain high-value customers, while another may contain ordinary customers. Using clustering algorithms, the needs of different customer groups can be quickly identified, thereby optimizing the CRM system's caching strategy and improving customer service quality.

[0061] In the healthcare sector, medical imaging systems need to rapidly process and store large amounts of image data, placing high demands on data storage efficiency and query speed. To this end, access data is collected from the medical imaging system and access pattern characteristics, such as image access interval, image type, and image size, are extracted. Correlation analysis is performed on the image data to identify data items with similar image patterns. For example, image types frequently accessed within the same time period are identified. Next, the local density and relative distance of each image data item are calculated to determine cluster centers. For example, cluster centers are selected for images with high access frequency and similar types. The image data is then assigned to different clusters, each representing a different image pattern. For example, one cluster might contain frequently accessed X-ray images, while another might contain less frequently accessed MRI images. Using clustering algorithms, frequently accessed image data can be quickly identified, thereby optimizing caching strategies, reducing query latency, and improving the responsiveness of the imaging system.

[0062] In one embodiment, step S200 includes:

[0063] S201, calculating a hash fingerprint of each access data item using a hash algorithm;

[0064] S202, obtaining the size of the Bloom filter and the number of hash functions according to the expected data items and the false positive rate;

[0065] S203, inserting the hash fingerprint of each access data item into the Bloom filter through the hash function;

[0066] S204: The Bloom filter determines the cluster location of the accessed data item according to the hash function and the hash fingerprint, and updates the global counter of the corresponding cluster according to the cluster location.

[0067] In this embodiment, we achieve efficient data management and query by generating a hash fingerprint for each data item, including information such as the cluster ID and cluster offset. The hash fingerprint concatenates the cluster ID and cluster offset into a single binary string. For example, cluster ID "00001" and cluster offset "00010" are concatenated into "0000100010." To generate high-quality hash fingerprints, we choose efficient hash algorithms such as MurmurHash or CityHash, which can generate hash values ​​in a short time and reduce the probability of hash collisions.

[0068] The size of the Bloom filter and the number of hash functions are determined based on the expected number of data items and the allowable false positive rate. For example, if we expect 1 million data items and a false positive rate of 0.1%, we can calculate the size of the Bloom filter and the number of hash functions. The size of the Bloom filter is proportional to the number of data items and the false positive rate, while the number of hash functions needs to be adjusted based on the Bloom filter size and the distribution of data items. The hash fingerprint of each data item is inserted into the Bloom filter. The Bloom filter uses multiple hash functions to map the hash fingerprint into a bit array, marking the presence of the data item. For example, the hash fingerprint "0000100010" might be mapped to bits 5, 10, and 15 of the bit array, setting the values ​​of these positions to 1.

[0069] When querying the cluster to which a data item belongs, a Bloom filter can quickly determine whether the data item exists. If the Bloom filter returns a positive result, the cluster ID in the hash fingerprint is used to locate the specific cluster. However, it is important to note that Bloom filters can misjudge a data item, misidentifying it as existing when it doesn't exist. Therefore, in practice, adjustments and optimizations should be made based on the false positive rate.

[0070] When a data item is accessed, the system first uses a Bloom filter and hash fingerprint to quickly locate the cluster to which the data item belongs. It then updates only the global frequency counter for that cluster, not the counters for each data item individually. For example, if data item F belongs to cluster G, the system increments only the global frequency counter for cluster G. This reduces counter update overhead in large-scale data scenarios. Simultaneously, an asynchronous thread is initiated to periodically synchronize individual frequency offsets within the cluster. The synchronization frequency can be adjusted based on actual needs, such as every 10 minutes. The purpose of synchronization is to ensure that the frequency offset of each data item within the cluster is consistent with the global frequency counter, reducing errors. During synchronization, the asynchronous thread iterates over the data items within the cluster, calculates the actual access frequency, and updates the frequency offset. For example, if data item I in cluster H has been accessed five times since the last synchronization, and the global frequency counter for cluster H has increased 20 times, the frequency offset of data item I needs to be updated. This approach reduces real-time update overhead and ensures accurate frequency calculation.

[0071] For example, in the fintech sector, high-frequency trading systems need to rapidly process large volumes of market data and trade requests, placing extremely high demands on data real-time and accuracy. To this end, the system uses a hash algorithm to generate a hash fingerprint for each trade request. This hash fingerprint contains unique identifying information for the trade request, such as the trade ID and timestamp. The size of the Bloom filter and the number of hash functions are calculated based on the expected volume of trade data and an acceptable false positive rate. For example, if the system processes 100,000 trades per second and the false positive rate is set at 0.1%, the size of the Bloom filter and the number of hash functions can be calculated. The hash fingerprint of each trade request is inserted into the Bloom filter using a hash function. The Bloom filter is used to quickly determine whether a trade request already exists, thereby reducing duplicate processing. When a trade request is accessed, the Bloom filter uses the hash function and hash fingerprint to quickly determine whether the trade request exists. If so, the cluster ID in the hash fingerprint is used to determine the cluster to which the trade request belongs, and a global counter for that cluster is updated. The global counter records the access frequency of each cluster, helping the system dynamically adjust its caching strategy.

[0072] In the healthcare sector, telemedicine systems require real-time transmission and processing of patient medical data, placing high demands on data real-time performance and accuracy. To this end, the system uses a hash algorithm to generate a hash fingerprint for each medical data item. This hash fingerprint contains unique identifying information for the medical data, such as the data ID and transmission timestamp. The size of the Bloom filter and the number of hash functions are calculated based on the expected volume of medical data and an acceptable false positive rate. For example, if the system processes 100,000 data transmissions per hour and the false positive rate is set at 0.1%, the size of the Bloom filter and the number of hash functions can be calculated. The hash fingerprint of each medical data item is inserted into the Bloom filter using a hash function. The Bloom filter quickly determines whether the medical data item already exists, thereby reducing duplicate processing. When a medical data item is accessed, the Bloom filter uses the hash function and hash fingerprint to quickly determine whether the medical data item exists. If so, the cluster ID in the hash fingerprint is used to determine the cluster to which the medical data item belongs, and a global counter for that cluster is updated. The global counter records the access frequency of each cluster, helping the system dynamically adjust its caching strategy. Through Bloom filters and hash fingerprints, the existence of medical data items can be quickly determined, unnecessary queries and processing can be reduced, and the system's response speed and processing efficiency can be improved.

[0073] In one embodiment, step S300 includes:

[0074] S301: Obtain cache access logs, extract multi-dimensional features of the cache access logs, and construct the multi-dimensional features into training samples;

[0075] S302, inputting the training samples into a learning model for training to obtain a recurrent neural network;

[0076] S303: Analyze the access data item to obtain access frequency characteristics, time interval characteristics, spatial proximity characteristics, and service priority characteristics of the access data item;

[0077] S304, sorting the access frequency feature, time interval feature, spatial proximity feature, and service priority feature in chronological order to generate a time series sequence;

[0078] S305 inputs the time series into the recurrent neural network to generate a weighted three-dimensional vector.

[0079] In this embodiment, a dynamic weight prediction model based on a temporal neural network is constructed to optimize the elimination decision of the cache system, so as to solve the defect of static frequency characteristics in the traditional LFU algorithm and improve the adaptability of the cache system to changes in data access patterns and the accuracy of elimination decisions.

[0080] First, a training set is constructed using cache access logs. Cache access logs record in detail information such as the access time, access frequency, and access pattern of each data item. Multi-dimensional features are extracted from these logs to construct time series sequences as training samples. Specific features include access frequencies in the past hour, day, and week, as well as time intervals, spatial proximity, and business priority. For example, for a data item, its time series sequence may contain the following information: the access frequency in the past hour was 10 times, the access frequency in the past day was 50 times, and the access frequency in the past week was 200 times; the time interval between the most recent access and the previous access was 5 minutes; the spatial offset from the adjacent data item was 1 data block; and the business priority was high. These feature values ​​are arranged in chronological order to form a time series sequence, which serves as the input data.

[0081] Use incremental learning algorithms (such as SGD Momentum) for model training. These algorithms continuously update model parameters based on existing training data to adapt to new data. For example, every 10 minutes, new data is extracted from cache access logs to update the training set, and then the SGD Momentum algorithm is used to update the model parameters. This ensures that the model can capture changes in data access patterns in real time, improving its adaptability and accuracy.

[0082] Build a long short-term memory (LSTM) network that incorporates an attention mechanism. LSTM is a special type of recurrent neural network (RNN) that effectively handles long-term dependencies in time series data. The attention mechanism can enhance the model's focus on important features, improving prediction accuracy. Specifically, the network structure consists of multiple LSTM layers, with each LSTM unit processing feature information at a different time scale. Including an attention layer between each LSTM layer enables the model to dynamically focus on the importance of different time series features. The network's input is defined as the multi-dimensional feature time series sequence constructed above, and its output is a three-dimensional vector containing "recent frequency weight," "historical frequency weight," and "spatial correlation weight." For example, for a data item, the network predicts a recent frequency weight of 0.6, a historical frequency weight of 0.3, and a spatial correlation weight of 0.1. These weights reflect the importance of the data item across different dimensions and provide a basis for subsequent elimination decisions.

[0083] The comprehensive elimination priority is calculated in real time through a dynamic gating mechanism. The dynamic gating mechanism can dynamically adjust the contribution of each weight dimension based on the current system state and data access pattern. For example, if the system is currently in a high-concurrency state and has high real-time requirements, the dynamic gating mechanism may increase the contribution of recent frequency weights and reduce the contributions of historical frequency weights and spatial correlation weights. Based on the adjusted weight values, the comprehensive elimination priority of each data item is calculated, and the data items with the lowest comprehensive elimination priority are eliminated first.

[0084] Dynamic prediction and adaptive adjustment of frequency weights are achieved. The model continuously updates predictions based on real-time cache access logs and dynamically adjusts weights. For example, if the access frequency of a data item suddenly increases, the model will promptly capture this change and increase the recent frequency weight of the data item, thereby reducing its likelihood of eviction. This adaptive adjustment mechanism ensures that the cache system always makes optimal eviction decisions based on the latest data access patterns.

[0085] In the field of e-commerce, by analyzing user access logs to product pages, features such as access frequency and time intervals can be extracted. For example, if a user frequently browses a certain category of products between 8 and 10 p.m., the system can recommend relevant popular products to the user during this time period. In addition, based on the spatial proximity characteristics of access data items, that is, the correlation between the categories of products browsed by users, the system can predict which products are likely to be purchased at the same time. For example, when users frequently browse electronic products and accessories together, the system can group these products together in advance, which not only makes it easier for users to purchase but also optimizes inventory layout. Through these measures, the accuracy and personalization of product recommendations can be improved, thereby increasing users' willingness to buy. It also helps to optimize inventory management, reduce inventory backlogs, and improve capital turnover.

[0086] In the healthcare sector, we analyze doctors' access logs to medical images to extract features such as access frequency and time intervals. For example, for patients requiring frequent follow-up examinations, the system can automatically prioritize and cache relevant imaging data, facilitating quick access by doctors. Furthermore, we prioritize patient requests for examinations, treatments, and other procedures based on their business priorities. For example, for emergency patients, the system can prioritize the allocation of medical resources to ensure timely treatment. These measures can improve the efficiency of access to medical images, reduce doctors' workloads, and help optimize the allocation of medical resources, thereby improving the fairness and efficiency of medical services.

[0087] In one embodiment, step S400 includes:

[0088] S401, constructing a primary index based on the access data item and the related information of the cluster to which it belongs;

[0089] S402: Construct a secondary index based on the access data item and the hash fingerprint;

[0090] S403, constructing a three-level index according to the access data item and the weight three-dimensional vector;

[0091] S404: Determine a comprehensive elimination strategy based on the primary index, the secondary index, and the tertiary index.

[0092] In this embodiment, a primary index (cluster index) is created, and a hash table is created, whose key is the cluster ID and the value is a structure or object containing cluster metadata. During the data clustering and grouping stage, each cluster is assigned a unique cluster ID as the key of the hash table. The hash table is used to store cluster-related information, such as the global frequency (recording the current value of the global frequency counter of each cluster, updated when the data item is accessed, and synchronized regularly through an asynchronous thread), the number of data items (recording the number of data items contained in each cluster, updated when the data item is added to or removed from the cluster), and other metadata can also be stored as needed, such as the creation time of the cluster (used to analyze the life cycle of the cluster), the last update time (used to quickly determine whether the cluster is active), etc.

[0093] Create a secondary index (fingerprint index) and a skip list to store the fingerprint information of the data item. The skip list is a dynamic data structure based on a linked list that supports fast search operations. The hash fingerprint of each data item is inserted into the skip list. The hash fingerprint contains information such as the cluster ID and the offset within the cluster. When inserted, it is sorted by the value of the hash fingerprint to ensure the order of the skip list. Each node in the skip list not only stores the hash fingerprint, but also stores a pointer to the cluster index and the offset within the cluster. This allows the cluster index to be quickly located through the skip list, and then the global frequency and other metadata of the cluster can be obtained, achieving O(logN) level fast query, where N is the number of nodes in the skip list. When a data item is accessed, the fingerprint node of the data item is quickly located through the skip list, and then the cluster index is accessed through the cluster ID to update the global frequency counter. This structure greatly improves the efficiency of updates and queries.

[0094] Create a three-level index (weight index) and a minimum heap to store the predicted weight value of each data item. The minimum heap is a special binary tree in which the value of each parent node is less than or equal to the value of its child node. The predicted weight value of each data item is inserted into the minimum heap. The weight value is generated by the dynamic weight prediction model (AttLSTM) and contains comprehensive values ​​of dimensions such as "recent frequency weight", "historical frequency weight", and "spatial correlation weight". In the minimum heap, the data item with the smallest weight value is always at the top of the heap, which can quickly determine the least important data item as a candidate for elimination, thereby reducing the elimination decision delay to microseconds. When a data item needs to be eliminated, the top element of the minimum heap is directly extracted. Due to the structural characteristics of the minimum heap, the time complexity of this operation is O(1), which greatly reduces the delay of the elimination decision. When the weight value of a data item changes (for example, the data item is accessed and the weight value is updated), the minimum heap needs to be adjusted. The time complexity of the adjustment operation is O(logN), where N is the number of nodes in the minimum heap. This dynamic update mechanism ensures that the minimum heap always reflects the weight status of the current data item.

[0095] When a data item is accessed, the fingerprint node of the data item is first quickly located through the skip list (secondary index) to obtain the cluster ID and the offset within the cluster. Then, the cluster index (primary index) is accessed through the cluster ID to update the global frequency counter of the cluster. Then, based on the output of the dynamic weight prediction model, the weight value of the data item is updated, and the minimum heap (third-level index) is adjusted. When a data item needs to be eliminated, the top element of the heap is directly extracted from the minimum heap (third-level index) to obtain the data item with the lowest current weight value. Due to the efficient structure of the minimum heap, the elimination decision can be completed in microseconds, greatly improving the response speed and performance of the cache system. Through the above implementation details, the frequency weight fusion structure based on hierarchical index can efficiently store and query frequency information, while quickly responding to elimination decisions, significantly improving the performance and efficiency of the cache system.

[0096] For example, in the management of drug R&D data in the healthcare sector, access data items include key information such as a drug's chemical structure, pharmacological effects, and clinical trial stage. The three-dimensional weight vector is determined based on factors such as the different stages of drug R&D (e.g., drug discovery, preclinical research, clinical trials), as well as the drug's potential value. After constructing a three-level index, drug R&D data can be sorted and filtered based on the three-dimensional weight vector. For example, during the drug screening phase, the weight vector can be used to quickly identify the most promising drug candidate molecules and prioritize their subsequent research and development. This not only helps improve drug R&D efficiency but also rationally allocates R&D resources, accelerating the development of new drugs. Furthermore, a comprehensive elimination strategy is developed by comprehensively considering the primary index (e.g., the disease category to which the medical record belongs), the secondary index (e.g., the hash fingerprint of the image), and the tertiary index (e.g., the importance weight vector of the medical record or imaging data). For example, early medical records and imaging data of patients with common and stable conditions, given their relatively low importance, can be marked as eligible for elimination through the comprehensive elimination strategy. Data from patients with rare diseases or complex, rapidly evolving conditions, on the other hand, are prioritized for retention. This can not only ensure the integrity and availability of important data, but also effectively manage storage space and ensure the efficient operation of the medical data storage system.

[0097] In financial portfolio management within the FinTech sector, accessed data items include information such as the name, industry classification, and yield rate of investment targets (such as stocks, bonds, and funds). The three-dimensional weight vector is determined based on factors such as the target's risk level, expected return, and investment period. After constructing a three-level index, the three-dimensional weight vector can be used to sort and filter assets within the portfolio. For example, during asset allocation, the weight vector can be used to quickly identify investment targets that meet the investment strategy and risk profile, thereby optimizing the portfolio. This helps improve the scientific and accurate nature of investment decisions, maximizing investment returns and minimizing risks. Furthermore, a comprehensive elimination strategy is developed by comprehensively considering primary indexes (such as the market sector to which transaction data belongs), secondary indexes (such as the hash fingerprint of risk assessment reports), and tertiary indexes (such as the weight vector of the importance of assets in the portfolio). For example, outdated transaction records and risk assessment reports with little relevance to current market conditions can be marked as eligible for elimination through this comprehensive elimination strategy, while important customer transaction records and key investment decision data are prioritized for retention. This approach ensures the integrity and availability of important data while effectively managing storage space and ensuring the efficient operation of financial institutions' data storage systems.

[0098] In one embodiment, step S204 includes:

[0099] S2041. Determine whether the accessed data item exists through the Bloom filter;

[0100] S2042: When the Bloom filter returns an existence signal, determine the cluster location of the accessed data item according to the hash fingerprint;

[0101] S2043: When the accessed data item is accessed, update the global counter according to the cluster position.

[0102] This embodiment achieves efficient data access and management through Bloom filters and hash fingerprints. A Bloom filter is a space-efficient, probabilistic data structure used to quickly determine whether an element exists in a set. It may falsely identify a nonexistent element as present, but it will not falsely identify an existing element as absent. In a cache system, a Bloom filter is used to quickly determine whether the accessed data item exists in the cache, thereby reducing unnecessary query and computational overhead.

[0103] When a data item is accessed, the system first judges it through the Bloom filter. The Bloom filter uses multiple independent hash functions to map the hash fingerprint of the data item to a bit array. If the bits corresponding to all hash functions are 1, the Bloom filter returns a "present" signal; otherwise, it returns a "not present" signal. For example, suppose the Bloom filter uses three hash functions. The hash fingerprint of data item A is mapped to the 5th bit of the bit array by hash function 1, to the 10th bit of the bit array by hash function 2, and to the 15th bit of the bit array by hash function 3. If the 5th, 10th, and 15th bits of the bit array are all 1, the Bloom filter returns a "present" signal.

[0104] When the Bloom filter returns a "present" signal, the system determines the cluster location of the accessed data item based on the hash fingerprint. The hash fingerprint contains information such as the cluster ID and the cluster offset, which is used to quickly locate the cluster to which the data item belongs. The system uses the cluster ID in the hash fingerprint to search for the corresponding cluster in the primary index (cluster index) and further locates the specific location of the data item within the cluster using the cluster offset. For example, suppose the hash fingerprint is "0000100010", where the first five digits "00001" represent the cluster ID and the last five digits "00010" represent the cluster offset. The system uses the cluster ID "00001" to find the corresponding cluster in the primary index and uses the cluster offset "00010" to locate the specific location of the data item within the cluster. Using the cluster ID and cluster offset of the hash fingerprint, the system can quickly locate the cluster location of the accessed data item, avoiding traversing the entire cache and greatly improving query efficiency.

[0105] When a data item is accessed, the system updates the global counter of the cluster to which it belongs. The global counter records the sum of the access frequencies of all data items in each cluster. By updating the global counter, the system can track the access popularity of the cluster in real time and provide a basis for subsequent elimination decisions. For example, suppose data item A belongs to cluster C. When data item A is accessed, the system adds 1 to the global counter of cluster C. To reduce the real-time update overhead, the system regularly synchronizes the individual frequency offsets within the cluster through asynchronous threads to ensure that the value of the global counter is consistent with the actual access frequency, while avoiding the performance overhead caused by frequent updates. For example, the system can synchronize the individual frequency offsets within the cluster through asynchronous threads once every 10 minutes to ensure the accuracy of the global counter.

[0106] Through the above implementation details, the system uses Bloom filters to quickly determine whether the accessed data item exists, hash fingerprints to quickly locate the cluster to which the data item belongs, and updates global counters to track cluster access popularity in real time. This process not only improves query efficiency but also reduces storage and real-time update overhead, improving the overall performance of the cache system.

[0107] For example, in a distributed medical data storage system in the healthcare sector, a patient's medical records are typically stored in different storage nodes (clusters). When accessing a specific medical record data item is needed, a hash fingerprint can quickly determine the cluster location of the data item. For example, a large medical group has multiple hospitals, and their medical records are stored in a distributed storage system. When medical staff need to access a patient's medical record data, the hash fingerprint can quickly locate the storage node where the data item is located and then retrieve the data from that node. This helps improve the speed and accuracy of data access, especially when the data volume is large and widely distributed. In medical research, researchers may collect large amounts of data, such as genetic data and clinical trial data. This data may be stored in different clusters according to different research projects or data types. When researchers need to access a specific data item (such as a gene sequence data), the hash fingerprint can help quickly determine the cluster location of the data item, making it easier for researchers to quickly obtain the required data and accelerating the progress of medical research.

[0108] In the FinTech (FinTech) business sector, FinTech platforms store vast amounts of financial transaction data, such as stock and fund transaction records. When investors or financial institution staff need to query a specific transaction record, a Bloom filter can quickly determine whether the transaction data item exists. For example, on a stock trading platform, an investor might want to query the transaction record of a particular stock purchased on a specific date. A Bloom filter can first determine whether the transaction data item exists, avoiding unnecessary searches within massive amounts of transaction data and improving query efficiency. This is particularly important for FinTech systems that process high-frequency trading data, ensuring rapid response to user queries. Furthermore, financial institutions need to query large amounts of credit data when assessing customer creditworthiness. Bloom filters can be used to determine whether a customer's credit data item exists. For example, when approving loan applications, banks need to query a customer's credit history at various financial institutions. A Bloom filter can quickly verify the existence of a customer's credit data item, helping banks quickly identify customers with established credit records and thus expediting the loan approval process.

[0109] In one embodiment, the step S500 further includes:

[0110] S501, calculating the frequency storage compression ratio after implementing the cache optimization strategy based on the storage space usage;

[0111] S502, calculating the fingerprint hash collision rate based on the number of collisions of the hash fingerprint and the total number of queries;

[0112] S503, calculating the misjudgment rate based on the number of misjudgments of the Bloom filter and the total number of queries;

[0113] S504: Compare the actual access data with the prediction results of the recurrent neural network and calculate the prediction accuracy;

[0114] S505: Update the cache optimization strategy according to the frequency storage compression rate, fingerprint hash collision rate, misjudgment rate, and prediction accuracy rate.

[0115] In this embodiment, a series of optimization and maintenance processes are used to ensure that the technical solution achieves the expected optimization effect in actual application, thereby improving the performance and reliability of the cache system.

[0116] Evaluate the frequency storage compression ratio to ensure it reaches over 90%. The storage of hundreds of millions of data items is reduced from 80GB to less than 8GB, increasing cache capacity 10-fold under the same memory. Calculate the frequency storage compression ratio by comparing storage space usage before and after optimization. For example, if the storage space before optimization is 80GB and the storage space after optimization is 8GB, the compression ratio is ((808) / 80=0.9), or 90%. Regularly monitor the storage usage of the cache system to ensure that the cache capacity has indeed increased 10-fold under the same memory conditions.

[0117] By monitoring the fingerprint hash collision rate and Bloom filter false positive rate, we ensure that the fingerprint hash collision rate is below 0.1% and the Bloom filter false positive rate is below 0.01% to ensure the accuracy of frequency calculation. The fingerprint hash collision rate is calculated by counting the ratio of the number of hash fingerprint collisions to the total number of queries. For example, if there is one collision in 1,000 queries, the collision rate is 0.1%. The false positive rate is calculated by counting the ratio of the number of Bloom filter false positives to the total number of queries. For example, if there is one false positive in 10,000 queries, the false positive rate is 0.01%.

[0118] Test the detection latency of sudden hotspots to ensure it is less than 1 minute. Check whether the AttLSTM model's prediction accuracy for short-term frequency fluctuations reaches 92%, and whether it can identify hotspots 30 minutes earlier than the traditional LFU algorithm. Simulate sudden hotspot scenarios and record the time from the appearance of a hotspot to its detection by the system to ensure detection latency is less than 1 minute. Compare actual hotspot data with the AttLSTM model's prediction results to calculate prediction accuracy. For example, if the model predicts 100 hotspots and 92 of them are accurate, the prediction accuracy is 92%. Compare the difference in hotspot identification time between the traditional LFU algorithm and the optimized algorithm to ensure that the optimized algorithm can identify hotspots 30 minutes earlier.

[0119] Verify spatial locality utilization to ensure a 40% improvement in spatial locality utilization and an increase in cache hit rate to over 85% in continuous access scenarios. Calculate spatial locality utilization by counting the ratio of the number of times adjacent data items are accessed to the total number of accesses in continuous access scenarios. For example, if 40 out of 100 accesses are to adjacent data items, the spatial locality utilization is 40%. Calculate the cache hit rate by counting the ratio of cache hits to the total number of accesses. For example, if 85 out of 100 accesses are hits, the cache hit rate is 85%.

[0120] Enhanced multi-scenario adaptability is achieved through dynamic adjustments to weight dimensions based on the needs of different scenarios. For example, temporal proximity weighting is increased in video streaming scenarios, while high-priority business data is prioritized in database scenarios. Cache access logs are analyzed to identify the current scenario type. For example, the time interval and data item size in the access pattern are detected to determine whether it is a video streaming scenario. Weight dimensions are dynamically adjusted based on the scenario type. For example, temporal proximity weighting is increased in video streaming scenarios, while business priority weighting is increased in database scenarios.

[0121] To evaluate the improvement in concurrency performance, we tested the throughput of frequency update operations. We also tested whether lock-free cluster-level updates and the hierarchical index structure increased the throughput of frequency update operations from 100,000 to 150,000 per second, a 50% improvement in concurrency. We simulated high-concurrency scenarios and recorded the throughput of frequency update operations. For example, we used a stress testing tool to simulate 1,000 concurrent threads performing frequency update operations and recorded the number of updates completed per second. By comparing the throughput before and after optimization, we ensured that the throughput had increased by 50% after optimization.

[0122] Maintain resource elasticity and monitor whether the number of clusters can be dynamically adjusted. Using an adaptive clustering algorithm, clusters can be automatically split or merged when data distribution changes, avoiding the inefficiencies of fixed grouping. Determine whether data distribution has changed by monitoring the access patterns and distribution of data items. For example, by counting changes in the number of data items and access frequency in each cluster, determine whether the number of clusters needs to be adjusted. Automatically split or merge clusters based on changes in data distribution. For example, if a cluster has too many data items, it can be split into two clusters; if two clusters have too few data items, they can be merged into one cluster.

[0123] Check the compression of model parameters and ensure that they are compressed to less than 1MB using knowledge distillation technology to adapt to resource-constrained scenarios such as edge computing. Use knowledge distillation to compress the AttLSTM model to reduce the size of model parameters. For example, this can be achieved by training a small student model to mimic the behavior of a large teacher model. Check the size of the compressed model parameters to ensure they are within 1MB. Also, evaluate the performance of the compressed model in resource-constrained scenarios to ensure it meets actual application requirements.

[0124] In the healthcare sector, by analyzing storage space usage, the system calculates the frequency storage compression ratio after implementing cache optimization strategies. For example, for frequently accessed medical images, the system can use a higher compression ratio to save storage space while ensuring image access speed. Furthermore, by counting the number of hash fingerprint collisions and the total number of queries, the fingerprint hash collision rate is calculated. For hash fingerprints of medical images, the system can optimize the hash algorithm to reduce collisions, thereby improving retrieval efficiency. Furthermore, by recording the number of false positives and the total number of queries by the Bloom filter, the false positive rate is calculated. When querying patient medical records, the system can use the Bloom filter to quickly determine whether data exists, thereby reducing false positives and improving query efficiency. Furthermore, the system compares actual access data with the predictions of the recurrent neural network to calculate the prediction accuracy. For example, the system can predict which images or medical records are most likely to be accessed in the future and cache these data in advance, thereby improving access efficiency.

[0125] In the FinTech sector, storage space usage is similarly analyzed to calculate the frequency storage compression ratio after implementing cache optimization strategies. For example, for high-frequency trading data, the system can employ a higher compression ratio to save storage space while ensuring data access speed. Furthermore, the fingerprint hash collision rate is calculated by counting the number of hash fingerprint collisions and the total number of queries. For hash fingerprints of financial transaction data, the system can optimize the hash algorithm to reduce collisions, thereby improving retrieval efficiency. Furthermore, the false positive rate is calculated by recording the number of false positives and the total number of queries by the Bloom filter. When querying transaction records, the system can use the Bloom filter to quickly determine whether data exists, thereby reducing false positives and improving query efficiency. Furthermore, the system compares actual access data with the predictions of the recurrent neural network to calculate the prediction accuracy. For example, the system can predict which transaction data is most likely to be accessed in the future and cache this data in advance, thereby improving access efficiency.

[0126] In one embodiment, a cache optimization device based on frequency and weight is provided, and the cache optimization device based on frequency and weight corresponds to the cache optimization method based on frequency and weight in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a frequency and weight-based cache optimization device of the present invention. It includes a data allocation module 10, a data location module 20, a data weight module 30, a data index module 40, a cache optimization strategy module 50, and a data cache module 60. Each functional module is described in detail below:

[0127] A data allocation module 10 is configured to obtain access data items and allocate the access data items to clusters according to access pattern characteristics;

[0128] a data location module 20 for calculating a hash fingerprint of the accessed data item, determining a cluster location of the accessed data according to the Bloom filter and the hash fingerprint, and updating a global counter of the corresponding cluster according to the cluster location;

[0129] A data weight module 30 is configured to obtain the multi-dimensional features of the access data item and its time series sequence, input the time series sequence into a recurrent neural network, and generate a weight three-dimensional vector;

[0130] A data indexing module 40 is configured to construct an index based on the access data item, the cluster to which the access data item belongs, the hash fingerprint, and the weight three-dimensional vector, and determine a comprehensive elimination strategy through the index;

[0131] A cache optimization strategy module 50 is configured to determine a cache optimization strategy based on the global counter, the weight three-dimensional vector, and the comprehensive elimination strategy;

[0132] The data cache module 60 is configured to cache data according to the cache optimization strategy.

[0133] In one embodiment, the data distribution module 10 includes:

[0134] a data acquisition unit, configured to acquire access data and extract access data items from the access data;

[0135] a feature acquisition unit, configured to acquire access pattern features of each access to a data item;

[0136] a correlation unit, configured to perform correlation analysis on the access data items according to the access pattern characteristics to obtain data item correlation;

[0137] a data calculation unit, configured to calculate the local density and relative distance of the accessed data items;

[0138] A cluster center unit, configured to determine a cluster center according to the local density and relative distance;

[0139] The cluster allocation unit is used to allocate the access data items to clusters through cluster centers.

[0140] In one embodiment, the data location module 20 includes:

[0141] A hash fingerprint unit, used to calculate a hash fingerprint of each accessed data item by using a hash algorithm;

[0142] a Bloom filter unit, configured to obtain a size of the Bloom filter and a number of hash functions according to expected data items and a false positive rate;

[0143] a data inserting unit, configured to insert the hash fingerprint of each accessed data item into the Bloom filter by using the hash function;

[0144] A cluster location unit is used for the Bloom filter to determine the cluster location of the accessed data item according to the hash function and the hash fingerprint, and to update the global counter of the corresponding cluster according to the cluster location.

[0145] In one embodiment, the data weight module 30 includes:

[0146] A training sample unit, configured to obtain cache access logs, extract multi-dimensional features of the cache access logs, and construct the multi-dimensional features into training samples;

[0147] A model training unit, configured to input the training samples into a learning model for training to obtain a recurrent neural network;

[0148] a feature acquisition unit, configured to analyze the access data item to acquire an access frequency feature, a time interval feature, a spatial proximity feature, and a service priority feature of the access data item;

[0149] A time series unit, configured to sort the access frequency feature, time interval feature, spatial proximity feature, and service priority feature in chronological order to generate a time series sequence;

[0150] The weight unit is used to input the time series sequence into the recurrent neural network to generate a weight three-dimensional vector.

[0151] In one embodiment, the data index module 40 includes:

[0152] A primary index unit, used to construct a primary index based on the accessed data item and the related information of the cluster to which it belongs;

[0153] A secondary index unit, configured to construct a secondary index based on the access data item and the hash fingerprint;

[0154] A three-level indexing unit, configured to construct a three-level index based on the access data item and the weight three-dimensional vector;

[0155] The comprehensive elimination strategy unit is used to determine the comprehensive elimination strategy according to the primary index, the secondary index and the tertiary index.

[0156] In one embodiment, the cluster location unit comprises:

[0157] Determining whether the accessed data item exists by using the Bloom filter;

[0158] When the Bloom filter returns a presence signal, determining a cluster location of the accessed data item according to the hash fingerprint;

[0159] When the accessed data item is accessed, the global counter is updated according to the cluster position.

[0160] In one embodiment, the cache optimization strategy module 50 further includes:

[0161] A frequency storage compression ratio unit is used to calculate the frequency storage compression ratio after implementing the cache optimization strategy based on storage space usage;

[0162] A fingerprint hash collision rate unit, configured to calculate a fingerprint hash collision rate based on the number of collisions of the hash fingerprint and the total number of queries;

[0163] The false positive rate unit is used to calculate the false positive rate based on the number of false positives and the total number of queries through the Bloom filter;

[0164] The prediction accuracy unit is used to compare the actual access data with the prediction results of the recurrent neural network and calculate the prediction accuracy;

[0165] A strategy updating unit is used to update the cache optimization strategy according to the frequency storage compression rate, fingerprint hash collision rate, misjudgment rate and prediction accuracy rate.

[0166] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a cache optimization method based on frequency and weight.

[0167] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a cache optimization method based on frequency and weight.

[0168] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0169] Obtaining access data items, and assigning the access data items to clusters according to access pattern characteristics;

[0170] Calculating a hash fingerprint of the accessed data item, determining a cluster location of the accessed data according to the Bloom filter and the hash fingerprint, and updating a global counter of the corresponding cluster according to the cluster location;

[0171] Obtaining multi-dimensional features of the access data item and its time series sequence, inputting the time series sequence into a recurrent neural network to generate a weighted three-dimensional vector;

[0172] Building an index based on the access data item, the cluster to which the access data item belongs, the hash fingerprint, and the weight three-dimensional vector, and determining a comprehensive elimination strategy through the index;

[0173] Determine a cache optimization strategy based on the global counter, the weight three-dimensional vector and the comprehensive elimination strategy;

[0174] Cache data according to the cache optimization strategy.

[0175] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0176] Obtaining access data items, and assigning the access data items to clusters according to access pattern characteristics;

[0177] Calculating a hash fingerprint of the accessed data item, determining a cluster location of the accessed data according to the Bloom filter and the hash fingerprint, and updating a global counter of the corresponding cluster according to the cluster location;

[0178] Obtaining multi-dimensional features of the access data item and its time series sequence, inputting the time series sequence into a recurrent neural network to generate a weighted three-dimensional vector;

[0179] Building an index based on the access data item, the cluster to which the access data item belongs, the hash fingerprint, and the weight three-dimensional vector, and determining a comprehensive elimination strategy through the index;

[0180] Determine a cache optimization strategy based on the global counter, the weight three-dimensional vector and the comprehensive elimination strategy;

[0181] Cache data according to the cache optimization strategy.

[0182] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0183] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0184] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0185] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A cache optimization method based on frequency and weight, characterized in that: The following steps are involved: Obtaining access data items, and assigning the access data items to clusters according to access pattern characteristics; Calculating a hash fingerprint of the accessed data item, determining a cluster location of the accessed data according to the Bloom filter and the hash fingerprint, and updating a global counter of the corresponding cluster according to the cluster location; Obtaining multi-dimensional features of the access data item and its time series sequence, inputting the time series sequence into a recurrent neural network to generate a weighted three-dimensional vector; Building an index based on the access data item, the cluster to which the access data item belongs, the hash fingerprint, and the weight three-dimensional vector, and determining a comprehensive elimination strategy through the index; Determine a cache optimization strategy based on the global counter, the weight three-dimensional vector and the comprehensive elimination strategy; Cache data according to the cache optimization strategy.

2. The cache optimization method based on frequency and weight according to claim 1, characterized in that: The obtaining of access data items and allocating the access data items to clusters according to access pattern characteristics includes: Acquiring access data, and extracting access data items from the access data; Obtain access pattern characteristics for each access to a data item; Performing a correlation analysis on the access data items according to the access pattern characteristics to obtain data item correlation; Calculating the local density and relative distance of the access data items; Determining a cluster center according to the local density and relative distance; The access data items are allocated to clusters via cluster centers.

3. The cache optimization method based on frequency and weight according to claim 1, characterized in that: The step of calculating the hash fingerprint of the accessed data item, determining the cluster location of the accessed data according to the Bloom filter and the hash fingerprint, and updating the global counter of the corresponding cluster according to the cluster location, includes: Calculate the hash fingerprint of each accessed data item through a hash algorithm; Obtain the size of the Bloom filter and the number of hash functions based on the expected data items and the false positive rate; inserting the hash fingerprint of each access data item into the Bloom filter through the hash function; The Bloom filter determines the cluster position of the accessed data item according to a hash function and a hash fingerprint, and updates a global counter of the corresponding cluster according to the cluster position.

4. The cache optimization method based on frequency and weight according to claim 1, wherein: The step of obtaining the multi-dimensional features and the time series sequence of the multi-dimensional features of the access data item, inputting the time series sequence into a recurrent neural network, and generating a weighted three-dimensional vector includes: Obtaining cache access logs, extracting multi-dimensional features of the cache access logs, and constructing the multi-dimensional features into training samples; Inputting the training samples into a learning model for training to obtain a recurrent neural network; Analyzing the access data items to obtain access frequency characteristics, time interval characteristics, spatial proximity characteristics, and service priority characteristics of the access data items; Sorting the access frequency feature, time interval feature, spatial proximity feature, and service priority feature in chronological order to generate a time series; The time series sequence is input into the recurrent neural network to generate a weight three-dimensional vector.

5. The cache optimization method based on frequency and weight according to claim 1, wherein: The step of constructing an index based on the access data item, the cluster to which the access data item belongs, the hash fingerprint, and the weight three-dimensional vector, and determining a comprehensive elimination strategy through the index includes: Build a primary index based on the accessed data item and the related information of the cluster to which it belongs; Building a secondary index based on the access data item and the hash fingerprint; Constructing a three-level index based on the access data item and the weight three-dimensional vector; A comprehensive elimination strategy is determined based on the primary index, the secondary index, and the tertiary index.

6. The cache optimization method based on frequency and weight according to claim 3, characterized in that: The Bloom filter determines the cluster location of the accessed data item according to a hash function and a hash fingerprint, and updates a global counter of the corresponding cluster according to the cluster location, including: Determining whether the accessed data item exists by using the Bloom filter; When the Bloom filter returns a presence signal, determining a cluster location of the accessed data item according to the hash fingerprint; When the accessed data item is accessed, the global counter is updated according to the cluster position.

7. The cache optimization method based on frequency and weight according to claim 1, characterized in that: The determining of the cache optimization strategy according to the global counter, the weight three-dimensional vector and the comprehensive elimination strategy further includes: Calculate the frequency storage compression ratio after implementing the cache optimization strategy based on storage space usage; Calculate the fingerprint hash collision rate by the number of collisions of the hash fingerprint and the total number of queries; Calculate the misjudgment rate by the number of misjudgments of the Bloom filter and the total number of queries; Compare the actual access data with the prediction results of the recurrent neural network and calculate the prediction accuracy; The cache optimization strategy is updated according to the frequency storage compression rate, fingerprint hash collision rate, misjudgment rate and prediction accuracy.

8. A cache optimization device based on frequency and weight, characterized in that: The cache optimization device based on frequency and weight includes: a data allocation module, configured to obtain access data items and allocate the access data items to clusters according to access pattern characteristics; a data location module, configured to calculate a hash fingerprint of the accessed data item, determine a cluster location of the accessed data based on the Bloom filter and the hash fingerprint, and update a global counter of the corresponding cluster based on the cluster location; A data weight module is used to obtain the multi-dimensional features of the access data item and its time series sequence, input the time series sequence into a recurrent neural network, and generate a weight three-dimensional vector; A data indexing module is used to construct an index based on the access data item, the cluster to which the access data item belongs, the hash fingerprint and the weight three-dimensional vector, and determine a comprehensive elimination strategy through the index; A cache optimization strategy module, configured to determine a cache optimization strategy based on the global counter, the weight three-dimensional vector, and the comprehensive elimination strategy; The data cache module is used to cache data according to the cache optimization strategy.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a frequency- and weight-based cache optimization program stored in the memory and capable of running on the processor. When the frequency- and weight-based cache optimization program is executed by the processor, the steps of the frequency- and weight-based cache optimization method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a cache optimization program based on frequency and weight. When the cache optimization program based on frequency and weight is executed by a processor, the steps of the cache optimization method based on frequency and weight are implemented as described in any one of claims 1 to 7.

Images