Data elimination method and device, equipment and storage medium

Abstract

The invention discloses a data elimination method and device, equipment and a storage medium, is applied to a cache system, and relates to the technical field of cache management, and the data elimination method comprises the steps of determining a preset number of candidate data items when the usage amount of a cache space of the cache system reaches a preset threshold value; performing weighted calculation on the feature vector of each candidate data item through a preset weighted scoring model to obtain an elimination priority score; the feature vectors are vectors representing access frequency, timeliness, data size, access delay and relevance; the relevance is the ratio of the member number of the data cluster corresponding to the data item to the target number; the target number is the member number of the historical maximum data cluster; and executing a data elimination operation on the candidate data items corresponding to the elimination priority scores meeting a preset score condition. Therefore, the influence of the data cluster to which the data item belongs on the data item is considered, and data value evaluation is performed in combination with the multi-dimensional feature vector, so that the data elimination accuracy can be improved to match actual access requirements.

Description

Technical Field

[0001] This invention relates to the field of cache management technology, and in particular to a data eviction method, apparatus, device, and storage medium. Background Technology

[0002] Caching systems can significantly reduce the access pressure on the backend database and substantially reduce the response latency of data requests, thereby improving the overall performance and user experience of the application. However, memory, as an expensive and limited resource, has a much smaller capacity than backend persistent storage. Therefore, when the cache space is full, an efficient cache eviction policy must be adopted to determine which data items should be removed to make room for new data. Traditional cache eviction algorithms, such as Least Recently Used (LRU) and Least Frequently Used (LFU), are widely used due to their simplicity. The LRU algorithm evicts the data that has not been accessed for the longest time, but its fatal weakness lies in its inability to handle the "cache pollution" problem, that is, a one-time batch data scan or occasional access may squeeze out a large amount of truly valuable hot data from the cache. The LFU algorithm evicts the data with the lowest access frequency, but it suffers from the "cache laziness" problem, that is, data that was accessed very frequently in the past but is no longer popular will occupy the cache for a long time and cannot quickly adapt to changes in access patterns. These traditional algorithms rely on a single dimension (time or frequency) for decision-making, making it difficult to adapt to the complex and ever-changing access patterns in current business scenarios.

[0003] Therefore, improving the accuracy of cached data eviction to match the access requirements of business scenarios is a problem that needs to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a data removal method, apparatus, device, and storage medium that considers the impact of the data cluster to which a data item belongs on the data item itself, and combines multi-dimensional feature vectors for data value assessment, thereby improving the accuracy of data removal to match actual access needs. The specific solution is as follows:

[0005] Firstly, this application provides a data eviction method applied to a caching system, comprising:

[0006] When the usage of the cache space in the cache system reaches a preset threshold, a preset number of candidate data items are determined from the cache space;

[0007] The feature vectors of each candidate data item are weighted and calculated using a pre-set weighted scoring model to obtain the elimination priority score corresponding to each candidate data item; the feature vector is a vector representing access frequency, timeliness, data size, access latency, and correlation; the correlation is the ratio of the number of members in the data cluster corresponding to the data item to the target number; the target number is the number of members in the historical maximum data cluster;

[0008] A data elimination operation is performed on the candidate data items corresponding to the elimination priority scores that meet the preset score conditions.

[0009] Optionally, the cache space includes a dynamic area and a compressed area; the dynamic area is a region configured based on a hash table and a doubly linked list for storing data items that meet preset burst access conditions, and the compressed area is a region configured based on an array and a hash function for storing data items that meet preset hot data conditions.

[0010] Accordingly, determining a preset number of candidate data items from the cache space includes:

[0011] A first data item is obtained by randomly selecting a first preset number of data items from the tail of the doubly linked list corresponding to the dynamic area of ​​the cache space, and a second data item is obtained by randomly selecting a second preset number of data items from the compressed area.

[0012] The first data item and the second data item are determined as candidate data items.

[0013] Optionally, when the usage of the cache space in the caching system reaches a preset threshold, determining a preset number of candidate data items from the cache space includes:

[0014] Collect the current status parameters of the caching system; the current status parameters include processor load rate, network bandwidth utilization rate, and the number of query requests processed per second.

[0015] The current state parameters are calculated using a preset environmental pressure calculation formula to obtain the current environmental pressure index corresponding to the cache system.

[0016] Based on the data lifespan corresponding to the current environmental pressure index, new data items are written into the cache space, and it is determined whether the usage of the cache space has reached a preset threshold.

[0017] If the usage of the cache space reaches the preset threshold, then a preset number of candidate data items are determined from the cache space.

[0018] Optionally, the method further includes:

[0019] If a data request is not found in the cache space, and the cache system's eviction loop queue contains a feature vector of the current request data item corresponding to the data request, then the weighted scoring model at the current moment is adjusted so that the feature vectors of each candidate data item can be weighted and calculated using the adjusted weighted scoring model.

[0020] The elimination circular queue is used to store the feature vectors of eliminated data items.

[0021] Optionally, adjusting the weights of the weighted scoring model at the current moment includes:

[0022] The weights corresponding to the access frequency, timeliness, and relevance in the weighted scoring model at the current moment are increased and adjusted by a preset weight increase method, and the weights corresponding to the data size and access delay are decreased and adjusted by a preset weight decrease method, so as to obtain the adjusted initial weight combination.

[0023] The weights in the initial weight combination are normalized to obtain the weighted scoring model after weight adjustment.

[0024] Optionally, the method further includes:

[0025] Cluster the feature vectors in the elimination circular queue to obtain the clustering results;

[0026] Based on the clustering results, the mapping relationship between data items and data clusters in the association rule base of the caching system is updated, so as to determine the feature vector of each candidate data item according to the updated association rule base, so as to perform weighted calculation on the feature vector of each candidate data item through a preset weighted scoring model;

[0027] The association rule base is used to store the mapping relationship between data items and data clusters through a hash table.

[0028] Optionally, the method further includes:

[0029] If a data request is not found in the cache space, and the currently requested data item corresponds to a data cluster in the association rule base at the current time, the currently requested data item is loaded into the cache space from the backend data source through an asynchronous task queue.

[0030] Secondly, this application provides a data eviction device applied to a caching system, comprising:

[0031] The data item determination module is used to determine a preset number of candidate data items from the cache space when the usage of the cache space of the cache system reaches a preset threshold.

[0032] The score calculation module is used to perform weighted calculation on the feature vectors of each candidate data item using a pre-set weighted scoring model to obtain the elimination priority score corresponding to each candidate data item; the feature vector is a vector representing access frequency, timeliness, data size, access latency, and correlation; the correlation is the ratio of the number of members in the data cluster corresponding to the data item to the target number; the target number is the number of members in the historical maximum data cluster;

[0033] The data elimination module is used to perform data elimination operations on candidate data items corresponding to the elimination priority scores that meet the preset score conditions.

[0034] Thirdly, this application provides an electronic device, comprising:

[0035] Memory, used to store computer programs;

[0036] A processor for executing the computer program to implement the data replacement method as described above.

[0037] Fourthly, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the data replacement method described above.

[0038] Therefore, in this application, the cache system first determines a preset number of candidate data items from the cache space when the cache space usage reaches a preset threshold. Then, a pre-set weighted scoring model is used to weight the feature vectors of each candidate data item to obtain the eviction priority score corresponding to each candidate data item. The feature vector represents access frequency, timeliness, data size, access latency, and correlation. The correlation is the ratio of the number of members in the data cluster corresponding to the data item to the target number. The target number is the number of members in the historically largest data cluster. Afterwards, a data eviction operation is performed on the candidate data items corresponding to the eviction priority scores that meet the preset score conditions. In this way, this application considers the impact of the data cluster to which the data item belongs in the cache space on the data item itself, and combines multi-dimensional feature vectors to perform a comprehensive data value assessment of the data item. This improves the accuracy of data value assessment, enhances the accuracy of final data eviction, and matches the access needs of actual business scenarios. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 This is a flowchart of a data elimination method disclosed in this application;

[0041] Figure 2 This is a flowchart of a specific data elimination method disclosed in this application;

[0042] Figure 3 This is a schematic diagram of the structure of a data erasure device disclosed in this application;

[0043] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] See Figure 1 As shown, this embodiment of the invention discloses a data eviction method applied to a caching system, comprising:

[0046] Step S11: When the usage of the cache space of the cache system reaches a preset threshold, a preset number of candidate data items are determined from the cache space.

[0047] In this embodiment, the usage of cache space in the caching system can be monitored. When new data is loaded into the cache, it can be determined whether the usage of cache space has reached a preset threshold. If it has, it means that the data items in the cache space need to be cleaned up. Then, a certain number of candidate data items are determined from the cache space.

[0048] In a specific embodiment, the cache space may include a dynamic area and a compressed area; the dynamic area is a region configured based on a hash table and a doubly linked list for storing data items that meet preset burst access conditions, and the compressed area is a region configured based on an array and a hash function for storing data items that meet preset hot data conditions; correspondingly, determining a preset number of candidate data items from the cache space may include: randomly selecting a first preset number of data items from the tail of the doubly linked list corresponding to the dynamic area of ​​the cache space to obtain a first data item, and randomly selecting a second preset number of data items from the compressed area to obtain a second data item; the first data item and the second data item are determined as candidate data items. Specifically, the cache space can be divided into a dynamic area and a compressed area, and the dynamic area may account for a certain percentage. It is used to store newly admitted users and data with bursty access characteristics. Its underlying data structure is a combination of a hash table and a doubly linked list, designed for high efficiency. Access and eviction operations with time complexity. The compressed area can account for a certain percentage. This is used to store tested, long-term access value, and frequently accessed data. In a specific embodiment, to save memory, the frequency statistics of the data in the compressed area are not stored directly, but can be stored using... Data structure; this structure consists of a two-dimensional integer array (depth) ,width The array consists of 4 hash functions and d independent hash functions. When the access frequency of a key needs to be increased, the four hash functions are used to calculate four positions in the array, and the counters at these positions are incremented. When querying the frequency, the minimum value of the counts at these four positions is returned as an approximate estimate of the actual frequency. Furthermore, in the process of selecting candidate data items from the cache space, a first number of data items can be randomly selected from the tail of the doubly linked list corresponding to the dynamic area (i.e., the least recently accessed element), and a second number of data items can be randomly selected from the compressed area to form candidate data items.

[0049] In one specific embodiment, when the cache space usage of the caching system reaches a preset threshold, determining a preset number of candidate data items from the cache space may include: collecting the current status parameters of the caching system; the current status parameters include processor load rate, network bandwidth utilization, and the number of query requests processed per second; calculating the current status parameters using a preset environmental pressure calculation formula to obtain the current environmental pressure index corresponding to the caching system; writing new data items into the cache space according to the data lifetime corresponding to the current environmental pressure index, and determining whether the cache space usage has reached the preset threshold; if the cache space usage reaches the preset threshold, then determining a preset number of candidate data items from the cache space. Specifically, when determining whether the cache space usage of the caching system has reached the preset threshold, the current status parameters of the caching system can be collected in real time. These collected status parameters include the average processor load rate, network bandwidth utilization, and the number of query requests processed per second (QPS). It can be a historical average calculated using the exponential moving average algorithm. Next, the current environmental pressure index of the system is obtained through the constructed environmental pressure calculation formula. Correspondingly, to cope with environmental pressure, the lifetime of data items in the cache space can be dynamically adjusted; based on the calculated current environmental pressure index, the lifetime of new data items under the corresponding environmental pressure can be further calculated. Then, based on the calculated lifetime of the new data items, the new data items are written to the cache space; after writing, it is determined whether the usage of the cache space has reached a preset threshold. If the usage reaches the threshold, a data eviction process can be triggered, determining a preset number of candidate data items to be evicted from the cache space.

[0050] Step S12: The feature vectors of each candidate data item are weighted and calculated using a pre-set weighted scoring model to obtain the elimination priority score corresponding to each candidate data item; the feature vector is a vector representing access frequency, timeliness, data size, access delay and correlation; the correlation is the ratio of the number of members of the data cluster corresponding to the data item to the target number; the target number is the number of members of the historical maximum data cluster.

[0051] In this embodiment, the above steps determine a certain number of candidate data items from the cache space. Then, a pre-set weighted scoring model is used to weight the feature vectors of each candidate data item to obtain the elimination priority score of the corresponding candidate data item. Here, the feature vectors are feature vectors related to dimensions such as access frequency, timeliness, data size, access latency, and correlation of the data item. It should be noted that the correlation here is the ratio of the number of members in the data cluster corresponding to the data item to the target number, and the target number is the number of members in the historical largest data cluster. In a specific embodiment, the feature vectors of each dimension need to be normalized first to eliminate the influence of dimensions, so that each feature can be compared on the same scale.

[0052] In one specific embodiment, the method may further include: if a data request miss occurs in the cache space, and the eviction circular queue of the caching system contains a feature vector of the current request data item corresponding to the data request, then the weighted scoring model at the current moment is adjusted to perform weighted calculation on the feature vectors of each candidate data item using the adjusted weighted scoring model; wherein, the eviction circular queue is used to store the feature vectors of evicted data items. Specifically, the caching system can construct an eviction circular queue to record evicted data items in the cache space; then, at the moment of cache miss, it checks whether there is a data item corresponding to the relevant data request in the queue, i.e., whether the data item corresponding to the data request has been erroneously evicted; if it exists in the queue, it can be determined that this data item has been erroneously evicted, and then the weighted scoring model at the current moment is adjusted to optimize the accuracy of subsequent data eviction. Further, in a specific embodiment, when it is determined that a data item has been erroneously evicted, the original feature vector of the data item at the time of evicting can be extracted from the eviction circular queue to respond to the relevant data request based on the feature vector.

[0053] In another specific embodiment, adjusting the weights of the weighted scoring model at the current moment may include: increasing the weights corresponding to the access frequency, timeliness, and relevance in the weighted scoring model at the current moment using a preset weight increase method, and decreasing the weights corresponding to the data size and access latency using a preset weight decrease method, to obtain an adjusted initial weight combination; normalizing each weight in the initial weight combination to obtain the weighted scoring model after weight adjustment. Specifically, based on the feature vector of the incorrectly eliminated data items, the weights of favorable features are increased: for features such as frequency, timeliness, and relevance, which should be retained more frequently, their weights are increased; and the weights of unfavorable features are decreased: for features such as data size and access latency, which should be eliminated more frequently, their weights are decreased. To prevent the weights from growing indefinitely or becoming negative, the weights need to be normalized.

[0054] Step S13: Perform a data elimination operation on the candidate data items corresponding to the elimination priority scores that meet the preset score conditions.

[0055] In this embodiment, the elimination priority score corresponding to each candidate data item can be obtained through the above steps. Data items with scores greater than a certain threshold are then eliminated according to pre-set score conditions. It is understood that in some embodiments, when eliminating a data item, the feature vector corresponding to the data item can be treated as a record and pushed into the elimination circular queue of the cache system, providing a data basis for subsequent scoring weight adjustments.

[0056] In one specific embodiment, the method may further include: clustering the feature vectors in the eviction circular queue to obtain clustering results; updating the mapping relationship between data items and data clusters in the association rule base of the caching system based on the clustering results, so as to determine the feature vectors of each candidate data item according to the updated association rule base, so as to perform weighted calculation on the feature vectors of each candidate data item through a preset weighted scoring model; wherein, the association rule base is used to store the mapping relationship between data items and data clusters through a hash table. Specifically, to optimize cache hits, the feature vectors stored in the eviction circular queue can be clustered, and the obtained clustering results represent hot data, which can reflect the relationship between data requests to a certain extent; furthermore, the caching system can pre-set an association rule base to store the mapping relationship between data items and data clusters through a hash table; in this way, the mapping relationship stored in the association rule base can be updated through periodic clustering. It is understood that the correlation characteristics of data items need to consider the mapping relationship in the association rule base, and through the above clustering update steps, the association rule base can provide real-time correlation queries.

[0057] In another specific embodiment, the method may further include: if a data request misses in the cache space, and the currently requested data item corresponds to a data cluster in the association rule base at the current time, then the currently requested data item is loaded into the cache space from the backend data source via an asynchronous task queue. Specifically, when a data request misses the cache, the association rule base can be queried to see if a data cluster corresponding to the data item of the data request exists. If a corresponding data cluster is found in the association rule base, and the number of members in the data cluster is greater than a certain number, then the data request can be considered a potential hotspot backflow signal; and the data item corresponding to the data request is loaded into the cache space from the backend data source via an asynchronous task queue. It should be noted that for potential hotspot backflow signals, it is possible to choose to load all data items corresponding to all members of the corresponding data cluster into the cache space. Simultaneously, to prevent preloading storms, a global preloading rate limiter can be set.

[0058] Therefore, this application can dynamically adjust the lifespan of new data based on the monitored "environmental stress index" of the caching system, enabling the cache to accelerate its turnover under high pressure and remain more stable under low pressure. This gives the cache environmental adaptability, allowing it to actively participate in system load adjustment and significantly improve the stability and performance resilience of the entire application under dynamic loads. Furthermore, this application considers the impact of the data cluster to which a data item belongs in the cache space on the data item itself, and combines multi-dimensional feature vectors to conduct a comprehensive data value assessment of the data item, which can improve the accuracy of data value assessment and the accuracy of data eviction, matching the access needs of actual business scenarios. Further, by monitoring "erroneous eviction" events in real time and using this as feedback, the weights of its five-dimensional eviction scoring model are adjusted online and automatically; this gives the eviction scoring model the ability to continuously learn and self-evolve, enabling it to autonomously adapt to changes in business access patterns over a long period, thereby achieving a cache hit rate that is continuously optimized over time and far exceeds that of static strategies. Furthermore, by analyzing the characteristics of the obsolete data at the moment of obsolescence, potential clusters of relationships between data can be discovered in real time. This enables accurate and real-time prediction of relationships and data preloading, effectively addressing challenges such as sudden changes in access patterns and cache cold starts, and significantly improving the accuracy of cache prediction.

[0059] like Figure 2 As shown, this embodiment discloses a specific data eviction method applied to a caching system, which specifically includes:

[0060] In this embodiment, the cache system is first initialized and the data structure is established. The cache space can be divided into a dynamic area and a compressed area. The dynamic LRU area (denoted as T1 area) can account for a certain percentage. It is used to store newly admitted users and data with bursty access characteristics; its underlying data structure is a combination of a hash table and a doubly linked list, used to achieve high efficiency. Access and eviction operations with time complexity. Correspondingly, the compressed LFU area (denoted as T2 area) can account for a certain percentage. This is used to store tested, long-term access value, frequently accessed data. Furthermore, to save memory, the frequency statistics of this frequently accessed data are not stored directly, but rather... The data structure consists of a two-dimensional integer array and d independent hash functions. When the access frequency of a key needs to be increased, the four hash functions calculate four positions in the array and increment the counters at these positions. When querying the frequency, the minimum value of the counts at these four positions is returned as an approximate estimate of the true frequency. Furthermore, the caching system can set up a Ghost area for evicted metadata (denoted as B1 area) and a Ghost area for the association rule base (denoted as B2 area); neither of these areas occupies the main cache. B1 area can be configured as a fixed-size (e.g., a capacity of 10,000 records) circular queue (i.e., an evicting circular queue). This queue stores the complete five-dimensional feature vector of the recently evicted data item, serving as the data source for subsequent clustering analysis and model weight adjustment. When the queue is full, new records overwrite the oldest records. Correspondingly, B2 area can be configured as two hash tables; the first hash table... Used to store the mapping between data keys and their respective cluster IDs (Identity Documents); A second hash table This is used to store the mapping from cluster IDs to a list of all member keys for that cluster. Together, these two hash tables form a relational rule base that can be queried in real time.

[0061] Then, the caching system is monitored in real time and its multi-dimensional features are collected. Specifically, the background monitoring thread of the caching system can be set to collect the average CPU (Central Processing Unit) load rate through the standard system interface at a 1-second interval. and network bandwidth utilization Meanwhile, a continuous tracking system (Number of query requests processed per second), and a smoothed historical average is calculated using the Exponential Moving Average (EMA) algorithm. It is used for the stable calculation of the Environmental Pressure Index (EMA). The formula for calculating EMA is shown below:

[0062] ;

[0063] in, Let be the average QPS at time t. Let be the real-time value at time t. for The average QPS at time point.

[0064] Furthermore, each data item (Key) in the cache space maintains a corresponding metadata object in memory; this object stores the following micro-features: precise access timestamp: recording the timestamp of the most recent access; cumulative access frequency: for data in T1 area, it is counted directly; for data in T2 area, it is counted by querying the compressed area. Estimate its frequency; data item size in bytes: determined when data is first written. (Refers to the percentage of currently used memory to the total memory) The command calculates and stores; Average access latency: Uses a circular array of size 100 as a sliding window to record the latency of the last 100 accesses and calculates its average value in real time.

[0065] Furthermore, the caching system can control the writing of new data based on environmental pressure. When new data needs to be loaded into the cache, the system does not immediately put it into the cache. Instead, it first calculates the environmental pressure index of the caching system and determines the dynamic lifetime of the new data before storing it in the cache. Specifically, it first calculates the real-time environmental pressure index (EPI) based on the collected real-time multidimensional features. , , and historical average The Environmental Stress Index (EPI) of the current system is calculated using the following deterministic formula. This index is a floating-point number between 0 and 1, quantifying the current load stress of the system.

[0066] ;

[0067] Then calculate the dynamic time to live (T1) of the new data item. Base survival time The time can be fixed at 60 seconds, the pressure sensitivity coefficient k is fixed at 3, and the dynamic survival time is calculated using the following formula:

[0068] ;

[0069] Finally, the new data item is stored in area T1, and its expiration timestamp is set to the current timestamp + 1. This mechanism ensures that new data flows faster under high system pressure, prioritizing space for high-value data.

[0070] When cache usage reaches a certain threshold, a data eviction process can be triggered. In a specific embodiment, to avoid the performance overhead of global scanning, the system uses a sampling method to generate a candidate set. Ten data items are randomly selected from the tail of the doubly linked list in area T1 (i.e., the least recently accessed element), and ten data items are randomly selected from area T2, forming a candidate set (candidate data items) containing 20 data items. Then, a standardized five-dimensional feature vector is calculated for the candidate set; specifically, for each data item in the candidate set... Information is extracted from its metadata object and association rule base, and normalized to generate feature vectors. Normalization aims to eliminate the influence of dimensions, allowing features to be compared on the same scale. Among these, frequency features... : Access frequency Mapped to via the Sigmoid function The range is designed to smooth out the effects of extreme high frequencies and to focus on changes in the mid-to-high frequency range. Timeliness is a key characteristic. Calculated using the exponential decay function:

[0071] ;

[0072] in, =0.001 is the time decay constant, and this function can sensitively reflect the importance of recent visits.

[0073] Size characteristics : Size of data items Divide by a system-preset baseline size and perform linear normalization:

[0074] ;

[0075] Delay characteristics : Average access latency Divide by an acceptable latency threshold for the system and perform linear normalization:

[0076] ;

[0077] Relationship characteristics Real-time query of the defined B2 area .like If it exists in a cluster, then Divide the number of members in its cluster by the historical maximum number of cluster members; if no such cluster exists, then... =0.

[0078] Furthermore, the feature vector V of each candidate data item is... i The input is fed into a pre-initialized dynamically weighted linear scoring model, based on the current weight vector W (including...). , , , , Calculate the elimination priority score The higher the score, the lower its value in the cache, and the higher its priority for eviction.

[0079] ;

[0080] This formula clearly reflects the decision-making logic: the larger the data and the slower the access, the higher the score; the more frequent the access, the more recent the data, and the stronger the correlation, the lower the score. The model ultimately outputs a candidate set with the highest... The key name of the data item. Based on the calculated eviction priority score, the corresponding data item is deleted from the cache. Before deletion, its complete five-dimensional feature vector is retrieved. As a record, it is pushed into the circular queue of area B1 to provide a data foundation for subsequent cluster analysis and model weight adjustment.

[0081] In a specific implementation, clustering can be performed every 60 seconds on the data in area B1 in the background of the caching system to identify potential hotspot clusters. Specifically, all feature vector records are retrieved from the circular queue of area B1; the DBSCAN algorithm is used for clustering, where the distance between vectors is not the standard Euclidean distance, but a weighted Euclidean distance to highlight the importance of specific dimensions. The distance calculation formula is:

[0082] ;

[0083] in These are the feature dimension weights. To highlight the similarity of access patterns, the weights of frequency, timeliness, and relevance dimensions can be set to 2.0, while other dimensions can be set to 1.0.

[0084] After clustering is complete, the two hash tables for region B2 are updated based on the clustering results. Alternatively, the two hash tables for region B2 can be cleared, the clustering results iterated through, and for each formed cluster, its cluster ID and a list of all member keys stored. And store the mapping relationship between each member key and its cluster ID. .

[0085] In one specific embodiment, when a request If a cache miss occurs in the main cache (T1 / T2), the system will query the B2 region. Check the Did it historically belong to a certain data cluster? If so... exist It was found in the middle, and its cluster If the number of members exceeds a preset threshold of 3, the system considers this a strong potential hotspot backflow signal. The system will immediately... Get from All other member keys are then added to a low-priority background task queue, which asynchronously loads this data from the backend data source and places it into the T1 area. To prevent preload storms, the system can set a global preload rate limiter, such as setting it to preload a maximum of 50 keys per second.

[0086] In another specific embodiment, to achieve system self-optimization and ensure that the scoring model can adapt to changes in business load over a long period, the system continuously monitors all client cache miss requests and dynamically adjusts the weights of the scoring model. Specifically, if a cache miss occurs... If a device is found in the circular queue of area B1 (meaning it was recently evicted), the system classifies it as an erroneous evictment event. The system will then retrieve the erroneous evictment record from the records in area B1. The original feature vector at the time of elimination Upon identifying an erroneous elimination event, it indicates... The initial elimination score The score was incorrectly overcalculated. The system needs to adjust the weights to reduce the scores of similar items in the future. Based on the current scoring formula, the system can perform the following targeted weight updates:

[0087] (1) Increase the weight of advantageous features: For features such as frequency, timeliness, and relevance, which should be retained as much as possible, increase their weight.

[0088] ;

[0089] ;

[0090] ;

[0091] (2) Reduce the weight of unfavorable features: For features such as data size and access latency, which are more likely to be eliminated as their values ​​increase, reduce their weight.

[0092] ;

[0093] ;

[0094] Among them, learning rate The weights are fixed at 0.001. Furthermore, after the weights are updated, normalization is necessary to prevent them from growing indefinitely or becoming negative. Calculate the sum of all new weights:

[0095] ;

[0096] Then divide each new weight by This yields the final updated weight vector. This step ensures that the weights remain consistent. Within the range and with a sum of 1, the scoring model remains stable.

[0097] Furthermore, in a high-concurrency e-commerce platform scenario, the initial state of the scenario is as follows:

[0098] (1) System time: ;

[0099] (2) System load: at a normal level. =0.3、 =0.2、 =1000、 =1000;

[0100] (3) Model weights: ;

[0101] (4) Cache status: Memory utilization reaches It is about to be phased out.

[0102] Furthermore, the data structures for areas T1, T2, B1, and B2 have all been established. DBSCAN parameters, learning rate, etc., have all been configured. Furthermore, the system's background monitoring module continues to run. At a certain time point... (Right now Sixty seconds later, the platform launched a flash sale, causing a sharp increase in system load. The monitoring module collected new macro-indicators: =0.8、 =0.7、 =5000、 =1400;

[0103] At the point of time When a user requests access to a product details page, data needs to be loaded. The data was not found in the cache. After loading the data, the system calculates the Real-Time Environmental Stress Index (EPI).

[0104] Calculate EPI based on the collected system-related metrics:

[0105] ;

[0106] Calculated based on EPI Dynamic TTL in T1 region:

[0107] ;

[0108] The system will Stored in T1 area, with its expiration time set to the current timestamp + 16.16 seconds. Due to the immense system pressure, the default lifespan of new data has been significantly shortened to accelerate cache turnover.

[0109] Furthermore, due to The write caused the cache space usage to exceed the limit. A threshold is set, immediately triggering the data eviction process. Specifically, the system samples and generates a candidate set containing four data items. Feature vectors are calculated for the four candidate data items at the current time. :

[0110]

[0111] Original value: (Unrelated);

[0112] Feature vector : ;

[0113] :

[0114] Original value: (Unrelated);

[0115] Feature vector : ;

[0116] :

[0117] Original value: (Unrelated);

[0118] Feature vector : ;

[0119] :

[0120] Original value: (belonging to a large cluster);

[0121] Feature vector : ;

[0122] Then use the pre-set scoring model ( Calculate the elimination score for each key:

[0123] ;

[0124] ;

[0125] ;

[0126] ;

[0127] It scored the highest (least negative) and was therefore chosen for elimination. The main reason was its enormous size. =0.78) and high access latency ( =0.9).

[0128] The system removed it from memory At the same time, its complete feature vector =[0.27, 0.74, 0.78, 0.9, 0.0] are pushed into the circular queue of area B1 as a record.

[0129] After a period of time, the clustering task in the system background (every 60 seconds) is executed. At this time, the circular queue of area B1 contains... And other recently phased-out report-type data, such as and Their feature vectors are highly similar in both "size" and "latency" dimensions. The DBSCAN algorithm processes the data in region B1. Because these report-type data are very close in the weighted feature space, they are identified as a new cluster, denoted as... The system clears and rebuilds the hash table in area B2. The information was written.

[0130] Furthermore, at a certain point in time A back-end administrator requested access to the weekly report. The request failed to find a match in regions T1 and T2. The system queried the KeyToClusterMap in region B2 and found... belong . The number of members is 3, satisfying the trigger condition of greater than 3 (for simplicity, the threshold is set to greater than or equal to 3). The system determines this to be a "report data feedback" signal. It will immediately be requested... Load back to T1 area, and simultaneously... Other members, namely and It is placed in the background task queue for asynchronous preloading.

[0131] At the point of time ,Right now 140 seconds after being eliminated (less than a 300-second window), another business request also needs to access the daily report. The request was not hit in area T1 / T2. The system checked the circular queue in area B1 and found... The records were clearly listed, confirming that this was an "erroneous elimination event." The system extracted... Feature vector at the time of elimination The system is based on and learning rate To update the weights and punish this incorrect decision:

[0132] ;

[0133] ;

[0134] ;

[0135] ;

[0136] ;

[0137] Calculate the sum of the new weights:

[0138] ;

[0139] Divide each new weight by The final updated weight vector is obtained.

[0140] Therefore, the T1 zone of the caching system in this application directly links the macroscopic load state of the system (CPU, network, etc.) with the lifecycle of microscopic data items through the "Environmental Stress Index" (EPI) model, dynamically calculating their Time-to-Live (TTL). This makes the cache no longer a passive component isolated from the external environment, but an intelligent agent that can actively adapt to system pressure. By storing the complete five-dimensional feature vector of evicted data items and performing DBSCAN clustering on them, deep, non-explicit correlations between data can be actively discovered. This correlation analysis based on the context of "eviction" rather than "access" provides a new and more accurate predictive dimension for solving cache cold start and responding to sudden changes in access patterns. Furthermore, this application constructs a five-dimensional scoring model that integrates frequency, timeliness, data size, access latency, and potential correlations, which can more comprehensively evaluate data value; this model is a "live" model with closed-loop feedback and online adaptive adjustment capabilities. When evicted data is requested again within a short period of time, the system will judge it as a decision error. At this point, the system uses the feature vector of the data at the time of its elimination as a negative feedback signal. Through a gradient update algorithm, it fine-tunes the weight coefficients of each dimension in the five-dimensional scoring model online and automatically. This achieves a closed-loop self-optimization that integrates decision execution, effect feedback, and model correction. This allows the elimination strategy to break free from the constraints of static rules, continuously learn from real business feedback, and autonomously adapt to constantly changing access patterns, thus achieving true intelligence in elimination decisions.

[0141] like Figure 3 As shown, this embodiment discloses a data eviction device applied to a caching system, comprising:

[0142] The data item determination module 11 is used to determine a preset number of candidate data items from the cache space when the usage of the cache space of the cache system reaches a preset threshold.

[0143] The score calculation module 12 is used to perform weighted calculation on the feature vectors of each candidate data item using a pre-set weighted scoring model to obtain the elimination priority score corresponding to each candidate data item; the feature vector is a vector representing access frequency, timeliness, data size, access delay and correlation; the correlation is the ratio of the number of members of the data cluster corresponding to the data item to the target number; the target number is the number of members of the historical maximum data cluster;

[0144] The data elimination module 13 is used to perform data elimination operations on candidate data items corresponding to the elimination priority scores that meet the preset score conditions.

[0145] Therefore, this application considers the impact of the data cluster to which the data item belongs in the cache space on the data item itself, and combines multi-dimensional feature vectors to conduct a comprehensive data value assessment of the data item, which can improve the accuracy of data value assessment, improve the accuracy of final data eviction, and match the access needs of actual business scenarios.

[0146] In one specific embodiment, the cache space includes a dynamic area and a compressed area; the dynamic area is a region configured based on a hash table and a doubly linked list for storing data items that meet preset burst access conditions, and the compressed area is a region configured based on an array and a hash function for storing data items that meet preset hot data conditions; correspondingly, the data item determination module 11 may include:

[0147] The data item selection unit is used to randomly select a first preset number of data items from the tail of the doubly linked list corresponding to the dynamic area of ​​the cache space to obtain a first data item, and randomly select a second preset number of data items from the compressed area to obtain a second data item.

[0148] A data item determination unit is used to determine the first data item and the second data item as candidate data items.

[0149] In one specific embodiment, the data item determination module 11 may include:

[0150] The status parameter acquisition unit is used to acquire the current status parameters of the cache system; the current status parameters include processor load rate, network bandwidth utilization rate, and the number of query requests processed per second.

[0151] An environmental pressure index calculation unit is used to calculate the current state parameters using a preset environmental pressure calculation formula to obtain the current environmental pressure index corresponding to the cache system.

[0152] The cache space usage determination unit is used to write new data items into the cache space according to the data lifetime corresponding to the current environmental pressure index, and to determine whether the usage of the cache space has reached a preset threshold.

[0153] The data item determination submodule is used to determine a preset number of candidate data items from the cache space when the usage of the cache space reaches the preset threshold.

[0154] In one specific embodiment, the device may further include:

[0155] The weight adjustment module is used to adjust the weight of the weighted scoring model at the current time when a data request is missed in the cache space and the feature vector of the current request data item corresponding to the data request exists in the eviction circular queue of the cache system, so as to perform weighted calculation on the feature vector of each candidate data item through the adjusted weighted scoring model.

[0156] The elimination circular queue is used to store the feature vectors of eliminated data items.

[0157] In another specific embodiment, the weight adjustment module may include:

[0158] The weight adjustment unit is used to increase and adjust the weights corresponding to the access frequency, the timeliness and the relevance in the weighted scoring model at the current time by using a preset weight increase method, and to decrease and adjust the weights corresponding to the data size and the access delay by using a preset weight decrease method, so as to obtain the adjusted initial weight combination.

[0159] The normalization unit is used to normalize each weight in the initial weight combination to obtain the weighted scoring model after weight adjustment.

[0160] In yet another specific embodiment, the device may further include:

[0161] The clustering module is used to cluster the feature vectors in the elimination circular queue to obtain the clustering results;

[0162] The mapping relationship update module is used to update the mapping relationship between data items and data clusters in the association rule base of the caching system based on the clustering results, so as to determine the feature vector of each candidate data item according to the updated association rule base, so as to perform weighted calculation on the feature vector of each candidate data item through a preset weighted scoring model; wherein, the association rule base is used to store the mapping relationship between data items and data clusters through a hash table.

[0163] In one specific embodiment, the device may further include:

[0164] The data item loading module is used to load the currently requested data item from the backend data source into the cache space through an asynchronous task queue when a data request is not found in the cache space and the currently requested data item corresponds to a data cluster in the association rule base at the current time.

[0165] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0166] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data replacement method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0167] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0168] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0169] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the data replacement method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0170] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned data replacement method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0171] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0172] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0173] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0174] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0175] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data eviction method, characterized in that, Applied to caching systems, including: When the usage of the cache space in the cache system reaches a preset threshold, a preset number of candidate data items are determined from the cache space; The feature vectors of each candidate data item are weighted and calculated using a pre-set weighted scoring model to obtain the elimination priority score corresponding to each candidate data item; the feature vector is a vector representing access frequency, timeliness, data size, access latency, and correlation; the correlation is the ratio of the number of members in the data cluster corresponding to the data item to the target number; the target number is the number of members in the historical maximum data cluster; A data elimination operation is performed on the candidate data items corresponding to the elimination priority scores that meet the preset score conditions.

2. The data elimination method according to claim 1, characterized in that, The cache space includes a dynamic area and a compressed area; the dynamic area is a region configured based on a hash table and a doubly linked list for storing data items that meet preset burst access conditions, and the compressed area is a region configured based on an array and a hash function for storing data items that meet preset hot data conditions. Accordingly, determining a preset number of candidate data items from the cache space includes: A first data item is obtained by randomly selecting a first preset number of data items from the tail of the doubly linked list corresponding to the dynamic area of ​​the cache space, and a second data item is obtained by randomly selecting a second preset number of data items from the compressed area. The first data item and the second data item are determined as candidate data items.

3. The data elimination method according to claim 1, characterized in that, When the usage of the cache space in the caching system reaches a preset threshold, determining a preset number of candidate data items from the cache space includes: Collect the current status parameters of the caching system; the current status parameters include processor load rate, network bandwidth utilization rate, and the number of query requests processed per second. The current state parameters are calculated using a preset environmental pressure calculation formula to obtain the current environmental pressure index corresponding to the cache system. Based on the data lifespan corresponding to the current environmental pressure index, new data items are written into the cache space, and it is determined whether the usage of the cache space has reached a preset threshold. If the usage of the cache space reaches the preset threshold, then a preset number of candidate data items are determined from the cache space.

4. The data elimination method according to any one of claims 1 to 3, characterized in that, Also includes: If a data request is not found in the cache space, and the cache system's eviction loop queue contains a feature vector of the current request data item corresponding to the data request, then the weighted scoring model at the current moment is adjusted so that the feature vectors of each candidate data item can be weighted and calculated using the adjusted weighted scoring model. The elimination circular queue is used to store the feature vectors of eliminated data items.

5. The data elimination method according to claim 4, characterized in that, The weight adjustment of the weighted scoring model at the current moment includes: The weights corresponding to the access frequency, timeliness, and relevance in the weighted scoring model at the current moment are increased and adjusted by a preset weight increase method, and the weights corresponding to the data size and access delay are decreased and adjusted by a preset weight decrease method, so as to obtain the adjusted initial weight combination. The weights in the initial weight combination are normalized to obtain the weighted scoring model after weight adjustment.

6. The data elimination method according to claim 4, characterized in that, Also includes: Cluster the feature vectors in the elimination circular queue to obtain the clustering results; Based on the clustering results, the mapping relationship between data items and data clusters in the association rule base of the caching system is updated, so as to determine the feature vector of each candidate data item according to the updated association rule base, so as to perform weighted calculation on the feature vector of each candidate data item through a preset weighted scoring model; The association rule base is used to store the mapping relationship between data items and data clusters through a hash table.

7. The data elimination method according to claim 6, characterized in that, Also includes: If a data request is not found in the cache space, and the currently requested data item corresponds to a data cluster in the association rule base at the current time, the currently requested data item is loaded into the cache space from the backend data source through an asynchronous task queue.

8. A data elimination device, characterized in that, Applied to caching systems, including: The data item determination module is used to determine a preset number of candidate data items from the cache space when the usage of the cache space of the cache system reaches a preset threshold. The score calculation module is used to perform weighted calculation on the feature vectors of each candidate data item using a pre-set weighted scoring model to obtain the elimination priority score corresponding to each candidate data item; the feature vector is a vector representing access frequency, timeliness, data size, access latency, and correlation; the correlation is the ratio of the number of members in the data cluster corresponding to the data item to the target number; the target number is the number of members in the historical maximum data cluster; The data elimination module is used to perform data elimination operations on candidate data items corresponding to the elimination priority scores that meet the preset score conditions.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the data replacement method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the data replacement method as described in any one of claims 1 to 7.

Images