Data processing method, device and equipment based on cache storage

Abstract

The invention discloses a data processing method, device and equipment based on cache storage. Comprising the following steps: collecting a cache data sample set in a cache; determining a sliding window to be used this time according to the configured sliding window duration; judging whether a sample of which the access record falls outside the sliding window exists in the cache data sample set or not; if the cache data sample set does not exist, calculating a data retention score of each sample in the cache data sample set according to the data access time and the data access frequency of each sample in the cache data sample set for the time range of the sliding window; and according to the data retention score, screening out one or more samples to be eliminated from the cache data sample set, and cleaning the samples to be eliminated from the cache.

Description

Technical Field

[0001] This specification relates to the field of cache technology, and in particular to a data processing method, apparatus, and device based on cache storage. Background Art

[0002] Cache storage facilitates fast data access. For example, cache storage can be implemented based on Redis (Remote Dictionary Server). Redis is an open-source, in-memory data structure storage system that can be used as a database, cache, and messaging middleware. Redis supports a variety of data structures, such as strings, hashes, lists, sets, and sorted sets, and provides a rich set of operations to manipulate these data structures.

[0003] Since the storage space of the cache is much smaller than the ordinary storage space (usually the hard disk space), the data in the cache needs to be cleaned up frequently.

[0004] In the prior art, when data is stored in a cache, an expiration time is set for the data. According to the expiration time, if the data is determined to have not expired, the data will be retained in the cache. Conversely, if the data is determined to have expired, the data will be cleared from the cache as soon as possible, thereby freeing up the cache space it occupies.

[0005] Based on this, a more flexible and practical cache cleaning solution is needed. Summary of the Invention

[0006] One or more embodiments of this specification provide a data processing method, apparatus, device, and storage medium based on cache storage to solve the following technical problem: a more flexible and practical cache cleaning solution is needed.

[0007] To solve the above technical problems, one or more embodiments of this specification are implemented as follows:

[0008] One or more embodiments of this specification provide a data processing method based on cache storage, including:

[0009] Collecting a cache data sample set in the cache;

[0010] Determine the sliding window to be used this time based on the configured sliding window duration;

[0011] Determine whether there is a sample in the cache data sample set whose access records all fall outside the sliding window;

[0012] If not, then for the time range of the sliding window, calculate the data retention score of each sample according to the data access time and data access frequency of each sample in the cached data sample set;

[0013] According to the data retention score, one or more samples to be eliminated are screened out from the cache data sample set, and the samples to be eliminated are cleared from the cache.

[0014] One or more embodiments of this specification provide a data processing device based on cache storage, including:

[0015] A sample collection module collects a cache data sample set in the cache;

[0016] The window determination module determines the sliding window to be used this time according to the configured sliding window duration;

[0017] A sample judgment module, judging whether there is a sample in the cache data sample set whose access records all fall outside the sliding window;

[0018] If the score calculation module does not exist, the score calculation module calculates the data retention score of each sample in the cached data sample set according to the data access time and data access frequency of each sample in the time range of the sliding window;

[0019] The sample elimination module selects one or more samples to be eliminated from the cached data sample set according to the data retention score, and clears the samples to be eliminated from the cache.

[0020] One or more embodiments of this specification provide a data processing device based on cache storage, including:

[0021] at least one processor; and,

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions executable by the at least one processor, wherein the instructions are executed by the at least one processor to enable the at least one processor to perform:

[0024] Collecting a cache data sample set in the cache;

[0025] Determine the sliding window to be used this time based on the configured sliding window duration;

[0026] Determine whether there is a sample in the cache data sample set whose access records all fall outside the sliding window;

[0027] If not, then for the time range of the sliding window, calculate the data retention score of each sample according to the data access time and data access frequency of each sample in the cached data sample set;

[0028] According to the data retention score, one or more samples to be eliminated are screened out from the cache data sample set, and the samples to be eliminated are cleared from the cache.

[0029] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to:

[0030] Collecting a cache data sample set in the cache;

[0031] Determine the sliding window to be used this time based on the configured sliding window duration;

[0032] Determine whether there is a sample in the cache data sample set whose access records all fall outside the sliding window;

[0033] If not, then for the time range of the sliding window, calculate the data retention score of each sample according to the data access time and data access frequency of each sample in the cached data sample set;

[0034] According to the data retention score, one or more samples to be eliminated are screened out from the cache data sample set, and the samples to be eliminated are cleared from the cache.

[0035] At least one of the above-mentioned technical solutions adopted in one or more embodiments of this specification can achieve the following beneficial effects: there is no need to be strongly restricted by the expiration time limit, but it is possible to flexibly collect cache data sample sets each time, and flexibly set the sliding window each time, give priority to sample data outside the sliding window, and if there is no sample in the sliding window, comprehensively consider the data access time and data access frequency, calculate the data retention score, and use this to screen out samples to be eliminated, and then give priority to clearing these samples to be eliminated from the cache. This helps to manage the cache more flexibly and efficiently, and is more practical for objects that need to use data in the cache. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 A schematic flow chart of a data processing method based on cache storage provided in one or more embodiments of this specification;

[0038] Figure 2 A schematic flow chart of a sliding window determination solution provided in one or more embodiments of this specification;

[0039] Figure 3 A schematic flow chart of a more flexible active control solution for a sliding window provided in one or more embodiments of this specification;

[0040] Figure 4 A schematic diagram of a cache storage architecture of a cluster group provided for one or more embodiments of this specification;

[0041] Figure 5 One or more embodiments of this specification provide Figure 1 A schematic flow chart of a specific embodiment of the method;

[0042] Figure 6 A schematic diagram of the structure of a data processing device based on cache storage provided in one or more embodiments of this specification;

[0043] Figure 7 A schematic diagram of the structure of a data processing device based on cache storage provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0044] The embodiments of this specification provide a data processing method, apparatus, device, and storage medium based on cache storage.

[0045] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0046] In order to solve the problems in the background technology, on the one hand, we break through the limitation of expiration time, comprehensively consider the two dimensional indicators of data access time and data access frequency, and make the evaluation criteria of hot data more comprehensive and centralized by formulating a multi-factor calculation formula and optimizing the scoring strategy, so as to further improve the popularity of data in cache storage, thereby improving the hit rate and practicality of cached data, improving cache storage performance, and reducing database pressure; on the other hand, we break through the limitation of time series thinking and create a more flexible and fairer sliding window, especially a time-disordered sliding window, to more accurately cache data sample sets, select samples to be eliminated, and adjust the sliding strategy of the sliding window in reverse according to the situation after elimination, which will help to achieve better results in the future.

[0047] Based on this general idea, the solution of this application will be further explained below.

[0048] Figure 1 This is a flow chart of a data processing method based on cache storage provided in one or more embodiments of this specification. This process can be executed multiple times periodically or irregularly as needed to dynamically clear a portion of data from the cache each time.

[0049] Figure 1 The process in includes the following steps:

[0050] S102: Collecting a cache data sample set in the cache.

[0051] First, a portion of data in the cache is selected as the cache data sample set. The selection strategy adopted in this step can be diverse, such as random selection, selection from relatively older data, etc.

[0052] In one or more embodiments of this specification, some personalized configurations are supported, involving the calculation of data retention scores, which are used to decide which data to retain in the cache first and which data to eliminate and clear. Two of the configurations are, for example, the policy effective range and the storage type. It is assumed that the policy effective range can be "all data" (referring to all data in the cache) or "expired data" (referring to data with an expiration time set; or, referring to expired data), etc.; the storage type can be divided based on the scale dimension, for example, "small cache", "medium cache" or "large cache", etc., and of course it can also be divided based on other dimensions such as data result type, product type or business type.

[0053] In this case, when collecting samples, you can determine whether to collect the cached data sample set in the expired data based on the configured policy effectiveness scope; if so (the policy effectiveness scope is "expired data"), then determine the expired data set in the cache, and based on the configured storage type, in the expired data set in the cache, collect the cached data sample set whose scale matches the storage type (the larger the cache scale, the larger the collection scale); if not, then you can collect the cached data sample set whose collection scale matches the storage type in "all data".

[0054] Similarly, more desirable ranges can be set, and the specific meaning of each desirable range can be pre-defined as needed.

[0055] S104: Determine the sliding window to be used this time according to the configured sliding window duration.

[0056] For the same data in the cache, there may be access records, and there may be multiple access records, and these access records have corresponding data access times (i.e., the time of access). This application uses a sliding window to flexibly and preliminarily screen the candidate samples to be eliminated for the cached data sample set.

[0057] By using a sliding window, we attempt to identify data with high reference value. This data better reflects the true value of the data, making it easier to prioritize which data to eliminate. This true value at least includes the degree of demand for future access. Of course, if necessary, it can also include other dimensions such as the importance of the data itself.

[0058] In addition to configuring the sliding window duration, you can also configure the sliding window mode, including simple window mode. In simple window mode, a normal timeline is used, and the sliding window will slide along the normal timeline. For example, if the sliding window duration is set to three months, the sliding window will slide to the three-month time range that is closest to the current timeframe (higher timeliness and reference value).

[0059] In one or more embodiments of this specification, the sliding window mode may also include a disordered complex window mode. For the disordered complex window mode, time blocks can be reassembled as needed to form a disordered time axis, making the data distribution smoother. The sliding window can then be determined on the disordered time axis, and the sliding of the sliding window can be controlled. This facilitates more flexible and reliable selection of data for retention or elimination. More importantly, it can break through the constraints of "recent time" to help avoid instability or atypicality in recent situations, which can lead to unreliable subsequent processing results.

[0060] S106: Determine whether there is a sample in the cache data sample set whose access records all fall outside the sliding window.

[0061] In one or more embodiments of the present specification, if there are samples in the cached data sample set whose access records all fall outside the sliding window, since the current focus is relatively more on the data within the sliding window, it can be considered that the data outside the sliding window has a relatively low reference value (for example, it cannot reflect the true value of the data). For efficiency reasons, quick screening can also be performed. For example, one or more samples to be eliminated are directly screened out from the samples in the cached data sample set whose access records all fall outside the sliding window, and the samples to be eliminated are cleared from the cache.

[0062] In this case, the current sample can be filtered out of the sliding window, so that the cache cleaning work can be completed quickly. Furthermore, since the reference value of the data eliminated this time may not be very high, the time interval of the next cycle can be shortened proactively.

[0063] S108: If not, then for the time range of the sliding window, calculate the data retention score of each sample according to the data access time and data access frequency of each sample in the cache data sample set.

[0064] In one or more embodiments of the present specification, data access time and data access frequency are comprehensively considered to measure the degree of demand for the data in the cache to be read next. If the demand is higher, the corresponding calculated data retention score will be more capable (for example, the higher the score), and the data should be retained in the cache as much as possible. Conversely, if the demand is lower, the data can be considered as data to be eliminated and cleared from the cache first.

[0065] The data access time may specifically include one or more access times (assuming the time range of the sliding window is a recent time range). Based on the assumption that "recently used data is relatively more likely to be used again in the future", it can be considered that for data that has not been accessed for a long time, the above-mentioned demand level is relatively low, and for data that has been accessed recently, the above-mentioned demand level is relatively high. This will have a corresponding impact on the data retention score, and in particular, the most recent access time of the data within the time range of the sliding window can be considered. Similarly, the data access time can also reflect the number of accesses. If the number of accesses is high, it can also reflect a high level of demand. In this case, the time range of the sliding window does not necessarily need to be the most recent time range.

[0066] The data access frequency can specifically include the access frequency of data within a sliding window. Here, at least the access records within the sliding window are considered. Furthermore, if the access frequency of the entire data set is pre-recorded, the access frequency of the entire data set can be retrieved for the data within the sliding window. If no such records exist, the access frequency within the sliding window can be considered.

[0067] Based on the assumption that "the higher the access frequency, the more important", it can be considered that the data that is frequently accessed has a higher degree of demand, while if it is not frequently accessed, the degree of demand is relatively lower, which will have a corresponding impact on the data retention score.

[0068] In one or more embodiments of this specification, corresponding weights are set for data access time and data access frequency to comprehensively calculate the data retention score. In addition, more complex comprehensive calculation strategies can also be used.

[0069] It should be noted that whether a higher data retention score indicates that data should be retained rather than cleaned, or a lower data retention score indicates that data should be retained rather than cleaned, can be pre-defined based on actual needs. The corresponding score calculation method will also be adjusted accordingly. For ease of description, in some of the following embodiments, it is assumed that a higher data retention score indicates that data should be retained rather than cleaned.

[0070] In addition, regarding the data retention score, the applicant noticed that if only the data access time or the data access frequency is considered separately, there will be problems. For example, if only the data access time is considered separately, some frequently accessed but recently unused data may be mistakenly cleared; moreover, it is not friendly to sudden access patterns. When some unpopular data is suddenly accessed in large quantities, it may occupy the cache position of hot data. For another example, if only the data access frequency is considered separately, it is not friendly to new data. The newly added data is easily deleted by mistake due to the small number of accesses; moreover, it is difficult to cope with frequency changes. If the access frequency of the data fluctuates greatly over time, some outdated hot data may be retained, and it is impossible to respond to new access patterns in a timely manner. Based on this, the present application optimizes the scoring strategy and makes the evaluation criteria for hot data more comprehensive and centralized, thereby further improving the heat of the data in the cache storage, thereby improving the hit rate of the cache data and improving the cache storage performance.

[0071] S110: Filter out one or more samples to be eliminated from the cached data sample set according to the data retention score, and clear the samples to be eliminated from the cache.

[0072] Samples with low data retention scores can be preferentially selected as samples to be eliminated.

[0073] In one or more embodiments of this specification, this process is executed separately for multiple different caches. Different caches can be configured with differentiated personalization (e.g., policy scope, sliding window duration, etc.), which involves the calculation of data retention scores.

[0074] To facilitate management and control, and to adapt to actual business needs, caches can be divided according to business dimensions. For example, within a designated storage system (e.g., a cluster, a cluster group, or even multiple cluster groups), multiple caches can be allocated for different businesses to temporarily store data related to those businesses. Furthermore, based on the differences in different businesses, the corresponding caches can be configured with individualized configurations.

[0075] pass Figure 1 This method does not need to be strictly restricted by the expiration time. Instead, it can flexibly collect cache data sample sets each time and flexibly set the sliding window each time, giving priority to sample data outside the sliding window. If there is no sample in the sliding window, the data access time and data access frequency are comprehensively considered to calculate the data retention score, and the samples to be eliminated are screened out based on this, and then these samples to be eliminated are cleared from the cache first. This helps to manage the cache more flexibly and efficiently, and is more practical for objects that need to use the data in the cache.

[0076] based on Figure 1 This specification also provides some specific implementation plans and extension plans of the method, which will be described below.

[0077] In one or more embodiments of the present specification, personalized configuration may also include: weights set for data access time and data access frequency respectively; the weights are configured for the cache corresponding to the business based on the differences in the sensitivity of different businesses to data access time and data access frequency.

[0078] For example, a navigation service based on traffic congestion conditions has very high requirements for the timeliness of data because traffic congestion conditions are changing in real time and very quickly. This often means that the service is relatively more sensitive to data access time. Correspondingly, the weight for data access time can be set higher.

[0079] For example, self-media content businesses are more concerned about the popularity of data. Content that is viewed, browsed, and even forwarded by more users is more valuable data. This often means that they are relatively more sensitive to the frequency of data access. Correspondingly, the weight of data access frequency can be set higher.

[0080] Similarly, other personalized configurations can also be set differently for different caches corresponding to different businesses based on differences in business characteristics.

[0081] For example, with respect to the sliding window duration, if the business (such as food delivery business) is popular, data access is also intensive, and different data have more access opportunities, making it less likely to be misjudged. Therefore, a relatively shorter sliding window duration can be used. If the business (such as library collection query business) is less popular, its data access points are relatively sparse on the timeline. In order to reduce the possibility of misjudgment, a relatively longer sliding window duration can be used to improve fault tolerance.

[0082] Some personalized configurations have been mentioned above, such as the policy scope, storage type, sliding window duration, sliding window mode, time frequency weight, etc. In addition, there can also be model types, such as simple type and standard type based on operating efficiency.

[0083] If the configured model type is determined to be standard rather than simple, a longer timeframe can be considered instead of a sliding window. Furthermore, a more sophisticated metric that considers the actual situation (introducing more auxiliary parameters and adopting a more complex method that facilitates a more reliable fit) can be used to determine the data retention score. For example, a full lifecycle approach can be used instead of a sliding window approach. For the full lifecycle, a decay factor and / or an increment factor can be introduced. Based on these factors, the data retention score of each sample can be calculated to determine the samples to be eliminated.

[0084] The decay factor is used to actively decay old data access counts according to a first set condition, and the increment factor is used to actively increment new data access counts according to a second set condition. For example, for data access counts older than a set time limit, the older the data access count, the more decay is performed, making it smaller than the actual value, according to a set decay ratio related to time. In this case, the set time limit falls under the first set condition, and the set decay ratio related to time falls under the decay factor. For another example, for data access counts newer than a set time limit, the newer and more frequent data access counts are incremented, making them larger than the actual value, according to a set increment ratio related to time and frequency. In this case, the set time limit falls under the second set condition, and the set increment ratio related to time and frequency falls under the increment factor.

[0085] Conversely, if the configured model type is determined to be simple rather than standard, the calculation can be simplified. For example, in the simple model, after removing the preset decay and increment factors, the data retention score of each sample in the cached data sample set is calculated based on the data access time and frequency of each sample.

[0086] The previously mentioned disordered complex window mode can select data more flexibly and reliably. For ease of understanding, one or more embodiments of this specification provide a sliding window determination solution, see Figure 2 , Figure 2 The process is shown.

[0087] Figure 2 The process in includes the following steps:

[0088] S202: Obtain cache data access point distribution data on a normal time axis.

[0089] On a normal timeline, time is normally continuous. Divide the normal timeline into multiple continuous time zones at set intervals (a larger time range, such as one year, six months, three months, or one month, can be used). Count the cache data access point distribution data within each time zone to obtain the corresponding distribution density.

[0090] S204: Based on the differences in distribution density between different time zones on the normal time axis as reflected by the cached data access point distribution data, the different time zones are reassembled to generate a random time axis, such that the distribution density of cached data access points increases more smoothly in the direction of the random time axis (here, the set positive direction). Similarly, the distribution density of cached data access points can also decrease smoothly, but the control means need to be adjusted accordingly.

[0091] In one or more embodiments of this specification, a random timeline is considered instead of a normal timeline. This is because, when a normal timeline is used, no matter how the sliding window is selected, it is limited to a small continuous period of time, and the data may not be representative. Moreover, the data has poor opportunity fairness. Based on this, this application considers optionally creating an additional random timeline, determining a sliding window on the random timeline, and further flexibly controlling the sliding window to smooth the data, improve data representativeness and opportunity fairness, thereby facilitating more reliable data selection for retention or elimination.

[0092] On a disordered timeline, time is still uniform, but because the time areas are reorganized based on differences in distribution density, time appears disordered.

[0093] S206: Determine the sliding window to be used this time on the out-of-order time axis according to the configured sliding window duration.

[0094] Initially, a position on the timeline can be specified as the initial starting point of the sliding window. This initial starting point can be unaffected by changes in the position of the time zone on the timeline. Without considering other control measures, each sliding window determined can slide a certain distance in the set positive sliding direction of the timeline compared to the sliding window determined previously. As time progresses, new time zones and data will be generated, and the out-of-order timeline can be updated as needed. Accordingly, when time advances sufficiently, the above-mentioned initial starting point can also be appropriately advanced and updated.

[0095] In one or more embodiments of the present specification, the sliding window may include multiple time zones as described above, thereby facilitating obtaining more dispersed data and breaking through the limitation of the most recent time.

[0096] Since the distribution density of cache data access points increases more smoothly on the disordered time axis, the further the sliding window moves forward, the more valuable data will be encountered in terms of quantity. However, some of these data will also be suitable for purging. Based on this, Figure 2 Based on the solution of , one or more embodiments of this specification further provide a more flexible active control solution for the sliding window, making it easier to reliably and efficiently capture data suitable for clearing on an out-of-order time axis while reducing the accidental damage to data with sufficient practical value. Figure 3 , which shows the process of the active control scheme.

[0097] Figure 3 The process in includes the following steps:

[0098] S302: After the samples to be eliminated are cleared from the cache, it is determined whether the degree of subsequent access to the samples to be eliminated is higher than a set degree.

[0099] If yes, then it can be considered that the actual value of the sample to be eliminated is sufficient and it can be left as is, and steps S304 to S308 can be continued. Otherwise, it can be considered that the elimination and cleaning operation is appropriate and the process can jump directly to step S310.

[0100] S304: If yes, update the time zone distribution on the disordered time axis, and move the sliding window back a distance in the opposite direction of the forward sliding direction on the disordered time axis as the next sliding window; wherein the distance is positively correlated with the degree of access.

[0101] In the case of accidental injury, the present application controls the sliding window to roll back on the disordered timeline, so that next time in the time zone where the distribution density of cache data access points is relatively sparse, it can continue to try to sort out and mine the samples to be eliminated; through the positive correlation relationship mentioned above, it can be controlled so that if the degree of accidental injury is higher, the degree of rollback can be relatively higher.

[0102] Such a rollback operation can be performed in particular together with the time area distribution update operation on the disordered time axis. Furthermore, when updating here, in addition to advancing the time, it is also possible to consider adjusting the granularity of the time area. For example, the granularity can be reduced, thereby more effectively changing the time area included in the sliding window.

[0103] In a simpler processing mode, the update of the out-of-order time axis may not be performed so promptly, but may be performed separately and in due course. In addition, the distance to be retracted may also be a fixed value.

[0104] S306: According to the retraction distance, a predefined retraction amount accumulation parameter is updated accordingly.

[0105] In one or more embodiments of this specification, a rollback accumulation parameter is designed to accumulate and measure the distance rolled back, thereby more flexibly and softly probing the limit that can be reached on the upper disordered time axis. If this limit is exceeded, it can be considered that the possibility of accidental injury will exceed the tolerance range.

[0106] S308: When the rollback amount accumulation parameter reflects that the accumulated rollback amount exceeds a set threshold, the sliding window is rolled back to the initial starting point on the out-of-order time axis, and the accumulated rollback amount is cleared.

[0107] The accumulated backoff distance can be directly used as the value of the backoff amount accumulation parameter. More reliably, the backoff distance can be adjusted based on some environmental factors of each backoff (for example, the distribution density involved, the importance of the accidentally damaged data, etc.), and then accumulated to calculate the value of the backoff amount accumulation parameter.

[0108] When the rollback amount exceeds the set threshold, it is considered to have reached the limit, thus completing a large cycle. The sliding window can be actively rolled back to the initial starting point and the accumulation of the rollback amount can be restarted.

[0109] S310: After determining whether the degree of access to the sample to be eliminated is higher than a set degree, if not, increasing the sliding speed of the sliding window in the next one or more sliding directions.

[0110] In the absence of accidental injury, the sliding speed of the sliding window is actively increased. This helps to reduce the time the sliding window stays in the time area with high distribution density and helps to test the limit more efficiently.

[0111] In one or more embodiments of the present specification, an additional check area is proposed, thereby introducing an additional calculation factor for the calculation of the data retention score, thereby helping to avoid the problem of insufficiently obvious data changes when the rate of change of the access point distribution density on the disordered timeline is too low.

[0112] The solution specifically includes: on the disordered time axis, after determining the sliding window to be used this time, select at least one time area outside the sliding window forward in the positive sliding direction as an additional verification area; for the target sample with access records concentrated in the sliding window among the cached data samples, determine whether the target sample has an access record in the time area; if so, calculate the retention and supplement score of the target sample based on the situation that the target sample has an access record in the time area. If the access degree is higher, the corresponding retention and supplement score can be higher, so as to increase the possibility of the target sample being retained.

[0113] When selecting the additional verification area, preferably, the selection can be randomly jumped forward. According to the current distribution density change rate, the selection can be jumped to the forward time area with a sufficiently high change rate or change amplitude.

[0114] In order to perform additional verification more targetedly, an additional verification area selection scheme based on similarity comparison is also provided here, which specifically includes: in the positive sliding direction, randomly jumping forward to a time area outside the sliding window; calculating the similarity between the distribution of access records in the time area and the distribution of access records in the sliding window, and judging whether the similarity is greater than the set threshold; if so, determining the time area as the additional verification area, which helps to increase the possibility of hitting the access records of the target sample, thereby giving the truly important data more opportunities to avoid accidental damage; otherwise, you can jump again to select other time areas until a suitable additional verification area is found.

[0115] According to the above description, in order to facilitate understanding and be more intuitive, one or more embodiments of this specification also provide Figure 1 A specific embodiment of the method is exemplarily implemented in a cluster group environment. Figure 4 、 Figure 5 Provide explanation.

[0116] Figure 4 A schematic diagram of a cache storage architecture of a cluster group provided in one or more embodiments of this specification.

[0117] exist Figure 4 In [1], the cache storage system is segmented and defined into multiple cluster groups. Each cluster group is divided into multiple clusters, and each cluster has multiple services. A cluster group has the same common configuration, which contains common parameters shared by all clusters within the cluster group.

[0118] Furthermore, a cluster can be divided into multiple businesses, and the same business can be deployed across clusters for disaster recovery. Each business has its own personalized configuration, which can improve the fit between the storage system and the business scenario, allowing storage to better serve the business.

[0119] For personalized configuration, set the corresponding parameters and their values ​​to represent. The following parameters and their values ​​are used as examples, and other names can also be used as needed:

[0120] The parameter "allkeys_or_volatile" is used to indicate the scope of the policy. When the parameter value is "allkeys", it indicates all data in the cache storage; when the parameter value is "volatile", it indicates data with an expiration time set in the cache storage.

[0121] The parameter "storage_type" is used to indicate the storage type. When the parameter value is "small," it indicates a small cache (less than 100,000 keys); when it is "middle," it indicates a medium cache (100,000 to 1,000,000 keys); and when it is "large," it indicates a large cache (greater than 1,000,000 keys). Based on the storage type selected for your business, you can rationally configure the sample set to balance score calculation and resource consumption, ensuring system stability. For example, set the number of samples in the sample set corresponding to "small" to 5, the number of samples in the sample set corresponding to "middle" to 15, and the number of samples in the sample set corresponding to "large" to 80.

[0122] The parameter "sliding_window" is used to indicate the parameter sliding window situation. It can indicate the window time range for the statistical data, such as the past three months or other specified time range. It can also indicate the sliding window mode in detail, such as whether to use a normal time window or a disordered time window (in this case, the window may contain multiple discontinuous time periods). The user can set the relevant conditions of the sliding window according to their needs, and only count and sort the data within the sliding window. If the data in the sample set is outside the sliding window, it can be randomly eliminated from the data that meets this type. This makes the elimination strategy calculation simpler and reduces performance loss.

[0123] The parameter "dynamic_weight" is used to indicate the weighting between data access time and data access frequency (this parameter enables the aforementioned weighting). This value ranges from 0 to 1. For example, the default setting of 0.5 indicates that data access time and data access frequency are weighted equally. This parameter can be specified by the business based on its own scenario. If data access frequency is more sensitive, the value can be lowered; if data access time is more sensitive, the value can be increased.

[0124] Figure 5 One or more embodiments of this specification provide Figure 1 The flow chart of a specific embodiment of the method is shown in the figure, using the parameters listed above as an example. In order to facilitate a clearer view, several process jump connection marks are set in the figure, the mark "① out" is connected to "① in", and the two marks "② out" are both connected to "② in", so that Figure 5 The left and right sides of the diagram form a complete flowchart.

[0125] In summary, Figure 5 The process mainly includes the following steps:

[0126] 1. The background period executes the process asynchronously and regularly determines whether the current cluster needs to be eliminated.

[0127] 2. If elimination is required, select the corresponding data source based on the "allkeys_or_volatile" parameter configuration, and select the corresponding number of cache data sample sets based on the "storage_type" parameter configuration, such as randomly selecting sample sets from the data source.

[0128] 3. Based on the "sliding_window" parameter configuration, determine the current sliding window and use the last access time of the data in the sample set to determine whether there is any data outside the sliding window. For example, if the sliding window is within the past three months, and the last access time of the data in the sample set was four months ago, randomly select a data item from the data outside the sliding window and eliminate it. If there is no data outside the sliding window, obtain the access frequency of each data item within the sliding window.

[0129] 4. Put the data in the sample set into linked list A from beginning to end according to the data access time from new to old, and calculate the score of each data for the data access time indicator, recorded as recently_score.

[0130] 5. Put the data in the sample set into linked list B from beginning to end according to the data access frequency from high to low, and calculate the score of each data for the data access frequency index, recorded as frequently_score.

[0131] There are various ways to calculate recently_score and frequently_score. Generally speaking, the recentness of data access time can be positively correlated with the recently_score, and the frequency of data access can be positively correlated with the frequently_score. In this case, assuming there are five data points in the sample set, for the five data points in linked list A, the recent_score values, from newest to oldest, can be calculated as: 5, 4, 3, 2, and 1, respectively. The same applies to linked list B. Of course, the score calculation scheme can be more complex, considering more factors, as needed.

[0132] 6. Calculate the final data retention score for each data point according to the exemplary formula, recorded as score:

[0133] "score=[dynamic_weight]*recently_score+[1-dynamic_weight]*frequently_score".

[0134] 7. According to the data retention score, put the data in the sample set from beginning to end into linked list C from high to low, eliminate the tail data in linked list C and jump to step 1.

[0135] As can be seen from the above embodiments, the solution based on this application helps to more reliably eliminate and clear data from the cache, at least in the following aspects:

[0136] Most cache storage systems in the industry share a single cluster. This application divides storage into different business segments, allowing each business to have its own configuration without interfering with each other, making management more refined.

[0137] This application takes into account both data access time and data access frequency to prevent accidental deletion of newly generated data and accidental deletion of frequently accessed data.

[0138] This application optimizes the statistics of data access frequency and adds more flexible sliding windows, especially out-of-order time windows. The design of sliding windows can greatly reduce the computational complexity, making data scoring simpler and more transparent, and especially helping to dynamically select data with greater reference value.

[0139] This application comprehensively considers the two indicators of data access time and data access frequency, and formulates a mixed factor calculation formula to make the evaluation criteria of hot data more accurate. At the same time, the business can also make the elimination strategy more suitable for the business scenario by adjusting the weights.

[0140] Based on the same idea, one or more embodiments of this specification also provide devices and apparatuses corresponding to the above methods, such as Figure 6 、 Figure 7 The apparatus and device can accordingly execute the above method and related optional solutions.

[0141] Figure 6 A schematic structural diagram of a data processing device based on cache storage provided in one or more embodiments of this specification, the device comprising:

[0142] The sample collection module 602 collects a cache data sample set in the cache;

[0143] The window determination module 604 determines the sliding window to be used this time according to the configured sliding window duration;

[0144] The sample determination module 606 determines whether there is a sample in the cache data sample set whose access records all fall outside the sliding window;

[0145] If the score calculation module 608 does not exist, then for the time range of the sliding window, calculate the data retention score of each sample according to the data access time and data access frequency of each sample in the cached data sample set;

[0146] The sample elimination module 610 selects one or more samples to be eliminated from the cached data sample set according to the data retention score, and clears the samples to be eliminated from the cache.

[0147] Optionally, after determining whether there are samples in the cache data sample set whose access records all fall outside the sliding window, the sample elimination module 610 selects one or more samples to be eliminated from the samples outside the sliding window, and clears the samples to be eliminated from the cache.

[0148] Optionally, the sample collection module 602 determines whether to collect a cache data sample set from expired data according to the configured policy effective range;

[0149] If so, determine the set of expired data in the cache;

[0150] According to the configured storage type, in the expired data set in the cache, a cache data sample set whose scale matches the storage type is collected.

[0151] Optionally, the score calculation module 608 determines that the configured model type is a simple type rather than a standard type before determining the sliding window to be used this time according to the configured sliding window duration;

[0152] Wherein, under the simple type, when the preset decay factor and increment factor are removed, the data retention score of each sample in the cache data sample set is calculated according to the data access time and data access frequency of each sample;

[0153] If it is determined that the configured model type is the standard type rather than the simple type, the score calculation module 608 uses the full life cycle instead of the sliding window, and calculates the data retention score of each sample according to the decay factor and the increment factor for the full life cycle, so as to determine the sample to be eliminated;

[0154] The decay factor is used to actively decay the old data access count according to a first set condition, and the increment factor is used to actively increment the new data access count according to a second set condition.

[0155] Optionally, the window determination module 604 obtains cache data access point distribution data on a normal time axis;

[0156] Recombining the different time zones on the normal time axis based on differences in distribution density reflected by the cached data access point distribution data to generate a random time axis, such that the distribution density of cached data access points increases smoothly in a set positive sliding direction of the random time axis;

[0157] According to the configured sliding window duration, the sliding window to be used this time is determined on the out-of-order time axis.

[0158] Optionally, the window determination module 604 determines whether the degree of subsequent access to the sample to be eliminated is higher than a set degree after the sample to be eliminated is cleared from the cache;

[0159] If so, update the time zone distribution on the disordered time axis, and move the sliding window back a certain distance in the opposite direction of the forward sliding direction on the disordered time axis to serve as the next sliding window;

[0160] The distance is positively correlated with the degree of access.

[0161] Optionally, the window determining module 604 updates a predefined backoff amount accumulation parameter according to the backoff distance after the backoff in the direction opposite to the forward sliding direction.

[0162] When the rollback amount accumulation parameter reflects that the accumulated rollback amount exceeds a set threshold, the sliding window is rolled back to the initial starting point on the out-of-order time axis, and the accumulated rollback amount is cleared.

[0163] Optionally, after determining whether the degree of access to the sample to be eliminated is higher than a set degree, the window determination module 604 increases the sliding speed of the sliding window in the next one or more sliding positive directions if not.

[0164] Optionally, after determining the sliding window to be used this time on the out-of-order time axis, the window determination module 604 selects at least one time region outside the sliding window forward in the positive sliding direction as an additional verification region;

[0165] For a target sample having an access record concentrated in the sliding window, the cached data sample is determined to determine whether the target sample has an access record in the time zone;

[0166] If yes, the score calculation module calculates the retention and supplement score of the target sample according to whether the target sample has access records in the time zone;

[0167] The score calculation module calculates the data retention score of the target sample according to the retention supplement score of the target sample, as well as the data access time and the data access frequency.

[0168] Optionally, the window determination module 604 randomly jumps forward in the positive sliding direction to a time region outside the sliding window;

[0169] Calculating the similarity between the distribution of access records in the time zone and the distribution of access records in the sliding window, and determining whether the similarity is greater than a set threshold;

[0170] If yes, determining the time zone as an additional verification zone;

[0171] Otherwise, jump again to select another time zone.

[0172] Optionally, there are multiple caches, which are divided from a larger storage system according to different services respectively corresponding to them;

[0173] The plurality of caches are configured differently and personalized according to the differences in the services they correspond to, and the configuration involves the calculation of the data retention score.

[0174] Optionally, the configuration further includes: weights set for data access time and data access frequency respectively; the weights are configured for the cache corresponding to the business based on the differences in sensitivity of different businesses to data access time and data access frequency;

[0175] The score calculation module 608 calculates a score for the data access indicator based on the data access time of the sample, and calculates a score for the data access frequency based on the data access frequency of the sample;

[0176] The data retention score of the sample is weightedly calculated based on the score for the data access indicator, the score for the data access frequency, and their respective weights.

[0177] Figure 7 A schematic diagram of a data processing device based on cache storage provided in one or more embodiments of this specification, the device comprising:

[0178] at least one processor; and,

[0179] a memory communicatively connected to the at least one processor; wherein,

[0180] The memory stores instructions executable by the at least one processor, wherein the instructions are executed by the at least one processor to enable the at least one processor to perform:

[0181] Collecting a cache data sample set in the cache;

[0182] Determine the sliding window to be used this time based on the configured sliding window duration;

[0183] Determine whether there is a sample in the cache data sample set whose access records all fall outside the sliding window;

[0184] If not, then for the time range of the sliding window, calculate the data retention score of each sample according to the data access time and data access frequency of each sample in the cached data sample set;

[0185] According to the data retention score, one or more samples to be eliminated are screened out from the cache data sample set, and the samples to be eliminated are cleared from the cache.

[0186] Based on the same idea, one or more embodiments of this specification further provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0187] Collecting a cache data sample set in the cache;

[0188] Determine the sliding window to be used this time based on the configured sliding window duration;

[0189] Determine whether there is a sample in the cache data sample set whose access records all fall outside the sliding window;

[0190] If not, then for the time range of the sliding window, calculate the data retention score of each sample according to the data access time and data access frequency of each sample in the cached data sample set;

[0191] According to the data retention score, one or more samples to be eliminated are screened out from the cache data sample set, and the samples to be eliminated are cleared from the cache.

[0192] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0193] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0194] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0195] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0196] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0198] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0200] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0201] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0202] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0203] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0204] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0205] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0206] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0207] The foregoing description is merely one or more embodiments of this specification and is not intended to limit this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of one or more embodiments of this specification are intended to be within the scope of the claims of this specification.

Claims

1. A data processing method based on cache storage, comprising: Collecting a cache data sample set in the cache; Determine the sliding window to be used this time based on the configured sliding window duration; Determine whether there is a sample in the cache data sample set whose access records all fall outside the sliding window; If not, then for the time range of the sliding window, calculate the data retention score of each sample according to the data access time and data access frequency of each sample in the cached data sample set; According to the data retention score, one or more samples to be eliminated are screened out from the cache data sample set, and the samples to be eliminated are cleared from the cache.

2. The method of claim 1, wherein determining whether there are any access records after a sample whose access records all fall outside the sliding window in the cached data sample set further comprises: If so, one or more samples to be eliminated are screened out from the other samples, and the samples to be eliminated are cleared from the cache.

3. The method according to claim 1, wherein collecting a cache data sample set in the cache comprises: Determine whether to collect cached data sample sets from expired data based on the configured policy effectiveness scope; If so, determine the set of expired data in the cache; According to the configured storage type, in the expired data set in the cache, a cache data sample set whose scale matches the storage type is collected.

4. The method according to claim 1, before determining the sliding window to be used this time based on the configured sliding window duration, the method further comprises: Make sure the configured model type is a simple type, not a standard type; Wherein, under the simple type, when the preset decay factor and increment factor are removed, the data retention score of each sample in the cache data sample set is calculated according to the data access time and data access frequency of each sample; If it is determined that the configured model type is the standard type but not the simple type, the method further includes: The sliding window is replaced by the full life cycle, and for the full life cycle, the data retention score of each sample is calculated according to the decay factor and the increment factor, so as to determine the samples to be eliminated; The decay factor is used to actively decay the old data access count according to a first set condition, and the increment factor is used to actively increment the new data access count according to a second set condition.

5. The method according to claim 1, wherein determining the sliding window to be used this time based on the configured sliding window duration specifically comprises: Get the cache data access point distribution data on the normal timeline; Recombining the different time zones on the normal time axis based on differences in distribution density reflected by the cached data access point distribution data to generate a random time axis, such that the distribution density of cached data access points increases smoothly in a set positive sliding direction of the random time axis; According to the configured sliding window duration, the sliding window to be used this time is determined on the out-of-order time axis.

6. The method according to claim 5, after clearing the samples to be eliminated from the cache, the method further comprises: Determining whether the degree of access to the sample to be eliminated is higher than a set degree; If so, update the time zone distribution on the disordered time axis, and move the sliding window back a certain distance in the opposite direction of the forward sliding direction on the disordered time axis to serve as the next sliding window; The distance is positively correlated with the degree of access.

7. The method according to claim 6, wherein after retreating a distance in the direction opposite to the forward sliding direction, the method further comprises: According to the distance of the retreat, a predefined retreat amount accumulation parameter is updated accordingly; When the rollback amount accumulation parameter reflects that the accumulated rollback amount exceeds a set threshold, the sliding window is rolled back to the initial starting point on the out-of-order time axis, and the accumulated rollback amount is cleared.

8. The method according to claim 6, wherein after determining whether the degree of subsequent access to the sample to be eliminated is higher than a set degree, the method further comprises: If not, the sliding speed of the sliding window in the next one or more sliding directions is increased.

9. The method according to claim 5, wherein after determining the sliding window to be used this time on the out-of-order time axis, the method further comprises: In the positive sliding direction, selecting at least one time region outside the sliding window forward as an additional verification region; For a target sample having an access record concentrated in the sliding window, the cached data sample is determined to determine whether the target sample has an access record in the time zone; If yes, then calculating the retention and supplement score of the target sample according to whether the target sample has access records in the time period; Calculating the data retention score of each sample according to the data access time and data access frequency of each sample in the cache data sample set specifically includes: The data retention score of the target sample is calculated according to the retention supplement score of the target sample, as well as the data access time and the data access frequency.

10. The method according to claim 9, wherein the selecting, in the positive sliding direction, at least one time region outside the sliding window forward as an additional verification region specifically comprises: In the positive sliding direction, randomly jump forward to a time region outside the sliding window; Calculating the similarity between the distribution of access records in the time zone and the distribution of access records in the sliding window, and determining whether the similarity is greater than a set threshold; If yes, determining the time zone as an additional verification zone; Otherwise, jump again to select another time zone.

11. The method according to any one of claims 1 to 10, wherein the cache is multiple and is divided from a designated storage system according to different services respectively corresponding to the cache; The plurality of caches are configured differently and personalized according to the differences in the services they correspond to, and the configuration involves the calculation of the data retention score.

12. The method of claim 11, wherein the configuring further comprises: The weights set for data access time and data access frequency respectively; The weight is configured for the cache corresponding to the business based on the differences in sensitivity of different businesses to data access time and data access frequency; Calculating the data retention score of each sample according to the data access time and data access frequency of each sample in the cache data sample set specifically includes: Calculating a score for a data access indicator based on the data access time of the sample, and calculating a score for a data access frequency based on the data access frequency of the sample; The data retention score of the sample is weightedly calculated based on the score for the data access indicator, the score for the data access frequency, and their respective weights.

13. A data processing device based on cache storage, comprising: A sample collection module collects a cache data sample set in the cache; The window determination module determines the sliding window to be used this time according to the configured sliding window duration; A sample judgment module, judging whether there is a sample in the cache data sample set whose access records all fall outside the sliding window; If the score calculation module does not exist, the score calculation module calculates the data retention score of each sample in the cached data sample set according to the data access time and data access frequency of each sample in the time range of the sliding window; The sample elimination module selects one or more samples to be eliminated from the cached data sample set according to the data retention score, and clears the samples to be eliminated from the cache.

14. The device as described in claim 13, the sample elimination module, after determining whether there are samples in the cache data sample set whose access records all fall outside the sliding window, if so, screens out one or more samples to be eliminated from the samples outside the sliding window, and clears the samples to be eliminated from the cache.

15. The apparatus of claim 13, wherein the score calculation module determines that the configured model type is a simple type rather than a standard type before determining the sliding window to be used this time based on the configured sliding window duration; in, Under the simple type, after removing the preset decay factor and increment factor, the data retention score of each sample in the cache data sample set is calculated according to the data access time and data access frequency of each sample; If it is determined that the configured model type is the standard type rather than the simple type, the score calculation module uses the full life cycle to replace the sliding window, and calculates the data retention score of each sample according to the decay factor and the increment factor for the full life cycle, so as to determine the samples to be eliminated; The decay factor is used to actively decay the old data access count according to a first set condition, and the increment factor is used to actively increment the new data access count according to a second set condition.

16. The apparatus according to claim 13, wherein the window determination module obtains cache data access point distribution data on a normal time axis; Recombining the different time zones on the normal time axis based on differences in distribution density reflected by the cached data access point distribution data to generate a random time axis, such that the distribution density of cached data access points increases smoothly in a set positive sliding direction of the random time axis; According to the configured sliding window duration, the sliding window to be used this time is determined on the out-of-order time axis.

17. The apparatus of claim 16, wherein the window determination module, after determining the sliding window to be used this time on the out-of-order time axis, selects at least one time region outside the sliding window in the forward sliding direction as an additional verification region; For a target sample having an access record concentrated in the sliding window, the cached data sample is determined to determine whether the target sample has an access record in the time zone; If yes, the score calculation module calculates the retention and supplement score of the target sample according to whether the target sample has access records in the time zone; The score calculation module calculates the data retention score of the target sample according to the retention supplement score of the target sample, as well as the data access time and the data access frequency.

18. The device according to any one of claims 13 to 17, wherein the cache comprises a plurality of caches which are divided from a designated storage system according to the different services respectively corresponding thereto; The plurality of caches are configured differently and personalized according to the differences in the services they correspond to, and the configuration involves the calculation of the data retention score.

19. A data processing device based on cache storage, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, wherein the instructions are executed by the at least one processor to enable the at least one processor to perform: Collecting a cache data sample set in the cache; Determine the sliding window to be used this time based on the configured sliding window duration; Determine whether there is a sample in the cache data sample set whose access records all fall outside the sliding window; If not, then for the time range of the sliding window, calculate the data retention score of each sample according to the data access time and data access frequency of each sample in the cached data sample set; According to the data retention score, one or more samples to be eliminated are screened out from the cache data sample set, and the samples to be eliminated are cleared from the cache.

Images