A method for eliminating and maintaining hot data in a distributed cache system

Abstract

A method for eliminating and maintaining hotspot data in a distributed cache system includes: obtaining data access frequency, temporal locality score, business weight, and real-time data heat value to generate a dynamic priority score for the data; determining invalid data based on the dynamic priority score; dividing the data into hot zone data, warm zone data, and cold zone data; when cache space is insufficient, preferentially eliminating data in the cold zone data with the lowest dynamic priority score and a retention benefit value below a preset threshold; if cache space is still insufficient, eliminating the warm zone data based on the most recent access time of the warm zone data; encapsulating the cold zone eliminated data, warm zone eliminated data, and invalid data into a final transaction message; broadcasting the final transaction message via a preset protocol, and performing distributed consistency maintenance operations on the final transaction message via receiving nodes. The present invention can achieve intelligent and efficient cache data elimination and consistency maintenance management.

Description

Technical Field

[0001] The present invention relates to the technical field of cache management in a big data environment, and in particular to a method for eliminating and maintaining hotspot data in a distributed cache system. Background Art

[0002] With the rapid development of big data technology, the storage and processing of massive amounts of data has become an industry challenge. In big data environments, distributed caching technology has been widely used as an important means to improve system response speed and reduce data access latency. Traditional cache management methods are usually based on static algorithms, such as LRU (Least Recently Used) or LFU (Least Frequently Used). Although these methods are effective in certain scenarios, they have certain limitations when faced with the complex business needs of massive amounts of data. In particular, when processing real-time data and business-critical data, dynamic factors such as data access frequency, temporal locality, and business priority cannot be fully considered, resulting in low cache space utilization and unreasonable cache invalidation strategies.

[0003] Currently, many existing cache replacement methods do not fully consider the dynamic priority of data. When cache space is insufficient, they often rely on static elimination strategies, such as the least recently accessed data or the least used data. When responding to real-time data popularity changes and business-critical data needs, they often cannot make rapid adjustments, resulting in cache failure and inefficient replacement processes. Summary of the Invention

[0004] In view of this, the present invention provides a hotspot data elimination and maintenance method in a distributed cache system to dynamically adjust the priority and elimination strategy of cached data to achieve more intelligent and efficient cache data elimination and consistency maintenance management.

[0005] A method for eliminating and maintaining hotspot data in a distributed cache system, comprising:

[0006] Step 1: Obtain the data's access frequency, temporal locality score, business weight, and real-time data popularity value, and then generate a dynamic priority score for the data in the distributed cache system;

[0007] Step 2: Determine invalid data based on the dynamic priority score of data in the distributed cache system;

[0008] Step 3: Based on the dynamic priority scores of all data in the distributed cache system, the data in the system is divided into hot zone data, warm zone data, and cold zone data. When cache space is insufficient, data in the cold zone with the lowest dynamic priority score and a retention benefit value below the preset threshold is eliminated first. If cache space is still insufficient, data in the warm zone is eliminated based on the most recent access time.

[0009] Step 4: Encapsulate the cold spot area elimination data, warm spot area elimination data and invalid data into the final transaction message;

[0010] Step 5: The final transaction message is broadcast through the preset protocol, and then the receiving nodes perform distributed consistency maintenance operations on the final transaction message.

[0011] The method for eliminating and maintaining hotspot data in a distributed cache system provided by the present invention has the following beneficial effects:

[0012] The present invention generates a dynamic priority score for data in a distributed cache system by comprehensively considering dynamic factors such as data access frequency, time locality score, business weight and real-time data heat value. It can more accurately evaluate the importance of data in the cache, dynamically adjust the priority and replacement strategy of cached data, and thus optimize the cache invalidation and replacement process. By real-time updating and integrating multi-dimensional scores, this method can effectively improve the performance of the cache system, ensure the priority retention of key business data and high-frequency data, and reasonably eliminate inefficient and cold spot data, thereby realizing more intelligent and efficient cache data elimination and consistency maintenance management in the distributed cache system, and improving the utilization rate and response speed of cache space. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flow chart of a method for eliminating and maintaining hotspot data in a distributed cache system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.

[0015] See also Figure 1 An embodiment of the present invention provides a method for eliminating and maintaining hotspot data in a distributed cache system, comprising steps 1 to 5:

[0016] Step 1: Obtain the data's access frequency, temporal locality score, business weight, and real-time data heat value, and then generate a dynamic priority score for the data in the distributed cache system.

[0017] Among them, data access frequency evaluates how frequently data is accessed based on the number of times the data is accessed within a period of time.

[0018] Temporal locality is to determine the timeliness of data access and its recent access history by analyzing the time interval between data accesses.

[0019] Business weight is used to determine the importance of the data to be queried in a specific business scenario based on the urgency of the business needs.

[0020] Real-time data heat value refers to the heat evaluation of real-time data to determine its criticality in the real-time data stream.

[0021] It can be understood that dynamic priority scoring refers to a comprehensive scoring of all cached data based on various indicators (such as access frequency, time locality, etc.), thereby forming a priority assessment for each data. Low-priority data will be marked as invalid data.

[0022] In this embodiment, step 1 specifically includes:

[0023] Count the number of times data in the distributed cache system is accessed within a preset time window as the access frequency;

[0024] Determine the interval between the last access time of the data in the distributed cache system and the current time, and then determine the temporal locality score of the data in the distributed cache system. The shorter the interval, the higher the temporal locality score. Specifically, in this embodiment, the temporal locality score = 1 / (1 + time interval);

[0025] Assign business weights based on pre-configured business type tags and a preset tag-weight data table. Specifically, the weight of core business is 1.2 times that of non-core business. The business type tags of the data to be queried are read from the pre-configured metadata database. A weight of α=1.2 is assigned to the core business data, and a weight of β=1.0 is assigned to the non-core query business data.

[0026] Determine the real-time data heat value of the data in the distributed cache system based on a preset method, and determine the penalty value of high-frequency but currently unaccessed data;

[0027] The data access frequency, temporal locality score, business weight, and real-time data heat value are integrated to generate a dynamic priority score. In this embodiment, the dynamic priority score = (access frequency × business weight) + (temporal locality score × real-time data heat value) - penalty value.

[0028] It should be noted that the penalty value is related to factors such as the length of the inaccessible time and the access frequency. The longer the inaccessible time, the greater the penalty value. Typically, the inaccessible time can be measured by the difference between the last access time of the data and the current time. Regarding the access frequency, even if some data has not been accessed, if its access frequency is high, it may still occupy cache space. Therefore, the size of the penalty value is related to the access frequency. Therefore, the penalty value formula can be expressed as: Penalty value = γ × (current time - last access time) × access frequency. γ is a weighting factor that indicates the degree of impact of the inaccessible time on the penalty value and can be adjusted according to actual needs; (current time - last access time) represents the length of time the data has not been accessed; and the access frequency indicates how often the data is accessed. Data with a high access frequency will have a larger penalty value even if it has not been accessed.

[0029] The method of determining the real-time data heat value of the data in the distributed cache system based on a preset method includes:

[0030] Count the number of accesses to data in the distributed cache system within a sliding window of preset length and perform weighting;

[0031] The weighted cumulative number of visits is used to generate an initial heat value, which is then corrected and standardized based on the query business type to obtain a real-time data heat value.

[0032] In this embodiment, the number of accesses to data in the cache system within a sliding window of a preset length is counted and weighted, including: first, setting a sliding window of a fixed length for real-time counting of accesses to each data item in the cache system, for example, if a sliding window is 10 seconds long, then at each moment window the number of accesses to each data item in the cache within the past 10 seconds is counted; then, the number of each access is weighted, for example, more recent accesses are given a higher weight and more distant accesses are given a lower weight; an exponential weighting method can usually be used, with the weight decaying over time; this weighting method can reflect the instantaneous access frequency of the cached data.

[0033] In this embodiment, the weighted number of accesses is accumulated to generate an initial heat value, including: after obtaining the weighted number of accesses, these values ​​are accumulated to obtain an initial heat value. This initial heat value represents the immediate activity of the data in the cache. For example, if a certain data is accessed 5 times and weighted 3 times within a time period, its heat value is 15. The purpose of this step is to quantify the activity of the cached data and provide basic data for subsequent processing.

[0034] In this embodiment, the initial heat value is corrected and standardized according to the business type to obtain the real-time heat value, including: adjusting and standardizing the initial heat value according to different business scenarios. For example, in some application scenarios, the access frequency of certain data may be too high or too low due to the business characteristics. In this case, the heat value needs to be corrected according to the specific needs of the business. For example, the access frequency of voltage, current and power data collected by sensors in the industrial monitoring field may be higher than other data such as temperature and humidity. The corrected heat value can be used to further optimize the cache strategy; then, the corrected heat value is standardized to ensure that the heat values ​​of each data can be effectively compared and meet the requirements of the cache strategy.

[0035] Based on the above technical solution: By dynamically calculating the real-time heat value of the data based on the number of data accesses and the weighting mechanism within the sliding window, the heat changes of the cached data can be accurately evaluated. The initial heat value is generated by accumulating the weighted number of accesses, and then corrected and standardized according to the business type, so that the heat value is more in line with the actual query business needs. This method can promptly reflect the changing trends of data access, ensure that the cache system can quickly adapt to the heat changes of real-time data, effectively improve the utilization of cache space, optimize cache invalidation and replacement strategies, and thus improve the response speed and processing efficiency of the distributed cache system.

[0036] Step 2: Determine invalid data based on the dynamic priority score of data in the distributed cache system.

[0037] Among them, step 2 specifically includes:

[0038] Set a dynamic priority score threshold T1 to filter data from the distributed cache system with a score lower than T1;

[0039] For data with scores lower than T1, a multi-level survival period is assigned, and the lower the score, the shorter the survival period;

[0040] Monitor the version number change event of the data source and capture the version number increment signal;

[0041] When a version number increment signal change is detected, the dynamic priority score of the corresponding cache system data is forcibly reset to zero;

[0042] Data whose dynamic priority score is zero or whose life cycle expires is marked as invalid, thereby obtaining invalid data.

[0043] In this embodiment, the dynamic priority scoring threshold T1 is a pre-set scoring standard used to prioritize data in the cache. Data below threshold T1 is considered low-priority data and is suitable for invalidation. For example, the value of T1 can be adjusted based on business needs and cache space availability to ensure that inefficient, expired, or infrequently used data is cleared first.

[0044] In this embodiment, the multi-level life cycle means that different survival times are assigned to data based on its dynamic priority score. Data with low scores will have a shorter life cycle and will be eliminated first to ensure efficient use of cache space. For example, data with very low scores may only remain in the cache for a few minutes, while data with high scores may remain in the cache for a longer time.

[0045] In this embodiment, the version number increment signal means that when the data in the data source changes (for example, the version number increases or the data is updated), the system detects the data update by monitoring the change event, and thus adopts the corresponding cache update or invalidation strategy. For example, if the data source version number is updated, it means that the data content has changed, and the corresponding data in the cache needs to be updated.

[0046] In this embodiment, forced zeroing means that when a version number change event occurs, the priority score of the corresponding data is immediately cleared to ensure that expired or no longer valid data does not continue to occupy cache space. For example, when a certain data in the data source is updated, the corresponding data in the cache will no longer be considered high-priority data, forcing it to be re-evaluated or eliminated.

[0047] According to the above technical solution, by setting a dynamic priority score threshold, data with a score below the threshold is filtered out, and a lifecycle is assigned based on the score to ensure that low-priority data expires in a timely manner. By monitoring the version number change events of the data source, the priority score of the data is forced to zero when the version number increases, effectively avoiding the retention of outdated data. This method can flexibly handle the expiration and update of cached data, ensuring that the data in the cache system is always the latest and most relevant, improving the utilization efficiency of cache space, while reducing the impact of invalid data on system performance, and improving the processing efficiency of the distributed cache system.

[0048] Step 3: Divide the data in the distributed cache system into hot spot data, warm spot data, and cold spot data based on the dynamic priority scores of all data in the system. When the cache space is insufficient, prioritize eliminating the data in the cold spot data with the lowest dynamic priority score and a retention benefit value lower than the preset threshold. If the cache space is still insufficient, eliminate the data in the warm spot data based on the most recent access time of the warm spot data.

[0049] The hotspot data includes: data with a high dynamic priority score in the cache and frequently accessed data.

[0050] The data in the warm zone includes: data with moderate access frequency and lower priority.

[0051] The data in the cold zone includes: data with the lowest dynamic priority score and the least access frequency.

[0052] The retention benefit value measures the benefit of data existing in the cache. If it is lower than the preset threshold, it will be preferentially eliminated.

[0053] Among them, step 3 specifically includes:

[0054] Set the hot zone threshold H and the cold zone threshold T2, and divide the ranges of the hot zone data, warm zone data, and cold zone data;

[0055] Allocate a fixed cache space for the hot zone data and prohibit active elimination. When the cache space is insufficient, only allow natural competition updates through the dynamic priority scores of the data, and eliminate the hot zone data with the lowest dynamic priority score;

[0056] Determine the retention benefit value of the cold zone data;

[0057] Screen the cold zone data with a retention benefit value lower than the preset threshold and the lowest dynamic priority score to generate an elimination queue. If the scores of multiple data are the same, further sort them in ascending order according to the data size;

[0058] Release the cache space occupied by the cold zone data based on the order of the elimination queue. Each time a data is released, update the data in the cold zone and record the elimination log, and real-time detect the remaining cache space. If the requirements are met, terminate the elimination;

[0059] If the cache space is still insufficient after releasing the cold zone data, perform the elimination of the preset mechanism according to the sorting of the most recent access timestamps of the warm zone data.

[0060] In this embodiment, setting the hot zone threshold H and the cold zone threshold T2, and dividing the ranges of the hot zone, warm zone, and cold zone includes: dynamically adjusting the hot zone threshold H and the cold zone threshold T2 according to the system load and historical access patterns, ensuring that: the hot zone threshold H is always higher than the average dynamic priority score of the warm zone; the cold zone threshold T2 is lower than the lowest dynamic priority score of the warm zone; hot zone: dynamic priority score ≥ H; warm zone: T2 ≤ dynamic priority score < H; cold zone: dynamic priority score < T2.

[0061] In this embodiment, only allowing natural competition updates through the dynamic priority scores of the data means that for new data to enter the hot zone, it needs to meet the score ≥ H; if the space is insufficient, eliminate the data with the lowest score in the current hot zone and release the space for higher-score data to be loaded.

[0062] In this embodiment, the retention benefit value=dynamic priority score / (data size×storage medium unit cost), where the storage medium unit cost is pre-configured according to the SSD or HDD type.

[0063] In this embodiment, the preset mechanism of elimination is executed by sorting the data in the warm spot area according to the most recent access timestamp, including: executing warm spot area elimination: extracting the most recent access timestamps of all data in the warm spot area; sorting by timestamp in ascending order (the least recently accessed data comes first); eliminating data in sequence until the space meets the demand; conflict handling: if the timestamps are the same, eliminate them in ascending order of score.

[0064] In this embodiment, fixed cache space is allocated for hotspot data and active elimination is prohibited. Space adjustment only occurs in the hotspot data update scenario. That is, when new data needs to enter the hotspot and the cache space is insufficient, the data is updated naturally through dynamic priority scoring, and the hotspot data with the lowest dynamic priority score is eliminated. This process is independent of the elimination logic when global cache space is insufficient. The elimination of hotspot data is only triggered when its own dedicated space, that is, the fixed cache space, is insufficient. Because it has the highest dynamic priority score, it does not participate in the cold spot and warm spot elimination process when global cache space is insufficient.

[0065] In this embodiment, when global cache space is insufficient, cache space occupied by data in cold zones is prioritized for release based on the order of the eviction queue. Each time a piece of data is released, the cold zone data is updated and an eviction log is recorded. Remaining cache space is monitored in real time, and eviction is terminated if the required space is met. The global cache space represents the total capacity of the cache system, while the fixed cache space in hot zones is a designated portion of the global space (similar to a "reserved area"). For example, if the global cache space is 100GB, the fixed cache space allocated to the hot zone might be 40GB, leaving 60GB for dynamic allocation of data to warm and cold zones. When global cache space is insufficient, this actually refers to the remaining space (warm zone + cold zone available space) excluding the fixed space in the hot zone. In this case, cold zone data is prioritized for eviction, followed by warm zone data, without affecting the fixed space in the hot zone. In other words, the fixed space in the hot zone does not participate in the passive eviction when global cache space is insufficient.

[0066] The above technical solution achieves more flexible and efficient cache invalidation and replacement management by dividing cache data into hot, warm, and cold zones based on dynamic priority scores. Hot zone data is retained first and updated through dynamic scoring, ensuring that important data always resides in the cache. Cold zone data is eliminated based on its retention benefit value and score, prioritizing inefficient data and optimizing cache space utilization. If cold zone data cannot meet demand, warm zone data is eliminated based on its most recent access time, ensuring that cache space is always effectively utilized. This multi-level elimination mechanism can effectively improve cache efficiency and system performance in distributed cache systems.

[0067] Step 4: Encapsulate the cold spot area elimination data, warm spot area elimination data and invalid data into the final transaction message.

[0068] The final transaction message contains information about all eliminated and invalid data and is packaged in a preset format to facilitate subsequent broadcasting and processing.

[0069] Among them, step 4 specifically includes:

[0070] Determine the data to be packaged based on the cold spot area elimination data, warm spot area elimination data and failure data;

[0071] Obtain the operation type of the data to be encapsulated, extract the corresponding identifier from the preset cache metadata based on the operation type of the data to be encapsulated, assign a unique code to the data to be encapsulated for each operation type, and write the unique code, operation type, and identifier of the data to be encapsulated into the message header;

[0072] Based on the data identifier corresponding to the data to be encapsulated, the latest version number of the data source is obtained from the preset data source and the latest version number of the data source is bound to the data to be encapsulated. If the data source version number has not changed, the locally cached version number is used. At the same time, the message header of the data to be encapsulated is compressed and redundant fields are deleted.

[0073] Based on the identifier corresponding to the data to be encapsulated, the corresponding score of the local cache is obtained from the preset data source, and the consistency check is performed based on the operation type of the data to be encapsulated, and conflict events are marked;

[0074] Conflict handling is performed on conflict events based on the preset conflict handling logic;

[0075] Determine the sending priority and processing strategy of the data to be encapsulated based on the operation type and encapsulate the data to be encapsulated to obtain a transaction message;

[0076] Serialize the encapsulated transaction message according to the preset binary format, generate a binary stream and encode it in the preset format;

[0077] Add a checksum to the encoded binary stream and append the checksum to the end of the transaction message to generate the final transaction message.

[0078] In this embodiment, the data to be packaged is determined based on the cold spot area elimination data, the warm spot area elimination data and the invalid data queue, including: for the cold spot area elimination data, obtaining the data identification, dynamic priority score, storage efficiency value and release time; for the warm spot area elimination data, extracting the data identification, dynamic priority score, most recent access timestamp and release reason; for the invalid data, collecting the data identification, the dynamic priority score after zeroing (forced to be 0), and the invalidation trigger version number. Exception handling: If the data identification conflicts (such as existing in the elimination and invalidation queues at the same time), the invalidation event is used as the highest priority to overwrite the elimination operation. Technical details: The data identification is generated using SHA-256 hash to ensure global uniqueness; the elimination queue is managed through a lock-free linked list to avoid concurrency conflicts.

[0079] In this embodiment, the operation type of the data to be packaged is obtained, and the operation types are divided into three categories: cold spot area elimination, code 01, indicating that the data is eliminated due to low efficiency in the cold spot area; warm spot area elimination, code 02, indicating that the data is eliminated due to the LRU strategy of the warm spot area; failure event, code 03, indicating that the data is invalid due to version change or expiration of the life cycle.

[0080] In this embodiment, the message header serves as the basic identification part of data transmission and is used for identification and parsing of subsequent data processing; the message header includes a 2-bit operation type code, a 4-bit protocol version number and a 10-bit reserved field, and the total length of the message header is fixed at 2 bytes.

[0081] In this embodiment, the message header of the encapsulated data is compressed and redundant fields are deleted, including: field deletion: redundant fields such as reserved bits and retention flags are removed, and only the operation type, protocol version and reference timestamp are retained; timestamp compression: differential encoding is used, with the message generation time as the reference, and subsequent timestamps are recorded as offsets (unit: milliseconds); data alignment: the compressed message header is aligned to 4 bytes, and the total length is fixed to 8 bytes (2 bits of operation type + 4 bits of version + 26 bits of timestamp difference). Technical details: Huffman coding is used to further compress high-frequency operation types (such as failure events).

[0082] In this embodiment, consistency verification is performed based on the operation type of the data to be packaged, including: elimination operation (cold / warm spot area): the dynamic priority score in the transaction message is consistent with the score of the data in the local cache; failure event: the dynamic priority score in the transaction message must be 0 to prevent dirty data from being overwritten due to network delay or concurrent operations (for example, after node A eliminates data X, node B is not synchronized and still retains the old score); score consistency verification: elimination operation: verify that the message score is strictly consistent with the local score; failure operation: verify that the message score is 0, otherwise it is marked as illegal; partition legitimacy verification: elimination operation: verify that the partition (cold / warm spot area) to which the data currently belongs matches the operation type; failure operation: allow cross-partition execution; conflict classification: score conflict: the message is inconsistent with the local score; partition conflict: the operation type does not match the data partition (such as eliminating the hot spot area); illegal failure: the failure event score is not 0.

[0083] In this embodiment, conflict event types include inconsistent scores, mismatched operation types, and partition errors. The marked conflict events will trigger a subsequent data resynchronization process.

[0084] In this embodiment, the sending priority and processing strategy for the data to be encapsulated are determined, including: priority level: failure event (03) > cold spot elimination (01) > warm spot elimination (02); queue implementation: high-priority messages are stored in a memory queue and sent in real time by an independent thread; low-priority messages are stored in a disk buffer queue and processed asynchronously in batches; preemption mechanism: high-priority messages can interrupt low-priority transmissions, ensuring that failure events take effect immediately. Technical details: The Linux kernel's cgroups mechanism is used to limit the bandwidth usage of low-priority messages.

[0085] According to the above technical solution, cold spot zone elimination data, warm spot zone elimination data, and invalid data are encapsulated into final transaction messages, ensuring that data is properly processed and transmitted during the cache elimination process. By extracting information such as the operation type, unique code, version number, and score, consistency verification and conflict resolution are performed on the encapsulated data to ensure data integrity and consistency. Serialization in binary format and the addition of checksums enhance the reliability and security of data transmission. Furthermore, the priority and processing strategies during the encapsulation process optimize the efficiency of transaction message transmission, ensuring the efficiency and stability of cache invalidation and replacement in the distributed cache system.

[0086] Step 5: The final transaction message is broadcast through the preset protocol, and then the receiving nodes perform distributed consistency maintenance operations on the final transaction message.

[0087] The preset protocol refers to broadcasting messages through a pre-agreed protocol to ensure data consistency and transaction reliability.

[0088] Distributed consistency maintenance refers to the consistency verification of broadcast transaction messages by receiving nodes to ensure the consistency of data status within the system.

[0089] Among them, step 5 specifically includes:

[0090] Establish a transaction message broadcast channel based on the preset broadcast protocol, and configure the message push node and receiving node list;

[0091] The final transaction message with the encapsulated and checksum added is pushed to each receiving node through the broadcast channel;

[0092] After receiving the final transaction message, the receiving node verifies the integrity and correctness of the final transaction message. If the verification fails, a retransmission request is sent. If the verification passes, transaction preprocessing is performed.

[0093] Based on the operation type, data identifier, and version information in the final transaction message, the corresponding data copy is searched in the local cache and version consistency is checked.

[0094] If the version consistency check passes, the receiving node will perform a pre-update operation on the corresponding data copy according to the operation type indicated by the transaction message, and generate a pre-commit record locally;

[0095] The receiving node participates in the distributed consensus decision based on the preset consensus protocol and completes the formal submission of the transaction message;

[0096] After formal submission, the local cache status is updated and the pre-submission record is cleared. At the same time, the final transaction message processing result is fed back to the broadcast node to complete the transaction loop and implement distributed consistency maintenance operations.

[0097] In this embodiment, the preset broadcast protocol is to ensure the reliable transmission of transaction messages. A preset communication protocol (such as gRPC, Kafka, etc.) can be used to ensure that messages can be transmitted efficiently and reliably between nodes; for example, using gRPC can reduce network latency and support high throughput.

[0098] In this embodiment, the receiving node list includes all nodes that need to receive and process the transaction message, ensuring that each relevant node can participate in consistency maintenance after receiving the broadcast message. For example, the receiving node may be a cache server or a database node.

[0099] In this embodiment, the final transaction message includes an encapsulated and checksum-encoded message containing metadata about all required operations and related data, ensuring consistent transaction processing across all receiving nodes. For example, the final transaction message may include details about data update, delete, or insert operations.

[0100] In this embodiment, the check code specifically refers to a check code generated by encrypting or hashing the message content to ensure that the message has not been tampered with during transmission. For example, the check code is generated using the SHA-256 hash algorithm.

[0101] In this embodiment, message integrity and correctness verification includes verifying the integrity of the final transaction message using a checksum to ensure the data has not been corrupted or tampered with. If verification fails, the receiving node issues a retransmission request to the broadcasting node to ensure message authenticity. For example, if the checksum does not match, the receiving node will request a rebroadcast of the message.

[0102] In this embodiment, version consistency checking includes checking whether the version of the corresponding data copy in the local cache matches the latest version based on the data identifier and version information transmitted in the transaction message. If the versions are inconsistent, synchronization or rollback is required. For example, if the data version in the local cache is v1, and the version in the transaction message is v2, the latest version of the data needs to be synchronized from the data source.

[0103] In this embodiment, the pre-update operation includes temporarily updating the data copy locally at the receiving node in preparation for submission, for example, temporarily changing the value of a field in the database to the value required by the transaction.

[0104] In this embodiment, the pre-submission record includes: recording detailed information of the pre-updated data to ensure that subsequent submission operations meet consistency requirements, for example, recording the timestamp and operation type of the update operation.

[0105] In this embodiment, the consensus protocol, including distributed protocols such as Paxos and Raft, ensures that all receiving nodes reach consensus and that the final submission of transaction messages is synchronized and consistent. For example, the Raft protocol can ensure consensus among multiple nodes and avoid data inconsistencies.

[0106] In this embodiment, transaction submission includes: under the consistency protocol, all participating nodes reach consensus and confirm execution, and the transaction enters the formal submission state.

[0107] In this embodiment, the local cache update includes: updating the successfully submitted transaction data to the local cache and clearing all pre-submitted records to ensure that the cache state is consistent with the actual data. For example, updating the user information in the cache and clearing the temporary modification records.

[0108] In this embodiment, the transaction closed loop includes returning the transaction processing result (success or failure) to the broadcast node, notifying the final status of the transaction processing. For example, if the transaction submission is successful, a "success" message is fed back; if it fails, a "failure" message is fed back.

[0109] According to the above technical solution, the final transaction message is broadcast via a pre-set protocol, achieving consistent maintenance of cached data within the distributed cache system. The broadcast channel ensures efficient transmission of transaction messages. After verifying the integrity of the message, the receiving node performs preprocessing and version consistency checks based on the operation type and version information. Through the distributed consistency protocol, the receiving node participates in transaction decision-making, ensuring data correctness and consistency. After formal submission, the local cache is updated and the pre-commit record is cleared, completing the transaction loop. This effectively improves data consistency management and ensures the reliability and stability of cached data within the distributed cache system.

[0110] As a specific example, in step 5, when the distributed consistency maintenance operation is performed on the final transaction message by the receiving node, the following is also included:

[0111] The receiving node applies transaction locks to the data items involved during the preprocessing phase to prevent data consistency conflicts caused by concurrent access.

[0112] During the consensus protocol decision-making period, the receiving node dynamically adjusts the transaction lock holding timeout to avoid consistency failures caused by lock timeouts;

[0113] If some nodes time out or network anomalies occur during the consensus protocol decision-making process, the broadcast node automatically performs compensation or rollback operations based on the preset arbitration mechanism;

[0114] After the final transaction message is formally submitted, the receiving node synchronously updates the version number, priority score, and cache partition identifier of the data item to ensure that the cache system status is consistent with the data source status;

[0115] When the final transaction message is rolled back, the receiving node restores the original data state based on the pre-commit record, releases related resources, and records the exception log;

[0116] After completing transaction message processing, the receiving node asynchronously reports the transaction processing log to the centralized logging system based on the preset strategy;

[0117] After confirming that all receiving nodes have completed the processing, the broadcast node clears the completed transaction message cache and maintains the latest status of the broadcast transaction status table.

[0118] In this embodiment, the receiving node applies a transaction lock to the data items involved in the preprocessing stage, including: after receiving the transaction message, the receiving node adds a mutual exclusive access lock to each data item involved. For example, when different users modify the same type of telemetry data such as voltage or power cached in the cache space at the operating end, the system adds a transaction lock to the telemetry data record to prevent inconsistency in data query.

[0119] In this embodiment, during the consistency protocol decision, the receiving node dynamically adjusts the transaction lock holding timeout period, including dynamically extending or shortening the lock holding time according to the real-time progress of transaction execution and network latency. For example, when temporary network congestion is detected causing a slow response, the transaction lock timeout period is automatically adjusted from the default 30 seconds to 60 seconds to avoid data conflicts caused by premature lock release.

[0120] In this embodiment, if some nodes respond with timeout or network anomalies occur during the consistency protocol decision-making process, the broadcast node automatically performs compensation or rollback operations according to the preset arbitration mechanism, including: the broadcast node determines the success or failure of the transaction based on the majority rule, and coordinates all nodes to roll back the executed operations in the event of failure. For example, if three of the five receiving nodes confirm that the submission is successful and two nodes time out, the broadcast node determines that the transaction is successful based on the majority principle, and at the same time performs compensation synchronization on the unsuccessful nodes.

[0121] In this embodiment, after the final transaction message is formally submitted, the receiving node synchronously updates the version number, priority score and cache partition identifier of the data item, including: assigning a new version identifier to each data item to mark the new state of the data, and adjusting its priority in the cache according to the access frequency, and re-dividing it into the corresponding cache partition. For example, after the user information data item updates the mobile phone number, its version number +1, the priority score is increased, and it is transferred to the high-activity partition cache.

[0122] In this embodiment, when the final transaction message is rolled back, the receiving node restores the original data state based on the pre-commit record, including: restoring the original data before the transaction execution by reading the old data copy saved in the pre-processing stage, releasing the transaction lock and cleaning up the intermediate cache data. For example, in a financial transfer system, if a rollback operation is triggered, the balances of the debit account and the receiving account are restored to the pre-transfer state, and the abnormality log is recorded for subsequent investigation.

[0123] In this embodiment, after completing transaction message processing, the receiving node asynchronously reports the transaction processing log to the centralized log system based on a preset strategy. This includes packaging the transaction submission time, operation results, exception details, etc. into log entries and sending them to the centralized log server through asynchronous tasks. For example, after an order is confirmed, the order system asynchronously pushes log data such as "order ID, processing time, and result status" to the Elasticsearch cluster.

[0124] In this embodiment, after confirming that all receiving nodes have completed processing, the broadcast node clears the completed transaction message cache and maintains the latest status of the broadcast transaction status table, including: deleting the cache record corresponding to the transaction ID, updating the transaction status to "completed" and archiving it.

[0125] According to the above technical solution, a transaction lock mechanism is introduced in transaction message processing by the receiving node to effectively prevent data consistency conflicts caused by concurrent access. During the consistency protocol decision-making period, the transaction lock timeout period is dynamically adjusted to avoid consistency failures due to timeouts. In the event of a node response timeout or network anomaly, the broadcast node can automatically perform compensation or rollback operations to ensure the reliability of the transaction. After the final transaction is committed, the receiving node synchronously updates the data. After the final transaction is committed, the receiving node synchronously updates the data status and restores the data during rollback to ensure the consistency between the distributed cache system and the data source. In addition, the asynchronous reporting of transaction processing logs and the transaction status management of broadcast nodes improve the maintainability and stability of the system, ensuring efficient and stable operation under the distributed cache system.

[0126] In summary, the hotspot data elimination and maintenance method in the distributed cache system according to the above embodiment has the following beneficial effects:

[0127] The present invention generates a dynamic priority score for data in a distributed cache system by comprehensively considering dynamic factors such as data access frequency, time locality score, business weight and real-time data heat value. It can more accurately evaluate the importance of data in the cache, dynamically adjust the priority and replacement strategy of cached data, and thus optimize the cache invalidation and replacement process. By real-time updating and integrating multi-dimensional scores, this method can effectively improve the performance of the cache system, ensure the priority retention of key business data and high-frequency data, and reasonably eliminate inefficient and cold spot data, thereby realizing more intelligent and efficient cache data elimination and consistency maintenance management in the distributed cache system, and improving the utilization rate and response speed of cache space.

[0128] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A method for eliminating and maintaining hot data in a distributed cache system, characterized in that: include: Step 1: Obtain the data's access frequency, temporal locality score, business weight, and real-time data popularity value, and then generate a dynamic priority score for the data in the distributed cache system; Step 2: Determine invalid data based on the dynamic priority score of data in the distributed cache system; Step 3: Based on the dynamic priority scores of all data in the distributed cache system, the data in the system is divided into hot zone data, warm zone data, and cold zone data. When cache space is insufficient, data in the cold zone with the lowest dynamic priority score and a retention benefit value below the preset threshold is eliminated first. If cache space is still insufficient, data in the warm zone is eliminated based on the most recent access time. Step 4: Encapsulate the cold spot area elimination data, warm spot area elimination data and invalid data into the final transaction message; Step 5: The final transaction message is broadcast through the preset protocol, and the receiving nodes then perform distributed consistency maintenance operations on the final transaction message; Among them, step 1 specifically includes: Count the number of times data in the distributed cache system is accessed within a preset time window as the access frequency; Determine the time interval between the last access time of the data in the distributed cache system and the current time, and then determine the temporal locality score of the data in the distributed cache system. The temporal locality score = 1 / (1 + time interval); Assign business weights based on pre-configured business type tags and a preset tag-weight data table; Determine the real-time data heat value of the data in the distributed cache system based on a preset method, and determine the penalty value of high-frequency but currently unaccessed data; The data access frequency, temporal locality score, business weight, and real-time data heat value are integrated to generate a dynamic priority score. The dynamic priority score = (access frequency × business weight) + (temporal locality score × real-time data heat value) - penalty value. The penalty value = γ × (current time - last access time) × access frequency. γ is a weight factor that indicates the degree of influence of non-access time on the penalty value.

2. The method for eliminating and maintaining hot data in a distributed cache system according to claim 1, characterized in that: Determine the real-time data heat value of the data in the distributed cache system based on a preset method, including: Count the number of accesses to data in the distributed cache system within a sliding window of preset length and perform weighting; The weighted cumulative number of visits is used to generate an initial heat value, which is then corrected and standardized based on the query business type to obtain a real-time data heat value.

3. The method for eliminating and maintaining hot data in a distributed cache system according to claim 1, characterized in that: Step 2 specifically includes: Set a dynamic priority score threshold T1 to filter data from the distributed cache system with a score lower than T1; For data with scores lower than T1, a multi-level survival period is assigned, and the lower the score, the shorter the survival period; Monitor the version number change event of the data source and capture the version number increment signal; When a version number increment signal change is detected, the dynamic priority score of the corresponding cache system data is forcibly reset to zero; Data whose dynamic priority score is zero or whose life cycle expires is marked as invalid, thereby obtaining invalid data.

4. The method for eliminating and maintaining hot data in a distributed cache system according to claim 1, characterized in that: Step 3 specifically includes: Set the hot spot threshold H and the cold spot threshold T2 to divide the range of hot spot data, warm spot data and cold spot data; Allocate fixed cache space for hotspot data and prohibit active elimination. When cache space is insufficient, only natural competitive updates based on the data's dynamic priority score are allowed, eliminating the hotspot data with the lowest dynamic priority score. Determine the retention benefit value of cold spot data; Filter cold spot data with retention benefit values ​​below the preset threshold and the lowest dynamic priority score to generate an elimination queue. If multiple data have the same score, they are further sorted in ascending order by data size; The cache space occupied by data in the cold spot area is released in the order of the elimination queue. Every time a piece of data is released, the data in the cold spot area is updated and the elimination log is recorded. The remaining cache space is checked in real time and the elimination is terminated if the demand is met. If the cache space is still insufficient after releasing the cold spot area data, the preset mechanism will be used to eliminate the warm spot area data according to the most recent access timestamp sorting.

5. The method for eliminating and maintaining hot data in a distributed cache system according to claim 1, characterized in that: Step 4 specifically includes: Determine the data to be packaged based on the cold spot area elimination data, warm spot area elimination data and failure data; Obtain the operation type of the data to be encapsulated, extract the corresponding identifier from the preset cache metadata based on the operation type of the data to be encapsulated, assign a unique code to the data to be encapsulated for each operation type, and write the unique code, operation type, and identifier of the data to be encapsulated into the message header; Based on the data identifier corresponding to the data to be encapsulated, the latest version number of the data source is obtained from the preset data source and the latest version number of the data source is bound to the data to be encapsulated. If the data source version number has not changed, the locally cached version number is used. At the same time, the message header of the data to be encapsulated is compressed and redundant fields are deleted. Based on the identifier corresponding to the data to be encapsulated, the corresponding score of the local cache is obtained from the preset data source, and the consistency check is performed based on the operation type of the data to be encapsulated, and conflict events are marked; Conflict handling is performed on conflict events based on the preset conflict handling logic; Determine the sending priority and processing strategy of the data to be encapsulated based on the operation type and encapsulate the data to be encapsulated to obtain a transaction message; Serialize the encapsulated transaction message according to the preset binary format, generate a binary stream and encode it in the preset format; Add a checksum to the encoded binary stream and append the checksum to the end of the transaction message to generate the final transaction message.

6. The method for eliminating and maintaining hot data in a distributed cache system according to claim 1, characterized in that: Step 5 specifically includes: Establish a transaction message broadcast channel based on the preset broadcast protocol, and configure the message push node and receiving node list; The final transaction message with the encapsulated and checksum added is pushed to each receiving node through the broadcast channel; After receiving the final transaction message, the receiving node verifies the integrity and correctness of the final transaction message. If the verification fails, a retransmission request is sent. If the verification passes, transaction preprocessing is performed. Based on the operation type, data identifier, and version information in the final transaction message, the corresponding data copy is searched in the local cache and version consistency is checked. If the version consistency check passes, the receiving node will perform a pre-update operation on the corresponding data copy according to the operation type indicated by the transaction message, and generate a pre-commit record locally; The receiving node participates in the distributed consensus decision based on the preset consensus protocol and completes the formal submission of the transaction message; After formal submission, the local cache status is updated and the pre-submission record is cleared. At the same time, the final transaction message processing result is fed back to the broadcast node to complete the transaction loop and implement distributed consistency maintenance operations.

7. The method for eliminating and maintaining hot data in a distributed cache system according to claim 1, characterized in that: In step 5, when the receiving node performs distributed consistency maintenance on the final transaction message, the following steps are also included: The receiving node applies transaction locks to the data items involved during the preprocessing phase; During the consensus protocol decision period, the receiving node dynamically adjusts the transaction lock holding timeout period; If some nodes time out or network anomalies occur during the consensus protocol decision-making process, the broadcast node automatically performs compensation or rollback operations based on the preset arbitration mechanism; After the final transaction message is formally submitted, the receiving node synchronously updates the version number, priority score, and cache partition identifier of the data item to ensure that the cache system status is consistent with the data source status; When the final transaction message is rolled back, the receiving node restores the original data state based on the pre-commit record, releases related resources, and records the exception log; After completing transaction message processing, the receiving node asynchronously reports the transaction processing log to the centralized logging system based on the preset strategy; After confirming that all receiving nodes have completed the processing, the broadcast node clears the completed transaction message cache and maintains the latest status of the broadcast transaction status table.

Images