Data caching method and device, electronic equipment and readable storage medium

Abstract

The application discloses a data caching method and device, electronic equipment and readable storage medium, comprising: obtaining a local write operation based on a cache key, determining a cache entry to be written into a current node, writing the cache entry into a local mapping table of the current node, generating and sending a first broadcast message, obtaining a second broadcast message sent by a remaining node, finding a target cache entry corresponding to the cache key in the second broadcast message from the local mapping table of the current node, comparing metadata in the cache entry in the second broadcast message with metadata in the target cache entry in the local mapping table, determining a state bit of the cache key in the second broadcast message, and performing a cache operation on the cache entry in the second broadcast message according to the state bit of the cache key in the second broadcast message. The read-write path of any cache key is completely lock-free, so that the node can realize the final consistency of the local cache without holding any global lock and without relying on external consistency services.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and specifically relates to a data caching method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the rapid development of the Internet, the number of concurrent requests and the scale of data that need to be processed are growing exponentially. Traditional centralized architectures can no longer meet the needs. Therefore, distributed system architecture has emerged and been widely used. In distributed systems, in order to improve data read performance and achieve high availability of services, multiple service replica nodes are usually deployed. Each of these nodes maintains a local cache copy to share the read and write pressure and ensure that the system as a whole can still provide services when some nodes fail. Therefore, how to ensure that the same data replicas distributed on different nodes can eventually reach a consistent state after update operations, and avoid business logic errors caused by data inconsistency, has become a key issue of distributed systems.

[0003] Currently, the industry typically adopts a strong synchronization mechanism based on distributed locks, utilizing external coordination services such as Redis distributed locks or Zookeeper to attempt to acquire a global lock when writing data. Only the node that acquires the global lock can perform cache update operations, while other nodes are blocked or wait. However, in high-concurrency scenarios, contention for the global lock can become a performance bottleneck for the entire system, leading to a decrease in data throughput and even deadlocks, making it difficult to meet the requirements of eventual consistency of cached data among multiple replica nodes in high-performance, high-concurrency systems. Summary of the Invention

[0004] The purpose of this application is to provide a data caching method, apparatus, electronic device, and readable storage medium that can solve the problem that the current strong synchronization mechanism based on distributed locks is difficult to meet the requirement of eventual consistency of cached data among multiple replica nodes in a high-performance, high-concurrency system.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a data caching method, applied to any node in a distributed system, wherein the node is configured with a local mapping table, the method comprising: Obtain local write operations based on cache keys and determine the cache entry to be written to the current node; wherein, the cache entry includes cache data and metadata describing the cache data; Write the cached entry into the local mapping table of the current node; Generate and send a first broadcast message; the first broadcast message carries the cache key and the metadata in the cache entry written by the current node; Obtain the second broadcast message sent by the other nodes; the second broadcast message carries the cache key and the metadata in the cache entries written by the other nodes; Search the local mapping table of the current node for the target cache entry corresponding to the cache key in the second broadcast message; The metadata in the cached entry of the second broadcast message is compared with the metadata in the target cached entry in the local mapping table to determine the status bit of the cache key in the second broadcast message; Based on the status bit of the cache key in the second broadcast message, perform a cache operation on the cache entry in the second broadcast message.

[0006] Optionally, obtaining the local write operation based on the cache key and determining the cache entry to be written to the current node includes: Retrieve local write operations based on cache keys; The write data thread searches for the cache entry in the local mapping table of the distributed system based on the cache key. Based on the status bit of the cache key, determine the cache entry to be written to the current node.

[0007] Optionally, determining the cache entry to be written to the current node based on the status bit of the cache key includes: If the status bit of the cache key is valid, then the cache entry that matches the cache key is determined to be the cache entry to be written to the current node; If the status bit of the cache key is invalid or being updated, then the downgraded value of the cache entry matching the cache key is determined to be the cache entry to be written to the current node.

[0008] If the status bit of the cache key is conflicted, the write data thread executes a preset conflict resolution strategy to determine the cache entry after conflict resolution as the cache entry to be written to the current node.

[0009] Optionally, generating and sending the first broadcast message includes: According to the preset broadcast message body, the cache key and the metadata in the cache entry written by the current node are combined to generate the first broadcast message of the current node; Determine the message queue partition that matches the cache key, and send the first broadcast message of the current node to the message queue partition.

[0010] Optionally, obtaining the second broadcast message sent by the remaining nodes includes: The consumer thread retrieves the second broadcast message sent by the other nodes from the message queue partition; Determine the cache key carried in the second broadcast message, as well as the metadata in the cache entries written by the remaining nodes.

[0011] Optionally, the metadata includes the local version number and write timestamp of the cached data. The step of comparing the metadata in the cached entry of the second broadcast message with the metadata in the target cached entry of the local mapping table to determine the status bit of the cache key in the second broadcast message includes: The local version number and write timestamp of the metadata in the second broadcast message are compared with the local version number and write timestamp of the metadata in the local mapping table to determine the status bit of the cache key in the second broadcast message; If the local version number in the second broadcast message is lower than the local version number in the local mapping table, then the status bit of the cache key in the second broadcast message is determined to be invalid. If the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is earlier than the write timestamp in the local mapping table, then the status bit of the cache key in the second broadcast message is determined to be conflicted. If the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is later than the write timestamp in the local mapping table, then the status bit of the cache key in the second broadcast message is determined to be valid.

[0012] Optionally, performing a caching operation on the cache entry in the second broadcast message based on the status bit of the cache key in the second broadcast message includes: If the status bit of the cache key in the second broadcast message is invalid, then the second broadcast message is discarded; If the status bit of the cache key in the second broadcast message is conflicted, the consumer thread executes the preset conflict resolution strategy to update the cache entry that matches the cache key in the second broadcast message after the conflict is resolved to the local mapping table of the current node. If the status bit of the cache key in the second broadcast message is valid, then the cache entry that matches the cache key in the second broadcast message will be updated to the local mapping table of the current node.

[0013] Secondly, embodiments of this application provide a data caching device applied to any node in a distributed system, wherein the node is configured with a local mapping table, and the device includes: The first determining module is used to obtain local write operations based on cache keys and determine the cache entry to be written to the current node; wherein, the cache entry includes cache data and metadata describing the cache data; The write data module is used to write the cached entries into the local mapping table of the current node; The message sending module is used to generate and send a first broadcast message; the first broadcast message carries a cache key and metadata from the cache entry written by the current node; The message acquisition module is used to acquire the second broadcast message sent by the other nodes; the second broadcast message carries the cache key and the metadata in the cache entries written by the other nodes; The lookup module is used to search the local mapping table of the current node for the target cache entry corresponding to the cache key in the second broadcast message; The second determining module is used to compare the metadata in the cached entry of the second broadcast message with the metadata in the target cached entry in the local mapping table to determine the status bit of the cache key in the second broadcast message; The processing module is used to perform caching operations on the cache entries in the second broadcast message according to the status bit of the cache key in the second broadcast message.

[0014] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the data caching method as described in the first aspect.

[0015] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the data caching method as described in the first aspect.

[0016] This application provides a data caching method applied to any node in a distributed system. The node is configured with a local mapping table. A local write operation based on a cache key is obtained, and a cache entry to be written to the current node is determined. The cache entry includes cached data and metadata describing the cached data. The cache entry is written to the local mapping table of the current node, and a first broadcast message is generated and sent. The first broadcast message carries the cache key and the metadata from the cache entry written by the current node. A second broadcast message sent by other nodes is obtained. The second broadcast message carries the cache key and the metadata from the cache entries written by other nodes. A target cache entry corresponding to the cache key in the second broadcast message is searched in the local mapping table of the current node. The metadata from the cache entry in the second broadcast message is compared with the metadata from the target cache entry in the local mapping table to determine the status bit of the cache key in the second broadcast message. Based on the status bit of the cache key in the second broadcast message, a caching operation is performed on the cache entry in the second broadcast message. In this application embodiment, the read and write paths for any cache key are completely lock-free, and can be linearly scaled in high-concurrency complex scenarios. It avoids global lock contention. After a cache entry is written, the node generates a fixed-length broadcast message that is independent of the size of the cached data. A decision mechanism based on the cache key status bit is introduced to determine the status of the data to be cached in the broadcast messages of other nodes, thereby making a data caching operation that accurately matches the cache key status. This avoids data corruption or loss caused by forced consistency when the data is unstable. It enables nodes to achieve eventual consistency of local caches without holding any global locks or relying on external consistency services, thereby meeting the requirements of eventual consistency of cached data among multiple nodes in a high-performance, high-concurrency distributed system.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating the steps of a data caching method provided in an embodiment of this application; Figure 2 yes Figure 1 A schematic diagram of the architecture of a data caching method provided in an embodiment of this application; Figure 3 yes Figure 1A schematic diagram of the state mechanism in a data caching method provided in this application embodiment; Figure 4 yes Figure 1 One of the flowcharts of a data caching method provided in this application embodiment; Figure 5 yes Figure 1 A second flowchart illustrating a data caching method provided in this application embodiment; Figure 6 This is a schematic diagram of the structure of a data caching device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] The data caching method, apparatus, electronic device, and storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0022] Reference Figure 1 This diagram illustrates a flowchart of the data caching method provided in an embodiment of this application. It is applied to any node in a distributed system, where a local mapping table is configured. The method may include: Step 101: Obtain the local write operation based on the cache key and determine the cache entry to be written to the current node; wherein, the cache entry includes cache data and metadata describing the cache data.

[0023] In this embodiment, it is applied to any node in a distributed system, and the node is configured with a local mapping table, as shown below. Figure 2This diagram illustrates the architecture of a data caching method provided in this application, showcasing a distributed system based on a broadcast mechanism. The system comprises multiple nodes and a central message middleware. Taking nodes A, B, and C as an example, each node represents an independent server instance in the distributed system. Each node performs the same duties: handling read / write requests, maintaining local caches, and participating in broadcast synchronization. The central message middleware can be Kafka / Redis Pub / Sub. Both Kafka Pub / Sub and Redis Pub / Sub are implementations based on the publish / subscribe message communication model. Kafka is a distributed stream processing platform designed for high-throughput, persistent, massive data stream processing, while Redis is a high-performance in-memory database used for caching and simple message queues; specific limitations are not specified here. Each node internally consists of three core modules: a local cache, a state machine, and a broadcast transceiver module. The state machine drives local cache behavior, and the broadcast mechanism achieves distributed cache data synchronization, resulting in eventually consistent distributed caching.

[0024] Each node maintains a local cache, which is the actual storage location of the cached data. A cache entry is a cell or entry in the distributed cache, representing an independent cache record. The cache entry is the most basic storage unit in the caching system. The cache key is the unique identifier of the cache entry. The cache entry is the storage unit corresponding to the cache key. The cache entry includes cached data and metadata that describes the cached data. That is, the cache entry includes the cached data value itself, additional information describing the cached data, namely metadata, and the cache key used to locate the cache entry. The cached data can be any business data, and the metadata is used to record information about the cached data. In this embodiment, the cache entry and metadata are stored at the same address to ensure that all information can be obtained with a single pointer location during a read operation.

[0025] In this embodiment, the current node obtains a local write operation based on the cache key to determine the cache entry to be written to the current node. Specifically, when the current node receives a local write operation (e.g., add, modify) for a specific cache key, it determines the cache entry to be written to the current node based on the operation and the existing state of the cache key in the local cache. A state machine mechanism is introduced, where each cache key independently holds a state. All state changes are performed directly by the user thread within its own execution context, i.e., from a single-threaded perspective. No mutex locks or CAS (compare and swap) operations are used, and thread blocking is avoided. The state machine maintains and switches states, tracking the current state of each cache key. State bits include invalid, valid, updating, and conflicting, driving state changes based on read / write requests or broadcast messages, and determining the next operation based on the state. Each node is equipped with a broadcast transceiver module responsible for interacting with the central message middleware. After a node completes a local write operation, the broadcast transceiver module constructs and sends a broadcast message, continuously listening to the message queue and retrieving broadcast messages from other nodes.

[0026] Step 102: Write the cached entries into the local mapping table of the current node.

[0027] In this embodiment, after determining the cache entry to be written to the current node, the cache entry is written to the local mapping table of the current node. The cache entry includes cached data and metadata describing the cached data. The metadata is generated additionally based on the cached data and is used to record information about the cached data. The metadata includes the local version number and write timestamp of the cached data.

[0028] It's important to note that when any node performs a write operation, it atomically increments the local Redis instance to generate a globally unique version number `ver`. If Redis is not deployed, this degenerates into a Snowflake ID. For example, if the Redis service is not deployed, there is a network failure, or a connection timeout, it automatically switches to the backup solution, namely the Snowflake ID. Snowflake ID is an open-source distributed unique ID generation algorithm that can generate globally unique 64-bit integer IDs in a distributed system without relying on any centralized service (such as a database or Redis). Details will not be elaborated here. If Redis is deployed locally, the local version number can be atomically incremented to generate a globally unique, incremental version number. The Redis INCR command is an atomic operation command, short for INCREMENT. When multiple nodes simultaneously execute the INCR command on a cached key, the INCR command increments the value of the specified key by 1 and returns the result. Redis ensures that these operations are executed sequentially, without any concurrency conflicts.

[0029] For example, since metadata and cached data are stored at the same address, the read thread can parse the complete metadata information with a single memory access the moment it obtains the entry pointer, without needing additional pointer jumps. Metadata includes the local version number and write timestamp, and may specifically include the following fields, as shown in Table 1; Table 1: Meaning of Metadata Fields

[0030] Step 103: Generate and send a first broadcast message; the first broadcast message carries the cache key and the metadata in the cache entry written by the current node.

[0031] In this embodiment, each node is configured with a broadcast transceiver module. When the current node completes the local cache update, it will trigger a broadcast process to package the changed cache information into a highly compact message body that does not contain the cache data value itself, but contains the metadata in the cache entry. Specifically, the first broadcast message carries the cache key and the metadata in the cache entry written by the current node. It can be the first broadcast message generated by combining the cache key and the metadata in the cache entry written by the current node according to a preset broadcast message body. After determining the message queue partition that matches the cache key, the first broadcast message is sent to the message queue partition. The first broadcast message is a broadcast message sent by the current node to other nodes after writing or updating cache data.

[0032] Step 104: Obtain the second broadcast message sent by the other nodes; the second broadcast message carries the cache key and the metadata in the cache entries written by the other nodes.

[0033] In this embodiment, each node runs one or more consumer threads. The current node can retrieve second broadcast messages sent by other nodes from the message queue partition through the consumer thread. The second broadcast message is a broadcast message sent by other nodes in the distributed system after updating the cache. The second broadcast message carries the cache key and metadata from the cache entries written by other nodes. Based on the second broadcast message, the current node can determine the cache key carried in the second broadcast message and the metadata from the cache entries written by other nodes in order to execute the cache data update strategy. After obtaining the second broadcast message through the consumer thread, since the message body of the second broadcast message is a 32-byte binary array, it is necessary to parse out the carried cache key and the metadata from the cache entries written by other nodes for subsequent state judgment and decision-making.

[0034] Step 105: Search the local mapping table of the current node for the target cache entry that corresponds to the cache key in the second broadcast message.

[0035] Step 106: Compare the metadata in the cached entry in the second broadcast message with the metadata in the target cached entry in the local mapping table to determine the status bit of the cache key in the second broadcast message.

[0036] In this embodiment, the metadata in the cached entry of the second broadcast message is compared with the metadata in the target cached entry in the local mapping table to determine the status bit of the cache key in the second broadcast message. Specifically, the metadata includes the local version number and write timestamp of the cached data. Therefore, the local version number and write timestamp in the second broadcast message are compared with the local version number and write timestamp in the local mapping table, respectively, to determine the status bit of the cache key in the second broadcast message. The status bit includes one of valid, invalid, conflict, and updated. In a specific implementation, refer to... Figure 3 This diagram illustrates a state mechanism in a data caching method provided in this application embodiment. The state mechanism includes valid, invalid, conflict, and switching during updates. Each cache key independently holds a state, and the state of each cache key is independent. All changes to this state are performed directly by the user thread in its own execution context, i.e., from a single-threaded perspective. Mutex locks are not used, CAS (compare and swap) operations are not performed, and thread blocking is not caused.

[0037] In the specific implementation, the local version number in the second broadcast message is compared with the local version number in the local mapping table, and the write timestamp in the second broadcast message is compared with the write timestamp in the local mapping table. If the local version number in the second broadcast message is lower than the local version number in the local mapping table, the status bit of the cache key in the second broadcast message is determined to be invalid. If the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is earlier than the write timestamp in the local mapping table, the status bit of the cache key in the second broadcast message is determined to be conflicted. If the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is later than the write timestamp in the local mapping table, the status bit of the cache key in the second broadcast message is determined to be valid.

[0038] Step 107: Perform a caching operation on the cache entry in the second broadcast message according to the status bit of the cache key in the second broadcast message.

[0039] In this embodiment, based on the status bit of the cache key in the second broadcast message, a caching operation is performed on the cache entry corresponding to the cache key in the second broadcast message. Different processing is performed on the second broadcast message according to different status bits of the cache key to ensure accurate and efficient updating of the local cache when cache data is updated, achieving eventual consistency of cache data. Specifically, if the status bit of the cache key in the second broadcast message is invalid, the second broadcast message is discarded. If the status bit of the cache key in the second broadcast message is conflicting, a preset conflict resolution strategy is executed in the consumer thread, updating the cache entry corresponding to the cache key in the second broadcast message to the local mapping table of the current node after conflict resolution. If the status bit of the cache key in the second broadcast message is valid, the cache entry corresponding to the cache key in the second broadcast message is updated to the local mapping table of the current node.

[0040] This application provides a data caching method applicable to any node in a distributed system. The read / write path for any cache key is completely lock-free, allowing for linear scaling in high-concurrency complex scenarios. It avoids global lock contention. After a cache entry is written, the node generates a fixed-length broadcast message independent of the cached data size. A decision-making mechanism based on cache key status bits is introduced to determine the status of data requiring caching in the broadcast messages of other nodes, thereby enabling accurate data caching operations that match the cache key status. This avoids data corruption or loss caused by forced consistency when data is unstable, allowing nodes to achieve eventual consistency of their local cache without holding any global locks or relying on external consistency services. This satisfies the requirement for eventual consistency of cached data among multiple nodes in a high-performance, high-concurrency distributed system.

[0041] In some embodiments of this application, step 101 may include the following steps: Sub-step 1011: Obtain the local write operation based on the cache key; Sub-step 1012 involves the write data thread searching for cache entries in the local mapping table of the distributed system based on the cache key. Sub-step 1013: Determine the cache entry to be written to the current node based on the status bit of the cache key.

[0042] In this embodiment of the application, in a distributed system, writing data needs to consider concurrency conflicts and local state to ensure the safety and efficiency of local write operations. Therefore, when a client or other system initiates a local write operation for a cache key, the current node obtains the local write operation based on the cache key. The thread executing this write operation, i.e., the write data thread, will query the local mapping table to determine whether the cache key already exists in the local cache of the current node. That is, the write data thread searches for the cache entry in the local mapping table of the distributed system based on the cache key. The local mapping table is usually a Concurrent Hash Map. Concurrent Hash Map is a thread-safe hash table implementation provided by the Java concurrency package (java.util.concurrent). It is a concurrent version of Hash Map, used to provide high-throughput read and write operations in a multi-threaded environment while ensuring thread safety.

[0043] In this embodiment, the process of the write thread searching for cache entries in the local mapping table of the distributed system based on the cache key is lock-free. For example, using `Concurrent Hash Map.get()` can quickly obtain the current state without locking. `Concurrent Hash Map.get()` is a method provided by the `Concurrent Hash Map` class in Java, used to retrieve the corresponding value based on the key. If `entry != null`, it is a conditional expression indicating that the corresponding cache entry has been successfully found based on the cache key. Based on the state bit of the cache key, the cache entry to be written to the current node is determined. Since the metadata and cached data values ​​are stored at the same address, the read thread can parse the complete metadata information with a single memory access the moment it obtains the `entry` pointer, without requiring additional pointer jumps.

[0044] In the specific implementation, the cache entry to be written to the current node is determined based on the status bits of the cache key. The write thread then decides how to perform the write operation based on the status bits of the cache key. Metadata includes information such as status bits, local version number, and write timestamp. Status bits include valid, invalid, conflicting, and updating, reflecting the current state of the cache key to determine the specific content to be written. For example, if the status bit is valid, it means that the data of the cache entry corresponding to the current cache key is normal, and the write operation can directly write the new value. If the status bit is invalid or updating, it means that the data of the cache entry corresponding to the current cache key has expired or is being refreshed. In this case, the write operation can degrade the current cache entry. If the status bit is conflicting, it means that the data of the cache entry corresponding to the current cache key is in a conflicting state. The write operation is usually performed after the conflict is resolved to ensure cache data consistency.

[0045] For example, refer to Figure 4 This illustration shows one of the flowcharts of a data caching method provided in the application embodiment. Specifically, the application thread, which is the thread that handles user requests, is the write operation thread in this embodiment. It needs to update the cached data value corresponding to a certain cache key, and calls the local cache's `put(key, value)` method. `put(key, value)` is a method provided by the Java Map interface (including Hash Map, Concurrent Hash Map, etc.) for storing key-value pairs in a mapping table. The local cache generates a globally unique, strictly incrementing version number, updates the local state machine, sets the state of the cache key to VALID, and prepares to broadcast the metadata in the cache entry that matches the cache key. The local cache sends the constructed broadcast message to the broadcast channel Kafka / Redis Pub Sub. The broadcast channel is responsible for pushing the message to the other nodes in the distributed system. After receiving the broadcast message, the other nodes compare the metadata and update their state according to the comparison result.

[0046] The embodiments of this application use a decision-making mechanism based on status bits to make the most appropriate write decision, thereby avoiding data corruption or loss caused by forced writing when the data is unstable. Lock-free lookup and metadata reading avoid performance bottlenecks caused by lock contention and ensure high performance of write operations.

[0047] In some embodiments of this application, sub-step 1013, determining the cache entry to be written to the current node based on the status bit of the cache key, may specifically include: If the status bit of the cache key is valid, then the cache entry that matches the cache key is determined to be the cache entry to be written to the current node. If the status bit of the cache key is invalid or being updated, then the downgraded value of the cache entry that matches the cache key is determined to be the cache entry to be written to the current node.

[0048] If the status bit of the cache key is conflicted, the write data thread executes the preset conflict resolution strategy to determine the cache entry after the conflict is resolved as the cache entry to be written to the current node.

[0049] In this embodiment, determining the cache entry to be written based on the status bit of the cache key is a crucial step in the write operation. Specifically, driven by a state mechanism, differentiated processing of cached data with different status bits is implemented. The write thread uses different strategies to determine the final cache entry to be written based on the status bit parsed from the metadata.

[0050] In this embodiment, if the cache key's status bit is valid, the cache entry matching the cache key is in a normal and available state, and the local cache data is up-to-date. The write operation can directly perform subsequent operations based on this valid cache entry. Therefore, the cache entry matching the cache key is determined to be the cache entry to be written to the current node. If the cache key's status bit is invalid or updating, invalid means that the data has expired, but the old data still exists. Updating means that the data is being refreshed by other threads. In these two states, although the cache entry matching the cache key is not up-to-date, there are still old values ​​available, which can be downgraded. The write operation does not wait for the data to become new, but directly writes based on the current old value, i.e., the downgraded value. Since the old value is unreliable or about to be replaced, the new write operation can directly overwrite it without strict version checking. After the write is completed, the cache key's status will be set to valid. If the status bit of the cache key is conflicted, it means that a version contradiction occurred before (e.g., the version number is larger but the timestamp is smaller). It is impossible to automatically determine which data is correct. The write operation cannot be performed directly and the conflict must be resolved first. Therefore, the write data thread executes the preset conflict resolution strategy to determine the cache entry after the conflict is resolved as the cache entry to be written to the current node. That is, the preset conflict resolution strategy will return a new cache entry in a consistent state. The write operation will then perform subsequent operations based on the resolved cache entry.

[0051] It should be noted that the preset conflict resolution strategies may include: Last-Writer-Wins, Timestamp-Wins, Priority-Wins, and Merge-Func, etc. Last-Writer-Wins is the default strategy, based on the strict increment guarantee of the version number generator, prioritizing the higher version number (ver). It unconditionally trusts the local version number; the one with the larger version number is the final winner. When a conflict occurs, it directly compares msg.ver in the broadcast message with entry.meta.ver in the local mapping table. Timestamp-Wins is a timestamp-first strategy; events with updated timestamps are the truly latest events. Priority-Wins is a priority-first strategy; the importance of data updates depends on their priority level, not their timing. Merge-Func is a custom merge function. The caller registers a custom merge function, which handles the two conflicting data versions (old local data and new data in the broadcast) to the business-defined function, generating a completely new data version that incorporates the modifications from both sides. In this embodiment, the conflict resolution strategy is dynamically distributed in the configuration center according to the cache dimension, without requiring a restart.

[0052] This application's embodiments introduce a decision-making mechanism based on status bits. When the version is high but the timestamp is in reverse order, a conflict is forcibly temporarily stored. Once a conflict state is entered, the data reading thread resolves it immediately without blocking the background consumer thread.

[0053] In some embodiments of this application, step 103, generating and sending a first broadcast message, may specifically include the following steps: Sub-step 1031: According to the preset broadcast message body, combine the cache key and the metadata in the cache entry written by the current node to generate the first broadcast message of the current node; Sub-step 1032: Determine the message queue partition that matches the cache key, and send the first broadcast message of the current node to the message queue partition.

[0054] In this embodiment, after a node completes a local cache update, a broadcast process is triggered. The changed cache information is packaged into a highly compact message body that does not contain the cache data value itself, but rather metadata. Specifically, according to a preset broadcast message body, the cache key and the metadata from the cache entry written by the current node are combined to generate the current node's first broadcast message. A message queue partition matching the cache key is then determined, and the first broadcast message is sent to that partition. Each node is configured with a broadcast transceiver module, supporting sending broadcast messages to other nodes and also actively or passively receiving broadcast messages from other nodes regarding cache data updates.

[0055] It should be noted that the preset broadcast message body can specifically include a cache key, local version number, write timestamp, operation type, priority, verification, and reserved fields. The fields of the preset broadcast message body consist of key-hash, ver, ts-low, op, prio, checksum, and reserved. key-hash is a 128-bit (16-byte) hash value calculated from the cache key, used for partition routing and message deduplication. ver is the globally unique local version number used in this update, occupying 8 bytes. ts-low is the low-order bit of the current node's write timestamp, occupying 4 bytes, used to assist in conflict detection. op and prio refer to the operation type (e.g., update / delete) and priority, which can be encoded in one byte. checksum is the verification calculated from the preceding fields, occupying 1 byte, used to ensure message integrity. reserved is a reserved field, occupying 2 bytes, which can be expanded according to message requirements, but is not specifically limited here. For example, the fields are filled into a fixed-length buffer of 32 bytes (256 bits) according to a predefined bit order to form the message body of the broadcast message, as shown in Table 2 below: Table 2: Broadcast Message Body Structure Table

[0056] In this embodiment, the broadcast message body does not contain the actual cached data, i.e., the value. The message size is fixed at 32 bytes, which is completely independent of the length of the key and the size of the cached data value. Regardless of whether the cached data is a small string or a large object, the cost of sending and receiving broadcast messages is the same, avoiding the transmission of large amounts of data over the network, especially in scenarios where the value is very large. When a node receives the data, it only knows that the data has been updated and does not need to immediately obtain the updated business data value. The latest business data can be loaded from the database or data source as needed when reading.

[0057] In this embodiment, the cache key and the metadata in the cache entry written by the current node are combined according to the preset broadcast message body to generate the first broadcast message of the current node. The cache key and the metadata in the cache entry written by the current node are filled according to the preset broadcast message body to form the broadcast message to be sent. After the broadcast message is constructed, it is reliably sent to a message queue such as Kafka / RocketMQ. In order to ensure that the same key always falls into the same message queue partition, the message queue is ordered within a single partition. All update events of the same key will be processed in the consumer thread in the order of sending. Therefore, the message queue partition that matches the cache key is determined and the first broadcast message is sent to the message queue partition.

[0058] In practical implementation, if the number of message queue partitions equals the number of nodes, and each node consumes one message queue partition, if a certain key becomes a hotspot and is frequently updated, the message queue partition containing that key will be under heavy pressure, leading to excessive load on the corresponding node. This can be mitigated by setting the number of message queue partitions to be greater than or equal to twice the number of nodes, ensuring the number of partitions far exceeds the number of consumers. This way, the pressure on hotspot keys can be distributed across multiple nodes, with each node consuming multiple message queue partitions, while also reserving space for future node expansion. The message queue uses at-least-once semantics for sending messages. The sender waits for confirmation from the Broker; if no confirmation is received, it automatically retryes, ensuring messages are not lost over the network. The Broker (message broker / message middleware server) is the core server in the message queue, responsible for receiving, storing, routing, and forwarding messages.

[0059] This application embodiment generates a broadcast message by combining the cache key and the metadata in the cache entry according to a preset broadcast message body. The broadcast message has a fixed length and is independent of the size of the cached data value. The network bandwidth usage is stable, eliminating instantaneous traffic spikes caused by large amounts of broadcast messages, and meeting the broadcast requirements of high concurrency and complex scenarios.

[0060] In some embodiments of this application, step 104, obtaining the second broadcast message sent by the remaining nodes, may specifically include the following steps: Sub-step 1041: The consumer thread retrieves the second broadcast message sent by the other nodes from the message queue partition; Sub-step 1042: Determine the cache key carried in the second broadcast message, as well as the metadata in the cache entries written by the remaining nodes.

[0061] In this embodiment, a consumer thread retrieves the second broadcast message sent by other nodes from the message queue partition, thereby determining the cache key carried in the second broadcast message and the metadata in the cache entries written by other nodes, so as to execute the cache data update strategy. Each node runs one or more consumer threads, which continuously pull broadcast messages from a specified partition of a message queue such as Kafka / RocketMQ. The second broadcast message is a broadcast message sent by other nodes in the distributed system after updating the cache.

[0062] In this embodiment, the current node subscribes to relevant message queue partitions according to its own configuration. Since message queue partitions are divided by key, each node typically subscribes to all message queue partitions, or subscribes to a portion of message queue partitions based on load balancing, to ensure that it receives change notifications for all keys. In this embodiment, the consumer thread retrieves the second broadcast messages sent by other nodes from the message queue partitions. The consumer thread pulls messages from the Broker by repeatedly calling the `poll()` method, avoiding consumer overload that might occur in push mode. `poll()` is a core method in the message queue consumer client used to pull messages from the Broker. If there are multiple consumer instances, the message queue client automatically allocates partitions to ensure that each partition is consumed by only one consumer instance, achieving load balancing and parallel processing.

[0063] In the specific implementation, the consumer thread retrieves the second broadcast message sent by other nodes from the message queue partition. This allows it to determine the cache key carried in the second broadcast message and the metadata from the cache entries written by the other nodes, in order to execute the cache data update strategy. After receiving the second broadcast message, the consumer thread finds that the message body is a 32-byte binary array. It needs to parse out the crucial cache key and the metadata from the cache entries written by the other nodes for subsequent state judgment and decision-making. Therefore, according to the predefined broadcast protocol, the 32-byte binary array is parsed bit by bit into various fields. The message body contains the hash value of the cache key, which is used to locate the specific cache entry during subsequent cache lookups.

[0064] For example, refer to Figure 5 This diagram illustrates a second flowchart of a data caching method provided in this application. In the initial state, there are multiple nodes: node A, node B, and a configuration center. Node A and node B both cache data for the same key and have initial version numbers. Node A performs an update operation, updating the key value, generating a new version number, and sending a broadcast message to notify other nodes. If the broadcast message is lost during transmission, node B does not receive this update notification, resulting in data inconsistency among nodes in the cluster. If a read request hits node B at this time, it will return outdated data. Therefore, node B actively queries the cluster for the latest version of the key to confirm whether its version is outdated. If outdated, it requests the latest version data or resolves the conflict. Node B learns through some means that the latest version number is inconsistent with its own version number, chooses to enter a conflict state, and waits for a clear solution. Node B obtains the actual data of the latest version, writes the new data to its local cache, and changes the state from conflict back to valid.

[0065] This application embodiment obtains broadcast messages sent by other nodes from the message queue partition through the consumer thread, avoiding data corruption or loss caused by forced consistency when the data is unstable, and enabling nodes to achieve eventual consistency of local cache without holding any global locks or relying on external consistency services.

[0066] In some embodiments of this application, the metadata includes the local version number and write timestamp of the cached data. Step 106 compares the metadata in the cached entry in the second broadcast message with the metadata in the target cached entry in the local mapping table to determine the status bit of the cache key in the second broadcast message. Specifically, this may include: The local version number and write timestamp of the metadata in the second broadcast message are compared with the local version number and write timestamp of the metadata in the local mapping table to determine the status bit of the cache key in the second broadcast message; If the local version number in the second broadcast message is lower than the local version number in the local mapping table, then the status bit of the cache key in the second broadcast message is determined to be invalid. If the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is earlier than the write timestamp in the local mapping table, then the status bit of the cache key in the second broadcast message is determined to be conflicted. If the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is later than the write timestamp in the local mapping table, then the status bit of the cache key in the second broadcast message is determined to be valid.

[0067] In this embodiment, the local version number and write timestamp of the metadata in the second broadcast message are compared with the local version number and write timestamp of the metadata in the local mapping table, respectively, to determine the status bit of the cache key in the second broadcast message. This allows for the determination of the data cache update operation to be performed on the second broadcast message based on the version number and timestamp verification. Specifically, the local version number in the second broadcast message is compared with the local version number in the local mapping table, and the write timestamp in the second broadcast message is compared with the write timestamp in the local mapping table. Based on the comparison results, the status bit of the cache key in the second broadcast message is determined, thereby determining how this broadcast message should be processed.

[0068] In this embodiment, since the local version number of the cached data is globally strictly incremental, the smaller the local version number, the older the cached data is. If the version number in the received broadcast message is lower than the existing local version number of the local cached data, it indicates that the broadcast message is outdated or invalid. This may be due to network latency causing the old message to arrive late, or due to message queue retries causing duplicate messages.

[0069] In the specific implementation, the local version number in the second broadcast message is compared with the local version number in the local mapping table, and the write timestamp in the second broadcast message is compared with the write timestamp in the local mapping table. If the local version number in the second broadcast message is lower than the local version number in the local mapping table, the status bit of the cache key in the second broadcast message is determined to be invalid. If the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is earlier than the write timestamp in the local mapping table, the status bit of the cache key in the second broadcast message is determined to be conflicted. If the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is later than the write timestamp in the local mapping table, the status bit of the cache key in the second broadcast message is determined to be valid.

[0070] This application introduces a decision-making mechanism for cached data status bits, using globally incrementing version numbers and timestamps to determine valid, invalid, and conflicting statuses, providing a reliable basis for achieving eventual consistency of the local cache.

[0071] In some embodiments of this application, step 107 performs a caching operation on the cache entry in the second broadcast message based on the status bit of the cache key in the second broadcast message, which may specifically include: If the status bit of the cache key in the second broadcast message is invalid, then the second broadcast message will be discarded. If the status bit of the cache key in the second broadcast message is conflicted, the consumer thread executes the preset conflict resolution strategy and updates the cache entry that matches the cache key in the second broadcast message after the conflict is resolved to the local mapping table of the current node. If the status bit of the cache key in the second broadcast message is valid, then the cache entry that matches the cache key in the second broadcast message will be updated to the local mapping table of the current node.

[0072] In this embodiment, based on the status bit of the cache key in the second broadcast message, a caching operation is performed on the cache entry corresponding to the cache key in the second broadcast message. Different processing is determined for the broadcast message according to different statuses to ensure accurate and efficient updating of the local cache when the cache data is updated, thereby achieving eventual consistency of the cache data. Specifically, if the status bit of the cache key in the second broadcast message is invalid, the second broadcast message is discarded. If the status bit of the cache key in the second broadcast message is conflicted, a preset conflict resolution strategy is executed in the consumer thread to update the cache entry that matches the cache key in the second broadcast message to the local mapping table of the current node after conflict resolution. If the status bit of the cache key in the second broadcast message is valid, the cache entry that matches the cache key in the second broadcast message is updated to the local mapping table of the current node.

[0073] In this embodiment, if the cache key's status bit in the second broadcast message is invalid, it means the broadcast message is outdated, and there is already updated data locally with a higher version number. The information carried by this broadcast message is no longer valuable and is discarded without any local cache modification. If the cache key's status bit in the second broadcast message is conflicting, it means a contradiction has occurred; the version is higher but the timestamp is lower, and the data is inconsistent. This must be resolved immediately. Since it's impossible to automatically determine which data is correct, the conflict resolution strategy is executed synchronously in the current consumer thread. The result of the conflict resolution is used to update the local cache. Conflict resolution is performed synchronously in the consumer thread, and the processing of this broadcast message will be blocked until the conflict resolution is complete. If the resolution is asynchronous, other operations accessing this cache key during this period may read inconsistent data. If the cache key's status bit in the second broadcast message is valid, this broadcast message does indeed contain updated data with a higher version and a later timestamp. In this case, the cache entry corresponding to the cache key in the second broadcast message is updated to the local mapping table. Since the broadcast message only contains metadata and no actual cached data value, background loading needs to be triggered, or it needs to be loaded on demand during the next read.

[0074] This application's embodiments introduce a decision-making mechanism for cached data status bits to determine valid, invalid, conflicting, and other statuses, thereby making cache operations that match the data status. This enables nodes to achieve eventual consistency of the local cache without holding any global locks or relying on external consistency services.

[0075] Reference Figure 6 This diagram illustrates a data caching device according to an embodiment of this application, applied to any node in a distributed system. The node is configured with a local mapping table. The device includes: The first determining module 201 is used to obtain local write operations based on cache keys and determine the cache entry to be written to the current node; wherein, the cache entry includes cache data and metadata describing the cache data; The write data module 202 is used to write the cached entries into the local mapping table of the current node; The message sending module 203 is used to generate and send a first broadcast message; the first broadcast message carries a cache key and metadata from the cache entry written by the current node; The message acquisition module 204 is used to acquire the second broadcast message sent by the other nodes; the second broadcast message carries the cache key and the metadata in the cache entries written by the other nodes; The lookup module 205 is used to look up the target cache entry corresponding to the cache key in the second broadcast message from the local mapping table of the current node; The second determining module 206 is used to compare the metadata in the cached entry in the second broadcast message with the metadata in the target cached entry in the local mapping table to determine the status bit of the cache key in the second broadcast message; The processing module 207 is used to perform a caching operation on the cache entry in the second broadcast message according to the status bit of the cache key in the second broadcast message.

[0076] Optionally, the first determining module 201 includes: The first submodule is used to retrieve local write operations based on cache keys; The lookup submodule is used to search for cache entries in the local mapping table of the distributed system based on the cache key by the write data thread. The first determining submodule is used to determine the cache entry to be written to the current node based on the status bit of the cache key.

[0077] Optionally, the first determining submodule includes: The first determining unit is configured to determine, if the status bit of the cache key is valid, that the cache entry matching the cache key is the cache entry to be written to the current node. The second determining unit is used to determine, if the status bit of the cache key is invalid or being updated, the downgraded value of the cache entry matching the cache key is the cache entry to be written to the current node.

[0078] The third determining unit is used to execute a preset conflict resolution strategy in the write data thread if the status bit of the cache key is conflicted, and determine the cache entry after conflict resolution as the cache entry to be written to the current node.

[0079] Optionally, the message sending module 203 includes: The generation submodule is used to combine the cache key and the metadata in the cache entry written by the current node according to the preset broadcast message body to generate the first broadcast message of the current node; The sending submodule is used to determine the message queue partition that matches the cache key and send the first broadcast message of the current node to the message queue partition.

[0080] Optionally, the message acquisition module 204 includes: The second acquisition submodule is used to retrieve the second broadcast message sent by the other nodes from the message queue partition through the consumer thread; The second determination submodule is used to determine the cache key carried in the second broadcast message, as well as the metadata in the cache entries written by the remaining nodes.

[0081] Optionally, the metadata includes the local version number and write timestamp of the cached data, and the second determining module 206 includes: The comparison submodule is used to compare the local version number and write timestamp of the metadata in the second broadcast message with the local version number and write timestamp of the metadata in the local mapping table, respectively, to determine the status bit of the cache key in the second broadcast message; The third determining submodule is used to determine that the status bit of the cache key in the second broadcast message is invalid if the local version number in the second broadcast message is lower than the local version number in the local mapping table. The fourth determining submodule is used to determine that the status bit of the cache key in the second broadcast message is conflicted if the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is earlier than the write timestamp in the local mapping table. The fifth determining submodule is used to determine that the status bit of the cache key in the second broadcast message is valid if the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is later than the write timestamp in the local mapping table.

[0082] Optionally, the processing module 207 includes: The first processing submodule is used to discard the second broadcast message if the status bit of the cache key in the second broadcast message is invalid. The second processing submodule is used to execute a preset conflict resolution strategy in the consumer thread if the status bit of the cache key in the second broadcast message is conflicted, and update the cache entry that matches the cache key in the second broadcast message after the conflict is resolved to the local mapping table of the current node. The third processing submodule is used to update the cache entry that matches the cache key in the second broadcast message to the local mapping table of the current node if the status bit of the cache key in the second broadcast message is valid.

[0083] The data caching device provided in this application embodiment can implement the various processes of the data caching method in the above embodiments of this application. To avoid repetition, it will not be described again here.

[0084] This application provides a data caching device that can be applied to any node in a distributed system. The read and write paths for any cache key are completely lock-free. It can be linearly scaled in high-concurrency and complex scenarios, avoiding global lock contention. After a cache entry is written, the node generates a fixed-length broadcast message that is independent of the size of the cached data. A decision mechanism based on the cache key status bit is introduced to determine the status of the data to be cached in the broadcast messages of other nodes, thereby making a data caching operation that accurately matches the cache key status. This avoids data corruption or loss caused by forced consistency when the data is unstable. It enables nodes to achieve eventual consistency of local caches without holding any global locks or relying on external consistency services, thus meeting the requirements of eventual consistency of cached data among multiple nodes in a high-performance, high-concurrency distributed system.

[0085] Reference Figure 7 This application also provides an electronic device, such as... Figure 7 As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304. Processor 301, memory 303 for storing processor-executable instructions; The processor 301 is configured to execute the instructions to implement the data caching method described below: Obtain local write operations based on cache keys and determine the cache entry to be written to the current node; wherein, the cache entry includes cache data and metadata describing the cache data; Write the cached entry into the local mapping table of the current node; Generate and send a first broadcast message; the first broadcast message carries the cache key and the metadata in the cache entry written by the current node; Obtain the second broadcast message sent by the other nodes; the second broadcast message carries the cache key and the metadata in the cache entries written by the other nodes; Search the local mapping table of the current node for the target cache entry corresponding to the cache key in the second broadcast message; The metadata in the cached entry of the second broadcast message is compared with the metadata in the target cached entry in the local mapping table to determine the status bit of the cache key in the second broadcast message; Based on the status bit of the cache key in the second broadcast message, perform a cache operation on the cache entry in the second broadcast message.

[0086] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0087] The communication interface is used for communication between the aforementioned terminal and other devices.

[0088] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0089] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0090] In another embodiment provided in this application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements any of the data caching methods described in the above embodiments.

[0091] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0092] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0093] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0094] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A data caching method, applied to any node in a distributed system, wherein the node is configured with a local mapping table, characterized in that, The method includes: Obtain local write operations based on cache keys and determine the cache entry to be written to the current node; wherein, the cache entry includes cache data and metadata describing the cache data; Write the cached entry into the local mapping table of the current node; Generate and send a first broadcast message; the first broadcast message carries the cache key and the metadata in the cache entry written by the current node; Obtain the second broadcast message sent by the other nodes; the second broadcast message carries the cache key and the metadata in the cache entries written by the other nodes; Search the local mapping table of the current node for the target cache entry corresponding to the cache key in the second broadcast message; The metadata in the cached entry of the second broadcast message is compared with the metadata in the target cached entry in the local mapping table to determine the status bit of the cache key in the second broadcast message; Based on the status bit of the cache key in the second broadcast message, perform a cache operation on the cache entry in the second broadcast message.

2. The method according to claim 1, characterized in that, The step of obtaining the local write operation based on the cache key and determining the cache entry to be written to the current node includes: Retrieve local write operations based on cache keys; The write data thread searches for the cache entry in the local mapping table of the distributed system based on the cache key. Based on the status bit of the cache key, determine the cache entry to be written to the current node.

3. The method according to claim 2, characterized in that, The step of determining the cache entry to be written to the current node based on the status bit of the cache key includes: If the status bit of the cache key is valid, then the cache entry that matches the cache key is determined to be the cache entry to be written to the current node; If the status bit of the cache key is invalid or being updated, then the downgraded value of the cache entry matching the cache key is determined to be the cache entry to be written to the current node. If the status bit of the cache key is conflicted, the write data thread executes a preset conflict resolution strategy to determine the cache entry after conflict resolution as the cache entry to be written to the current node.

4. The method according to claim 1, characterized in that, The generation and transmission of the first broadcast message includes: According to the preset broadcast message body, the cache key and the metadata in the cache entry written by the current node are combined to generate the first broadcast message of the current node; Determine the message queue partition that matches the cache key, and send the first broadcast message of the current node to the message queue partition.

5. The method according to claim 1, characterized in that, The step of obtaining the second broadcast message sent by the remaining nodes includes: The consumer thread retrieves the second broadcast message sent by the other nodes from the message queue partition; Determine the cache key carried in the second broadcast message, as well as the metadata in the cache entries written by the remaining nodes.

6. The method according to claim 1, characterized in that, The metadata includes the local version number and write timestamp of the cached data. The step of comparing the metadata in the cached entry of the second broadcast message with the metadata in the target cached entry of the local mapping table to determine the status bit of the cache key in the second broadcast message includes: The local version number and write timestamp of the metadata in the second broadcast message are compared with the local version number and write timestamp of the metadata in the local mapping table to determine the status bit of the cache key in the second broadcast message; If the local version number in the second broadcast message is lower than the local version number in the local mapping table, then the status bit of the cache key in the second broadcast message is determined to be invalid. If the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is earlier than the write timestamp in the local mapping table, then the status bit of the cache key in the second broadcast message is determined to be conflicted. If the local version number in the second broadcast message is higher than the local version number in the local mapping table, and the write timestamp in the second broadcast message is later than the write timestamp in the local mapping table, then the status bit of the cache key in the second broadcast message is determined to be valid.

7. The method according to claim 6, characterized in that, The step of performing a caching operation on the cache entry in the second broadcast message based on the status bit of the cache key in the second broadcast message includes: If the status bit of the cache key in the second broadcast message is invalid, then the second broadcast message is discarded; If the status bit of the cache key in the second broadcast message is conflicted, the consumer thread executes the preset conflict resolution strategy to update the cache entry that matches the cache key in the second broadcast message after the conflict is resolved to the local mapping table of the current node. If the status bit of the cache key in the second broadcast message is valid, then the cache entry that matches the cache key in the second broadcast message will be updated to the local mapping table of the current node.

8. A data caching device, applied to any node in a distributed system, wherein the node is configured with a local mapping table, characterized in that, The device includes: The first determining module is used to obtain local write operations based on cache keys and determine the cache entry to be written to the current node; wherein, the cache entry includes cache data and metadata describing the cache data; The write data module is used to write the cached entries into the local mapping table of the current node; The message sending module is used to generate and send a first broadcast message; the first broadcast message carries a cache key and metadata from the cache entry written by the current node; The message acquisition module is used to acquire the second broadcast message sent by the other nodes; the second broadcast message carries the cache key and the metadata in the cache entries written by the other nodes; The lookup module is used to search the local mapping table of the current node for the target cache entry corresponding to the cache key in the second broadcast message; The second determining module is used to compare the metadata in the cached entry in the second broadcast message with the metadata in the target cached entry in the local mapping table to determine the status bit of the cache key in the second broadcast message; The processing module is used to perform caching operations on the cache entries in the second broadcast message according to the status bit of the cache key in the second broadcast message.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the instructions to implement the data caching method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, A computer program is stored on the readable storage medium, which, when executed by a processor, implements the data caching method as described in any one of claims 1 to 7.

Images