Method and apparatus for full synchronization of cached data

Abstract

This disclosure provides a method for full synchronization of cached data, relating to the field of cloud computing technology and applicable to the financial technology field. The method is applied to a cached database master node, which uses a dual-hash architecture to store data. The dual-hash architecture includes a master hash table and a cache hash table, with the cache hash table used to store temporary incremental data. The method includes: responding to a full synchronization command from the master and slave nodes, starting a full synchronization thread and setting a full synchronization flag to perform data synchronization, wherein the full synchronization flag is used to characterize the current full synchronization state of the master node; responding to an operation command from the client, determining the target slot and the client operation type; and performing data read and write operations based on the flag of the target slot, the full synchronization flag, and the client operation type, wherein the flag of the slot is used to characterize the current operation type of the slot. This disclosure also provides a cached data full synchronization device, apparatus, storage medium, and program product.

Description

Technical Field

[0001] This disclosure relates to the field of cloud computing technology, specifically to the field of data synchronization technology, and more specifically to a method, apparatus, device, storage medium, and program product for full synchronization of cached data. Background Technology

[0002] REDIS (Remote Dictionary Server) is a high-performance key-value in-memory database characterized by high performance, rich data types, and good scalability, making it widely used in the internet field. To ensure high availability and avoid single points of failure, two nodes are typically started in master-slave mode, maintaining data synchronization. Upon initial startup or during a prolonged network outage, the master and slave nodes perform a full synchronization. The master node needs to fork a child process to generate the RDB (Redis Database) file and then push all the data to the slave node.

[0003] However, because Redis briefly blocks during the fork call, the latency can range from a few milliseconds to hundreds of milliseconds depending on the storage capacity, which has a significant impact on latency-sensitive applications. Furthermore, during the fork process of generating the RDB file, if client write operations are frequent, the COW mechanism will copy the file twice, consuming a large amount of memory and resulting in low memory utilization and significant resource waste.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this disclosure provides a method, apparatus, device, storage medium and program product for full synchronization of cached data.

[0006] According to a first aspect of this disclosure, a method for full synchronization of cached data is provided, applied to a cache database master node, wherein the master node uses a dual-hash architecture to store data, the dual-hash architecture including a master hash table and a cache hash table, the cache hash table being used to store temporary incremental data, the method comprising:

[0007] In response to the master-slave node full synchronization command, a full synchronization thread is started and a full synchronization flag is set to perform data synchronization, wherein the full synchronization flag is used to characterize the current full synchronization status of the master node;

[0008] In response to the client's operation commands, determine the target slot and the client operation type; and

[0009] Data read and write operations are performed based on the target slot's flag bit, the full synchronization flag bit, and the client operation type, wherein the slot's flag bit is used to characterize the slot's current operation type.

[0010] According to embodiments of this disclosure, the data read / write operation based on the flag bit of the target slot, the full synchronization flag bit, and the client operation type includes:

[0011] When the client operation type is determined to be a read operation, the result is returned based on the data of the target slot in the cache hash table;

[0012] When it is determined that the client operation type is a write operation and the full synchronization flag is in progress, a data write operation is performed according to the flag of the target slot.

[0013] According to embodiments of this disclosure, the data write operation based on the flag bit of the target slot includes:

[0014] If it is determined from the flag bit of the target slot that the target slot is performing a full synchronization operation, update the flag bit of the target slot;

[0015] When it is confirmed that updating the flag bit of the target slot was successful, the client data is written into the target slot of the main hash table;

[0016] If updating the flag of the target slot fails, a blocking wait time is determined; and

[0017] When the blocking wait time exceeds a preset threshold, the client data is written into the target slot in the cache hash table.

[0018] According to embodiments of this disclosure, the return result based on the data of the target slot in the cache hash table includes:

[0019] If it is determined that the data for the target slot in the cache hash table is empty, then the data for the target slot in the main hash table is read; and

[0020] If it is determined that the data of the target slot in the cache hash table is not empty, then the data of the target slot in the cache hash table is read.

[0021] According to embodiments of this disclosure, after performing data read / write operations based on the target slot's flag bit, the full synchronization flag bit, and the client operation type, the method further includes:

[0022] Merge the data in the cache hash table into the main hash table.

[0023] According to embodiments of this disclosure, merging data from the cached hash table into the main hash table includes:

[0024] When it is determined that data exists in any slot of the cache hash table, the flag bit of that slot is changed; and

[0025] Move the data from the cache hash table to the main hash table.

[0026] According to embodiments of this disclosure, starting the full synchronization thread and setting the full synchronization flag for data synchronization includes:

[0027] Update the full synchronization flag and the flag of the slot to be synchronized;

[0028] Perform a data packing operation on the slot to be synchronized; and

[0029] After execution is complete, restore the flag bit of the slot to be synchronized.

[0030] A second aspect of this disclosure provides a cached data full synchronization device applied to a cached database master node. The master node uses a dual-hash architecture to store data, the dual-hash architecture including a master hash table and a cache hash table, the cache hash table being used to store temporary incremental data. The device includes:

[0031] The data synchronization module is used to respond to the full synchronization command from the master and slave nodes, start a full synchronization thread, and set a full synchronization flag to perform data synchronization. The full synchronization flag is used to indicate the current full synchronization status of the master node.

[0032] The determination module is used to determine the target slot and the client operation type in response to the client's operation instructions; and

[0033] The data operation module is used to perform data read and write operations based on the flag bit of the target slot, the full synchronization flag bit, and the client operation type, wherein the flag bit of the slot is used to characterize the current operation type of the slot.

[0034] According to embodiments of this disclosure, the data operation module includes: a first determination submodule and a second determination submodule.

[0035] The first determination submodule is used to return a result based on the data of the target slot in the cache hash table when the client operation type is determined to be a read operation.

[0036] The second determining submodule is used to perform a data write operation based on the flag bit of the target slot when it is determined that the client operation type is a write operation and the full synchronization flag bit is in the process of full synchronization.

[0037] According to embodiments of this disclosure, the second determining submodule includes: an update unit, a first determining unit, a second determining unit, and a write unit.

[0038] An update unit is configured to update the flag bit of the target slot if it is determined from the flag bit of the target slot that the target slot is performing a full synchronization operation;

[0039] The first determining unit is used to write client data into the target slot of the main hash table when it is determined that the flag bit of the target slot has been successfully updated;

[0040] The second determining unit is configured to determine a blocking wait time when determining that updating the flag bit of the target slot fails; and

[0041] The write unit is used to write client data into the target slot in the cache hash table when the blocking wait time is greater than a preset threshold.

[0042] According to embodiments of this disclosure, the first determining submodule includes a third determining unit and a fourth determining unit.

[0043] The third determining unit is configured to read the data of the target slot in the main hash table if it is determined that the data of the target slot in the cache hash table is empty; and

[0044] The fourth determining unit is used to read the data of the target slot in the cache hash table if it is determined that the data of the target slot in the cache hash table is not empty.

[0045] According to embodiments of this disclosure, it further includes a data merging module.

[0046] The data merging module is used to merge data from the cached hash table into the main hash table.

[0047] According to embodiments of this disclosure, the data merging module includes: a first update submodule and a data merging submodule.

[0048] The first update submodule is used to change the flag bit of any slot when it is determined that data exists in any slot of the cache hash table; and

[0049] The data merging submodule is used to move data from the cached hash table to the main hash table.

[0050] According to embodiments of this disclosure, the data synchronization module includes: a second update submodule, a data packaging module, and a third update submodule.

[0051] The first update submodule is used to update the full synchronization flag and the flag of the slot to be synchronized;

[0052] A data packaging module is used to perform data packaging operations on the slot to be synchronized; and

[0053] The second update submodule is used to restore the flag bit of the slot to be synchronized after execution.

[0054] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above-described method for full synchronization of cached data.

[0055] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described method for full synchronization of cached data.

[0056] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for full synchronization of cached data.

[0057] The cached data full synchronization method provided by the embodiments of this disclosure utilizes multi-threading for full synchronization. When a full synchronization instruction is received from the master-slave node, a full synchronization thread is started to synchronize data. The main thread handles client operation requests. Data is stored using a dual-hash architecture: the main hash table stores normal data, and the cache hash table stores temporary incremental data when the two threads conflict. Upon receiving a client operation instruction, the target slot and client operation type are determined, and data operations are performed based on the target slot's flag, the full synchronization flag, and the client operation type. Compared to related technologies, the cached data full synchronization method provided by the embodiments of this disclosure improves and upgrades the Redis master-slave full synchronization mechanism, avoiding processing delays caused by the brief blocking of fork child processes during full synchronization, as well as the additional memory consumption caused by the COW mechanism during RDB generation, thereby improving physical memory utilization and avoiding resource waste. Attached Figure Description

[0058] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0059] Figure 1 The illustration schematically depicts an application scenario of a method, apparatus, device, storage medium, and program product for full synchronization of cached data according to embodiments of the present disclosure.

[0060] Figure 2a A schematic diagram of a dual-hash architecture provided according to an embodiment of the present disclosure is shown.

[0061] Figure 2b This schematically illustrates the architecture of a cached data full synchronization apparatus provided according to embodiments of the present disclosure;

[0062] Figure 3 A flowchart illustrating a method for full synchronization of cached data according to an embodiment of the present disclosure is shown schematically.

[0063] Figure 4 A flowchart illustrating a method for data synchronization using a full synchronization thread according to another embodiment of this disclosure is shown schematically.

[0064] Figure 5 The flowchart illustrates a method for performing data read / write operations based on a flag bit of the target slot, a full synchronization flag bit, and a client operation type, according to another embodiment of this disclosure.

[0065] Figure 6 A flowchart illustrating a data synchronization method for a client operation type of read operation according to another embodiment of this disclosure is shown schematically;

[0066] Figure 7 A flowchart illustrating a data synchronization method for a client operation type of write operation according to another embodiment of the present disclosure is shown.

[0067] Figure 8a This illustration schematically shows one of the flowcharts of a data merging method provided according to yet another embodiment of the present disclosure;

[0068] Figure 8b This schematically illustrates a second flowchart of a data merging method according to yet another embodiment of the present disclosure;

[0069] Figure 9 A schematic block diagram illustrating a cached data full synchronization device according to an embodiment of the present disclosure is shown; and

[0070] Figure 10 A block diagram schematically illustrates an electronic device suitable for implementing a method for full synchronization of cached data according to an embodiment of the present disclosure. Detailed Implementation

[0071] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0072] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0073] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0074] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0075] First, the terminology appearing in the embodiments of this disclosure will be explained:

[0076] REDIS: Remote Dictionary Server, an open-source, in-memory, persistent key-value database written in C, providing client SDKs in multiple languages.

[0077] RDB: Redis Database, a full memory snapshot of Redis at a given moment, saved to disk in binary format.

[0078] Fork is a Linux system call that creates a new child process. The child process copies the parent process's data and stack space, and inherits the parent process's code, environment variables, working directory, and resource limits.

[0079] COW: Copy-on-Write. After a process calls fork(), the parent and child processes share memory pages instead of immediately copying them into two copies. Only when a process attempts to modify the data and triggers an exception will the operating system kernel perform the actual copying operation, and then the parent and child processes will each have their own copy of the data.

[0080] CAS: Compare and Swap, an atomic operation that ensures data consistency without locking. Specifically, it compares the old value with the new value and replaces it with the new value if the old value matches the expected value. It is an optimistic locking strategy, so it will not be blocked or suspended even if the update fails.

[0081] The expression CAS(V, E, N) means that for variable V, if the current value is E, then update it to N and return true; otherwise, do nothing and return false.

[0082] Redis is a high-performance key-value in-memory database, characterized by high performance, rich data types, and good scalability, and is widely used in the internet field. To ensure high availability and avoid single points of failure, two nodes are typically started in master-slave mode to maintain data synchronization. Upon initial startup or during a prolonged network outage, the master and slave nodes perform a full synchronization. The master node needs to fork a child process to generate the RDB file and then push all the data to the slave node.

[0083] However, the fork mechanism has the following problems:

[0084] 1) Fork blocking

[0085] Redis briefly blocks when calling fork, and the time taken can range from a few milliseconds to hundreds of milliseconds, depending on the storage capacity. This can have a significant impact on latency-sensitive applications.

[0086] 2) Memory consumption caused by the COW mechanism

[0087] During the generation of the RDB file via fork, if client write operations are frequent, the Copy-on-Write (COW) mechanism will create two copies, doubling the memory usage. In the worst case, all memory space of the parent and child processes is copied, resulting in only 50% effective memory utilization. Therefore, to avoid process crashes due to insufficient physical memory during COW, in practice, about half of the physical memory is usually reserved, leading to very low memory utilization and significant resource waste.

[0088] Based on the above-mentioned technical problems, embodiments of this disclosure provide a method for full synchronization of cached data, including: responding to a version deployment instruction, performing dynamic consistency first-order verification on the version deployment status during the version deployment process; after the version deployment is completed, performing joint consistency verification based on the fingerprint identification information of the version to be installed and the deployment file footprint feature information set, wherein the fingerprint identification information is used to characterize the features of the version to be installed, and the deployment file footprint feature information set is generated based on the installation check-in information during the deployment process; and outputting a version verification consistency report.

[0089] Figure 1 The illustration schematically depicts an application scenario of a method, apparatus, device, storage medium, and program product for full synchronization of cached data according to embodiments of the present disclosure.

[0090] like Figure 1As shown, application scenario 100 according to this embodiment may include a scenario of full synchronization of cached data. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0091] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0092] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0093] Server 105 can be a cache database master node application server. During master-slave full synchronization, this server uses a multi-threaded mode instead of forking child processes and employs a dual-hash architecture. One main hash function is identical to the original open-source version, while the other is a cache hash used to temporarily store incremental data during master-slave full synchronization. Upon receiving a master-slave full synchronization command, it starts a full synchronization thread and sets a full synchronization flag to synchronize data. Simultaneously, it responds to client operation commands through the main thread, performing data operations based on the client's operation type and target slot.

[0094] It should be noted that the full cache data synchronization method provided in this embodiment can generally be executed by server 105. Correspondingly, the full cache data synchronization device provided in this embodiment can generally be located in server 105. The full cache data synchronization method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the full cache data synchronization device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0095] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0096] It should be noted that the cache data full synchronization method and apparatus determined in the embodiments of this disclosure can be used in the field of cloud computing technology, the field of financial technology, and any field other than the financial field. The application field of the cache data full synchronization method and apparatus determined in the embodiments of this disclosure is not limited.

[0097] Figure 2a A schematic diagram illustrating a dual-hash architecture provided according to an embodiment of the present disclosure is shown. Figure 2b An architectural diagram of a cached data full synchronization apparatus provided according to embodiments of the present disclosure is illustrated. Figure 2a As shown, a dual-hash architecture is used: a primary hash table, which functions identically to the original open-source version and stores cached data; and a secondary hash table, used to temporarily store incremental data during master-slave full synchronization when conflicts arise between the full synchronization thread and the main thread. Figure 2b As shown, this includes Redis process A1, created when the Redis program starts, containing the main thread A2 and the full synchronization thread A3. The main thread A2 handles client requests, parses and executes corresponding commands, and performs scheduled tasks. The full synchronization thread A3 is used for full synchronization between master and slave nodes, packaging and pushing data to slave nodes. The main hash table A4 stores all Redis data by default for fast key lookup. The cached hash table A5 is used to temporarily store data during full synchronization if the main thread and the synchronization thread happen to be processing the same SLOT, which could lead to data inconsistency issues. Atomic variable A6 is a full synchronization flag indicating whether full synchronization is in progress; atomic variable A7 is a flag for each hash slot, indicating the current slot status. There is a one-to-one correspondence between slots in the main hash table and the cached hash table.

[0098] The following will be based on Figure 1 The application scenarios described and Figure 2a , Figure 2b The described architecture, through Figures 3-7 The method for full synchronization of cached data according to embodiments of this disclosure will be described in detail.

[0099] Figure 3 The flowchart illustrating a method for full synchronization of cached data according to an embodiment of the present disclosure is shown schematically. Figure 3 As shown, the cache data full synchronization method of this embodiment includes operations S210 to S230, which can be executed by a server or other computing device. The cache data full synchronization method of this embodiment is applied to a cache database master node. The cache database can be, for example, a Redis database. The master node uses a dual-hash architecture to store data, which includes a main hash table and a cache hash table. The cache hash table is used to store temporary incremental data.

[0100] When operating S210, in response to the master-slave node full synchronization command, the full synchronization thread is started and the full synchronization flag is set to perform data synchronization.

[0101] According to embodiments of this disclosure, the full synchronization flag is used to characterize the current full synchronization state of the master node.

[0102] When operating S220, in response to the client's operation command, the target slot and the client operation type are determined.

[0103] In operation S230, data read and write operations are performed based on the flag bit of the target slot, the full synchronization flag bit, and the client operation type.

[0104] According to embodiments of this disclosure, the flag bit of the slot is used to characterize the current operation type of the slot.

[0105] In one example, when the master node receives the full synchronization command from the master-slave node, it does not need to fork a child process. Instead, it starts a full synchronization thread to synchronize data and sets the full synchronization flag. The full synchronization flag FULL_SYNC_FLAG is used to indicate the current full synchronization status, whether full synchronization is in progress. Its value is divided into two categories: "TRUE" indicates that full synchronization is in progress; "FALSE" indicates that full synchronization is not in progress. The data synchronization process can be found in [link to documentation]. Figure 4 Operations S211 to S213 are shown.

[0106] In one example, while a slot in the main hash table is undergoing full synchronization, the main thread is still processing client operation requests, such as write or read requests. The client's read request does not conflict with the data synchronization operation of the full synchronization thread; that is, regardless of whether the target slot requested by the client is undergoing full synchronization, data can be read from the hash table and the result returned normally.

[0107] In one example, when the target slot requested by the client happens to be in the process of synchronization, and the client's request is a write request, there may be a situation where write permissions for the target slot cannot be obtained, which may lead to data consistency issues. Therefore, in this embodiment of the disclosure, a dual hash table architecture is adopted, and a cache hash table is set up. When a write conflict occurs, the data is written to the cache hash table after a preset waiting time.

[0108] In one example, to ensure data consistency among multiple threads and avoid read / write conflicts, this embodiment employs a CAS mechanism. Specifically, the target slot and operation type are determined based on the key in the client operation command; the flag bit of the target slot is updated according to the client operation type; and data operations are performed based on the update return result. For detailed procedures, please refer to... Figures 5-7 The operating steps are shown.

[0109] The cached data full synchronization method provided by the embodiments of this disclosure utilizes multi-threading for full synchronization. When a full synchronization instruction is received from the master-slave node, a full synchronization thread is started to synchronize data. The main thread handles client operation requests. Data is stored using a dual-hash architecture: the main hash table stores normal data, and the cache hash table stores temporary incremental data when the two threads conflict. Upon receiving a client operation instruction, the target slot and client operation type are determined, and data operations are performed based on the target slot's flag, the full synchronization flag, and the client operation type. Compared to related technologies, the cached data full synchronization method provided by the embodiments of this disclosure improves and upgrades the Redis master-slave full synchronization mechanism, avoiding processing delays caused by the brief blocking of fork child processes during full synchronization, as well as the additional memory consumption caused by the COW mechanism during RDB generation, thereby improving physical memory utilization and avoiding resource waste.

[0110] The following will combine Figure 4 This section describes the process of data synchronization performed by a full synchronization thread. Figure 4 A flowchart illustrating a method for data synchronization using a full synchronization thread according to another embodiment of this disclosure is shown schematically. Figure 4 As shown, operation 210 includes operations S211 to S213.

[0111] In operation S211, update the full synchronization flag and the flag of the slot to be synchronized.

[0112] In operation S212, a data packing operation is performed on the slot to be synchronized.

[0113] After operation S213 is completed, the flag bit of the slot to be synchronized is restored.

[0114] In one example, the slot's flag is defined as HASH SLOT, with four values: HASH_SLOT_IDLE = 0; indicating no thread is operating on this slot; HASH_SLOT_SYNC = 1; indicating this slot is undergoing full synchronization (read-only); HASH_SLOT_WRITE = 2; indicating this slot is undergoing a write operation; HASH_SLOT_MERGE = 3; indicating a data merge update operation is in progress. These values ​​are only checked and processed when FULL_SYNC_FLAG = TRUE.

[0115] In one example, after starting the full synchronization thread, the full synchronization flag is first set to TRUE, and the CAS command CAS(SLOTn, HASH_SLOT_IDLE, HASH_SLOT_SYNC) is executed. This sets the flag of the slot to be synchronized to HASH_SLOT_SYNC, indicating that the slot is undergoing full synchronization. The initial flag of the slot is HASH_SLOT_IDLE, indicating that no thread is operating on the slot. A data packaging process is then performed on the current SLOT, and the flag is restored after completion: CAS(SLOTn, HASH_SLOT_SYNC, HASH_SLOT_IDLE). It should be noted that the above synchronization process is executed by the full synchronization thread and does not consider the possibility of client requests in the main thread. That is, the flag of the slot to be synchronized is HASH_SLOT_IDLE by default, therefore the CAS command succeeds and returns TRUE.

[0116] The following will combine Figures 5-7 The process of full data synchronization when a client initiates a data operation request will be described in turn. When a client request occurs on the main thread, the following situations may occur depending on the response time of the client request, the type of operation, and the execution order of the full synchronization thread.

[0117] Figure 5 The diagram schematically illustrates a flowchart of a method for performing data read / write operations based on a flag bit of the target slot, a full synchronization flag bit, and the client operation type, according to another embodiment of this disclosure. Figure 5 As shown, operation S230 includes operations S231 to S232.

[0118] In operation S231, when it is determined that the client operation type is a read operation, the result is returned based on the data of the target slot in the cache hash table.

[0119] Figure 6 A flowchart illustrating a data synchronization method for a client operation type of read operation, according to another embodiment of this disclosure, is shown schematically. Figure 6 As shown, operation S231 includes operation S310 and operation S320.

[0120] In operation S310, if it is determined that the data of the target slot in the cache hash table is empty, the data of the target slot in the main hash table is read.

[0121] In operation S320, if it is determined that the data of the target slot in the cache hash table is not empty, the data of the target slot in the cache hash table is read.

[0122] In one example, when a read-only command is received from a client, there will be no conflict or interference even if the main thread and the full synchronization thread operate on the same slot. The process involves searching for the target slot corresponding to the current key, checking if the target slot has data in the cached hash table, and if so, reading from the cached hash table and returning the result if there is data, indicating a previous write operation conflicted with the full synchronization thread. If the cached hash table is empty, the result is read from the main hash table and returned.

[0123] In operation S232, when it is determined that the client operation type is a write operation and the full synchronization flag is in the process of full synchronization, a data write operation is performed according to the flag of the target slot.

[0124] In one example, if the client performs a write operation, the operation type of the target slot, i.e., the current state of the target slot, needs to be determined by the target slot's flag. When it is determined that the client operation type is a write operation and the full synchronization flag is in progress, the data write operation is performed based on the target slot's flag. See [link to details] for further information. Figure 6 Operations S410 to S440 are shown.

[0125] Figure 7 The flowchart illustrates a data synchronization method for a client operation type of write operation according to another embodiment of the present disclosure.

[0126] like Figure 7 As shown, operation S232 includes operations S410 to S440.

[0127] In operation S410, if it is determined that the target slot is performing a full synchronization operation based on the flag bit of the target slot, the flag bit of the target slot is updated.

[0128] In operation S420, when it is determined that updating the flag bit of the target slot was successful, the client data is written to the target slot in the main hash table.

[0129] In operation S430, when it is determined that updating the flag bit of the target slot has failed, a blocking wait time is determined.

[0130] In operation S440, when the blocking wait time is greater than a preset threshold, the client data is written into the target slot in the cache hash table.

[0131] In one example, when the full synchronization flag is determined to be TRUE, the main thread executes the command CAS(SLOTn, HASH_SLOT_IDLE, HASH_SLOT_WRITE). If the main thread and the full synchronization thread are not operating on the same slot, i.e., the target slot's current flag is HASH_SLOT_IDLE, indicating that no thread is operating on this target slot, the CAS command will always return true because it is not operating on the same slot as the full synchronization thread. At this time, the target slot's flag is updated to HASH_SLOT_WRITE (writing operation in progress), allowing parallel operations without interference. The client write command is executed, and after the command is completed, the current target slot flag is restored.

[0132] In one example, when the full synchronization flag is determined to be TRUE, the main thread executes the command CAS(SLOTn, HASH_SLOT_IDLE, HASH_SLOT_WRITE). If the main thread and the full synchronization thread are operating on the same slot, and the full synchronization thread operates first, then the target slot's flag is HASH_SLOT_SYNC. In this case, the CAS command fails and returns false. To prioritize the timeliness of client request responses, this embodiment records the blocking wait time for each write operation. Specifically, the blocking wait time is determined by the time the main thread receives the client's write request, the first execution of the CAS command (time T1), and the current time (T2) after receiving the false return. If the blocking wait time is less than a preset threshold (T2-T1 < TIME_CLIENT_BLOCK), the CSA command is executed repeatedly to update the target slot's flag. If the blocking wait time is greater than the preset threshold (T2-T1 >= TIME_CLIENT_BLOCK), priority is given to ensuring the timeliness of client request responses, and retrying is stopped. The data is temporarily stored in a cache hash table, and the processing result is returned to the client.

[0133] After each client command is executed by the main thread, a data merging operation is performed between the cache hash table and the main hash table to ensure that the data in the main hash table is up-to-date and complete.

[0134] Figure 8a This illustration schematically shows one of the flowcharts of a data merging method provided according to yet another embodiment of the present disclosure. Figure 8b A second flowchart illustrating a data merging method according to yet another embodiment of this disclosure is shown. Figure 8a As shown, it includes operation S510.

[0135] In operation S510, data in the cache hash table is merged into the main hash table.

[0136] like Figure 8b As shown, operation 510 includes operations S511 to S512.

[0137] In operation S511, when it is determined that data exists in any slot of the cache hash table, the flag bit of that slot is changed.

[0138] In operation S512, the data in the cache hash table is moved to the main hash table.

[0139] According to embodiments of this disclosure, the timestamps of data in the same slot in the cached hash table and the main hash table are compared. If the timestamp of the data in the same slot in the cached hash table is later than the timestamp of the data in the same slot in the main hash table, then the data in the cached hash table slot is synchronized to the same slot in the main hash table.

[0140] In one example, after the main thread executes each client command, the `beforeSleep()` function in the main loop checks if the cache hash table `SLOTn` contains data. If data is present, the command is executed, attempting to update the `SLOTn` flag: `CAS(SLOTn, HASH_SLOT_IDLE, HASH_SLOT_MERGE)`. If the update succeeds, `CAS` returns `true`, the current data in `SLOTn` is moved to the main hash table to maintain data integrity and consistency, and its flag is restored to `CAS(SLOTn, HASH_SLOT_MERGE, HASH_SLOT_IDLE)`. If the update fails, `CAS` returns `false`, the current `SLOT` index is incremented by 1, and the above operation continues in the next loop cycle.

[0141] In one example, during the data merging process, data from the same slot can be merged based on its timestamp. Specifically, the data with the timestamp closest to the current moment is retained, i.e., the latest data.

[0142] The cached data full synchronization method provided in this disclosure uses the CAS mechanism to ensure data consistency between the main thread and the full synchronization thread without blocking, and merges the cached hash table with the main hash table in a timely manner to ensure the integrity and accuracy of the data in the main hash table.

[0143] Based on the above-described method for full synchronization of cached data, this disclosure also provides a device for full synchronization of cached data. The following will be combined with... Figure 9 The device is described in detail.

[0144] Figure 9 A schematic block diagram illustrating a cached data full synchronization apparatus according to an embodiment of the present disclosure is shown. Figure 9As shown, the cached data full synchronization device 800 of this embodiment includes a data synchronization module 810, a determination module 820 and a data operation module 830.

[0145] The data synchronization module 810 is used to respond to the master-slave node full synchronization command, start a full synchronization thread and set a full synchronization flag bit for data synchronization, wherein the full synchronization flag bit is used to represent the current full synchronization status of the master node. In one embodiment, the data synchronization module 810 can be used to execute the operation S210 described above, which will not be repeated here.

[0146] The determining module 820 is used to determine the target slot and the client operation type in response to the client's operation command. In one embodiment, the determining module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0147] The data operation module 830 is used to perform data read and write operations based on the flag bit of the target slot, the full synchronization flag bit, and the client operation type, wherein the flag bit of the slot is used to characterize the current operation type of the slot. In one embodiment, the data operation module 830 can be used to execute the operation S230 described above, which will not be repeated here.

[0148] According to embodiments of this disclosure, the data operation module includes: a first determination submodule and a second determination submodule.

[0149] The first determining submodule is used to return a result based on the data of the target slot in the cache hash table when the client operation type is determined to be a read operation. In one embodiment, the first determining submodule can be used to perform the operation S231 described above, which will not be repeated here.

[0150] The second determining submodule is used to perform a data write operation based on the flag bit of the target slot when it is determined that the client operation type is a write operation and the full synchronization flag is in progress. In one embodiment, the second determining submodule can be used to execute the operation S232 described above, which will not be repeated here.

[0151] According to embodiments of this disclosure, the second determining submodule includes: an update unit, a first determining unit, a second determining unit, and a write unit.

[0152] The update unit is configured to update the flag bit of the target slot if it is determined from the flag bit of the target slot that the target slot is undergoing a full synchronization operation. In one embodiment, the update unit may be used to perform the operation S410 described above, which will not be repeated here.

[0153] The first determining unit is configured to write client data into the target slot of the main hash table when it is determined that the update of the flag bit of the target slot is successful. In one embodiment, the first determining unit may be used to perform the operation S420 described above, which will not be repeated here.

[0154] The second determining unit is used to determine a blocking wait time when it fails to determine that updating the flag bit of the target slot is impossible. In one embodiment, the second determining unit may be used to perform the operation S430 described above, which will not be repeated here.

[0155] The write unit is used to write client data into the target slot in the cache hash table when the blocking wait time exceeds a preset threshold. In one embodiment, the write unit can be used to perform the operation S440 described above, which will not be repeated here.

[0156] According to embodiments of this disclosure, the first determining submodule includes a third determining unit and a fourth determining unit.

[0157] The third determining unit is configured to read the data of the target slot in the main hash table if it is determined that the data of the target slot in the cache hash table is empty. In one embodiment, the third determining unit may be used to perform the operation S310 described above, which will not be repeated here.

[0158] The fourth determining unit is configured to read the data of the target slot in the cache hash table if it is determined that the data of the target slot in the cache hash table is not empty. In one embodiment, the fourth determining unit may be used to perform the operation S320 described above, which will not be repeated here.

[0159] According to embodiments of this disclosure, it further includes a data merging module.

[0160] The data merging module is used to merge data from the cached hash table into the main hash table. In one embodiment, the data merging module can be used to perform the operation S510 described above, which will not be repeated here.

[0161] According to embodiments of this disclosure, the data merging module includes: a first update submodule and a data merging submodule.

[0162] The first update submodule is used to change the flag bit of any slot when it is determined that data exists in any slot of the cache hash table. In one embodiment, the first update submodule can be used to perform the operation S511 described above, which will not be repeated here.

[0163] The data merging submodule is used to move data from the cached hash table to the main hash table. In one embodiment, the data merging submodule can be used to perform the operation S512 described above, which will not be repeated here.

[0164] According to embodiments of this disclosure, the data synchronization module 810 includes: a second update submodule, a data packaging submodule, and a third update submodule.

[0165] The second update submodule is used to update the full synchronization flag and the flag of the slot to be synchronized. In one embodiment, the second update submodule can be used to perform the operation S211 described above, which will not be repeated here.

[0166] The data packaging submodule is used to perform a data packaging operation on the slot to be synchronized. In one embodiment, the data packaging submodule can be used to perform the operation S212 described above, which will not be repeated here.

[0167] The third update submodule is used to restore the flag bit of the slot to be synchronized after execution. In one embodiment, the third update submodule can be used to perform the operation S213 described above, which will not be repeated here.

[0168] According to embodiments of this disclosure, any plurality of modules among the data synchronization module 810, the determination module 820, and the data operation module 830 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the data synchronization module 810, the determination module 820, and the data operation module 830 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the data synchronization module 810, the determination module 820, and the data operation module 830 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0169] Figure 10 A block diagram schematically illustrates an electronic device suitable for implementing a method for full synchronization of cached data according to an embodiment of the present disclosure.

[0170] like Figure 10As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0171] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0172] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 908 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0173] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method for full synchronization of cached data according to embodiments of this disclosure.

[0174] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0175] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the full synchronization method for cached data provided in embodiments of this disclosure.

[0176] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0177] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0178] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0179] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0180] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0181] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0182] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for full synchronization of cached data, characterized in that, The method is applied to a cache database master node, where the master node uses a dual-hash architecture to store data. The dual-hash architecture includes a master hash table and a cache hash table, where the cache hash table is used to store temporary incremental data. In response to the master-slave node full synchronization command, a full synchronization thread is started and a full synchronization flag is set to perform data synchronization, wherein the full synchronization flag is used to characterize the current full synchronization status of the master node; In response to the client's operation commands, determine the target slot and the client operation type; and Data read and write operations are performed based on the target slot's flag bit, the full synchronization flag bit, and the client operation type, wherein the slot's flag bit is used to characterize the slot's current operation type. The data read / write operation based on the target slot's flag bit, the full synchronization flag bit, and the client operation type includes: When the client operation type is determined to be a read operation, the result is returned based on the data of the target slot in the cache hash table; When it is determined that the client operation type is a write operation and the full synchronization flag is in progress, a data write operation is performed according to the flag of the target slot. The data write operation based on the flag bit of the target slot includes: If it is determined from the flag bit of the target slot that the target slot is performing a full synchronization operation, update the flag bit of the target slot; When it is confirmed that updating the flag bit of the target slot was successful, the client data is written into the target slot of the main hash table; If updating the flag of the target slot fails, a blocking wait time is determined; and When the blocking wait time exceeds a preset threshold, the client data is written into the target slot in the cache hash table.

2. The method according to claim 1, characterized in that, The result returned based on the data of the target slot in the cache hash table includes: If it is determined that the data for the target slot in the cache hash table is empty, then the data for the target slot in the main hash table is read; and If it is determined that the data of the target slot in the cache hash table is not empty, then the data of the target slot in the cache hash table is read.

3. The method according to claim 1 or 2, characterized in that, After performing data read / write operations based on the target slot's flag, the full synchronization flag, and the client operation type, the process further includes: Merge the data in the cache hash table into the main hash table.

4. The method according to claim 3, characterized in that, The step of merging data from the cached hash table into the main hash table includes: When it is determined that data exists in any slot of the cache hash table, the flag bit of that slot is changed; and Move the data from the cache hash table to the main hash table.

5. The method according to claim 4, characterized in that, The step of starting the full synchronization thread and setting the full synchronization flag for data synchronization includes: Update the full synchronization flag and the flag of the slot to be synchronized; Perform a data packing operation on the slot to be synchronized; and After execution is complete, restore the flag bit of the slot to be synchronized.

6. A device for full synchronization of cached data, characterized in that, An apparatus for use with a cache database master node, wherein the master node uses a dual-hash architecture to store data, the dual-hash architecture including a master hash table and a cache hash table, the cache hash table being used to store temporary incremental data, the apparatus comprising: The data synchronization module is used to respond to the full synchronization command from the master and slave nodes, start a full synchronization thread, and set a full synchronization flag to perform data synchronization. The full synchronization flag is used to indicate the current full synchronization status of the master node. The determination module is used to determine the target slot and the client operation type in response to the client's operation instructions; and The data operation module is used to perform data read and write operations based on the flag bit of the target slot, the full synchronization flag bit, and the client operation type, wherein the flag bit of the slot is used to characterize the current operation type of the slot; The data operation module is specifically used to return a result based on the data of the target slot in the cache hash table when the client operation type is determined to be a read operation; and to perform a data write operation based on the flag bit of the target slot when the client operation type is determined to be a write operation and the full synchronization flag bit is in progress. The data operation module is also used to update the flag bit of the target slot if it is determined that the target slot is performing a full synchronization operation based on the flag bit of the target slot; when it is determined that updating the flag bit of the target slot is successful, write the client data into the target slot of the main hash table; when it is determined that updating the flag bit of the target slot fails, determine the blocking waiting time; and when the blocking waiting time is greater than a preset threshold, write the client data into the target slot of the cache hash table.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors execute the cache data full synchronization method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method for full synchronization of cached data according to any one of claims 1 to 5.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method for full synchronization of cached data according to any one of claims 1 to 5.

Images