Data change processing method and device

Abstract

The invention provides a data change processing method and device, and relates to the technical field of data processing. The method comprises the following steps: issuing a change record to a cache node, updating a database storage table in a database node, and waiting for the cache node to send a change start notification message; the cache node sends a change start notification message to the data modification node; if the data modification node receives change start notification messages corresponding to all other service nodes respectively, submitting a data modification transaction and issuing a change completion message to the cache node; and the cache node updates the local cache according to the change completion message and the change record to complete the change processing. The device executes the method. According to the data change processing method and device provided by the embodiment of the invention, the consistency of the data in the local cache and the database can be ensured, and the data change processing response time is shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a data change processing method and device. Background Art

[0002] In a distributed system architecture, i.e., a distributed framework, with the growth of business scale and data access volume, cache technology has gradually become the core means to improve system performance. Among them, maintaining the consistency between cached data and database persistent data is an important evaluation index for the cache framework.

[0003] Traditional local cache frameworks (such as Ehcache, GuavaCache, etc.) achieve extremely low-latency responses by directly accessing cached data in memory, and it is easy to achieve strong consistency of single-node caches in terms of cache consistency. Their single-node characteristics lead to the problem of data islands in a distributed environment and cannot achieve cross-node cache consistency.

[0004] Mainstream centralized cache systems (such as Redis, Memcached, etc.) can partially solve the problem of cache data consistency through centralized storage. However, the additional network communication overhead significantly increases the system response time. Summary of the Invention

[0005] In view of the problems in the prior art, embodiments of the present invention provide a data change processing method and device, which can at least partially solve the problems existing in the prior art.

[0006] On the one hand, the present invention proposes a data change processing method, which is applied to a distributed framework; the distributed framework includes service nodes corresponding to each business service respectively, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; the data change processing method includes:

[0007] During the process of modifying data, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for a change start notification message sent by the cache node;

[0008] Wherein, the data modification node is the service node that executes the modification process, and the cache node is the node of the table to which the data to be modified cached locally by all service nodes belongs;

[0009] The cache node processes the data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node;

[0010] If the modified data node receives the change start notification messages respectively corresponding to all other service nodes, it submits a data modification transaction and publishes a change completion message to the cache node;

[0011] The cache node updates the local cache according to the change completion message and the change record, and completes the change processing.

[0012] Among them, the change record includes a new data record, a modified data record, and a deleted data record; correspondingly, the cache node processes data according to the change record, including:

[0013] For the new data record, if it is determined that multiple new data records are received for the same primary key, the multiple new data records are stored at the same level;

[0014] For the modified data record, the cache record that meets the modification conditions in the local cache is converted into a change record and placed in the change record list;

[0015] For the deleted data record, the cache record that meets the deletion conditions in the local cache is converted into a change record and placed in the change record list.

[0016] Among them, the cache node updates the local cache according to the change completion message and the change record, and completes the change processing, including:

[0017] Calculate the range to be updated in the local cache according to the change record, and read data from the database node according to the range and update it to the local cache.

[0018] Among them, the data change processing method further includes:

[0019] Calculate the target change records that have timed out according to the time when the change records are added to the change record list;

[0020] If the change record check process for all target change records is completed, the change record check is completed.

[0021] Among them, the data change processing method further includes:

[0022] Every time a target change record check is completed, update the local cache according to the target change record for which the check is completed, and complete the change processing.

[0023] Among them, the data change processing method further includes:

[0024] During the data query process, if it is determined that the query condition exists in the change records in the local cache, query the database row version;

[0025] If it is determined that the database row version is inconsistent with the row version in the local cache, query the database in a key-value (KV) manner and update the local cache according to the query result.

[0026] On the one hand, the present invention provides a data change processing device, which is applied to a distributed framework; the distributed framework includes service nodes respectively corresponding to each business service, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; the data change processing device includes:

[0027] A publishing unit, configured to publish a change record to the cache node by modifying a data node during the data modification process, update a database storage table in the database node, and wait for a change start notification message sent by the cache node;

[0028] Wherein, the modified data node is a service node executing a modification process, and the cache node is a node of a table to which data to be modified cached locally by all service nodes belongs;

[0029] A caching unit, configured to perform data processing on the change record by the cache node, cache the change record after data processing, and send a change start notification message corresponding to each other service node except the modified data node to the modified data node;

[0030] A submission unit, configured to submit a data modification transaction and send a change completion message to the cache node if change start notification messages corresponding to all other service nodes are received by the modified data node;

[0031] An update unit, configured to update the local cache by the cache node according to the change completion message and the change record to complete the change processing.

[0032] On the other hand, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following method is implemented:

[0033] During the data modification process, the modified data node publishes a change record to the cache node, updates the database storage table in the database node, and waits for a change start notification message sent by the cache node;

[0034] Wherein, the modified data node is a service node executing a modification process, and the cache node is a node of a table to which data to be modified cached locally by all service nodes belongs;

[0035] The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the modification data node to the modification data node;

[0036] If the modification data node receives the change start notification messages corresponding to all other service nodes, it submits a data modification transaction and publishes a change completion message to the cache node;

[0037] The cache node updates the local cache according to the change completion message and the change record to complete the change processing.

[0038] An embodiment of the present invention provides a computer-readable storage medium, including:

[0039] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following method is implemented:

[0040] During the data modification process, the modification data node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0041] Wherein, the modification data node is a service node that executes the modification process, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs;

[0042] The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the modification data node to the modification data node;

[0043] If the modification data node receives the change start notification messages corresponding to all other service nodes, it submits a data modification transaction and publishes a change completion message to the cache node;

[0044] The cache node updates the local cache according to the change completion message and the change record to complete the change processing.

[0045] An embodiment of the present invention further provides a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, the following method is implemented:

[0046] During the data modification process, the modification data node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0047] Among them, the modified data node is a service node that executes the modification processing procedure, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs;

[0048] The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the modified data node to the modified data node;

[0049] If the modified data node receives the change start notification messages corresponding to all other service nodes, it submits a data modification transaction and publishes a change completion message to the cache node;

[0050] The cache node updates the local cache according to the change completion message and the change record to complete the change processing.

[0051] The data change processing method and device provided by the embodiments of the present invention publish a change record from a modified data node to the cache node during the data modification process, update the database storage table in the database node, and wait for the cache node to send a change start notification message; among them, the modified data node is a service node that executes the modification processing procedure, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs; the cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the modified data node to the modified data node; if the modified data node receives the change start notification messages corresponding to all other service nodes, it submits a data modification transaction and publishes a change completion message to the cache node; the cache node updates the local cache according to the change completion message and the change record to complete the change processing, which can ensure the data consistency between the local cache and the database and reduce the data change processing response time. Description of the Drawings

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. In the drawings:

[0053] Figure 1 is a flowchart of a data change processing method provided by an embodiment of the present invention.

[0054] Figure 2 is a structural diagram of a distributed framework provided by an embodiment of the present invention.

[0055] Figure 3 It is a schematic flowchart of a data change processing method provided by another embodiment of the present invention.

[0056] Figure 4 It is a schematic flowchart of a data change processing method provided by another embodiment of the present invention.

[0057] Figure 5 It is a schematic flowchart of a data change processing method provided by another embodiment of the present invention.

[0058] Figure 6 It is a schematic flowchart of a data change processing method provided by another embodiment of the present invention.

[0059] Figure 7 It is a schematic structural diagram of a data change processing device provided by an embodiment of the present invention.

[0060] Figure 8 It is a schematic diagram of the entity structure of a computer device provided by an embodiment of the present invention. Detailed implementation manners

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer and more understandable, the following further describes the embodiments of the present invention in detail with reference to the accompanying drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention. It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be arbitrarily combined with each other.

[0062] Explanation of related terms:

[0063] Distributed cache: A cache mechanism that disperses data storage across multiple nodes to improve data access performance and availability.

[0064] Strong consistency algorithm: A computational method used to ensure that in a distributed environment, no matter which node accesses the data, the latest and consistent results can be obtained.

[0065] Row version: Records the time of data. Whenever data is updated, the time of recording the data is updated, and it is ensured that the updated time of recording the data is globally unique.

[0066] Figure 1 It is a schematic flowchart of a data change processing method provided by an embodiment of the present invention, as Figure 1As shown in the figure, the data change processing method provided by the embodiment of the present invention is applied to a distributed framework; the distributed framework includes service nodes corresponding to each business service respectively, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; the data change processing method includes:

[0067] Step S1: During the process of modifying data, the modifying data node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message.

[0068] Wherein, the modifying data node is a service node that executes the modification process, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs.

[0069] Step S2: The cache node processes the data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the modifying data node to the modifying data node.

[0070] Step S3: If the modifying data node receives the change start notification messages corresponding to all other service nodes respectively, it submits a data modification transaction and publishes a change completion message to the cache node.

[0071] Step S4: The cache node updates the local cache according to the change completion message and the change record to complete the change processing.

[0072] In the above step S1, during the process of modifying data, the modifying data node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message.

[0073] Wherein, the modifying data node is a service node that executes the modification process, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs. The device may be a computer device that executes this method. In the technical solution of this application, the acquisition, storage, use, processing, etc. of data all comply with relevant regulations.

[0074] In the above step S2, the cache node of the device processes the data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the modifying data node to the modifying data node.

[0075] In the above step S3, if the device's modified data node receives the change start notification messages respectively corresponding to all other service nodes, it submits a data modification transaction and publishes a change completion message to the cache node.

[0076] In the above step S4, the device's cache node updates the local cache according to the change completion message and the change record, completing the change processing.

[0077] As Figure 2 shown, the distributed framework is described as follows:

[0078] The distributed framework includes a gateway service, a business service, a centralized cache service, and a database service, where:

[0079] The gateway service is responsible for request distribution, does not cache data, and does not need to care about data consistency.

[0080] The business service is responsible for executing specific business processes, and locally caches a service node list and business-related data that needs to be cached, as well as the corresponding change records. Each business service can handle different businesses. Therefore, the data cached in the service nodes can have both the same parts and different parts, and the minimum granularity of the cached data is a table. Taking the business 1 service as an example, this business service can include multiple service nodes that execute in parallel. The local cache node list in each service node is the node list of all service nodes included in this business service; the local cache table 1 in each service node records the data that needs to be cached related to this business service; the local cache table 1 change record in each service node can record the change records related to this business service in the form of a change record list. The method of the present invention is described by taking each business service including one service node as an example.

[0081] The centralized cache service is responsible for storing all service node information, and divides the service node information into three parts: a centralized cache cluster version, a centralized cache node list, and a centralized cache message list.

[0082] Among them, the centralized cache cluster version is used to mark the changes of nodes in the distributed framework (specifically, the service nodes added and deleted in the centralized cache cluster). The centralized cache node list records all service nodes and cache-related information, specifically the sum of the local cache tables of multiple business services such as business 1 service and business 2 service; the centralized cache message list records the historical messages generated by all service nodes, specifically including change record historical messages. The messages in the centralized cache message list can be processed according to the preset trigger conditions in the order from first to last when joining the centralized cache message list. Therefore, the centralized cache message list records the historical messages generated by all service nodes recently.

[0083] The database service is responsible for persisting the data generated during the business processing, and realizes the strong consistency of data storage by controlling database transactions. In addition, it also provides an interface for querying the latest strongly consistent persisted data.

[0084] When the service node accesses the local cache, the process of maintaining strong consistency with the database is elaborated, including four main processes, namely, modifying data, querying data, node joining, and node leaving.

[0085] The process of modifying data is mainly divided into two parts: the first part is the processing process of modifying the data node, and the second part is the processing process of the cache node. The two parts are executed in parallel successively. The specific process is as follows:

[0086] During the process of modifying data, the strong consistency of data update is guaranteed by the database. During the modification process, a change record is generated for the modified data, and the situation of data change is notified to each node as much as possible through two asynchronous notifications. When other nodes do not feedback the receipt of the change record for a long time, the data update transaction is rolled back to achieve the purpose of data strong consistency.

[0087] As Figure 3 shown, 1->3->4->6->8 in the processing process of modifying the data node is a common data modification process, and three processing steps of 2, 5, and 7 are inserted in this process:

[0088] Among them, step 2 is to analyze the data to be updated (inserted, deleted, modified) in step 4, generate a change record and notify all cache data nodes, so that other nodes can know which data the current node is about to update. In order to reduce the transaction occupation time, step 2 is generally executed before step 3. In the specific implementation, it is also possible for step 2 to be after step 3 and before step 4, which does not affect the correctness of the method.

[0089] Step 5 is to wait for all nodes to feedback that they have received the change start message, indicating that other nodes know which data the current node is about to update. After all nodes have feedback, the current node can safely commit the transaction, so it is executed after step 4 and before step 6.

[0090] Step 7 is to notify all cache data nodes that the previous data update processing has been completed. After the transaction of modifying data is committed, the current node notifies other nodes that the current data modification has been completed, giving other nodes the opportunity to process the updated data.

[0091] The cache node is responsible for processing the two data change messages sent by the data modification node, extracting the change records in the messages and synchronizing them to the local change record list, and triggering the change completion processing when the preset trigger condition is met.

[0092] Change start processing procedure: First, extract the change records in the message and add them to the local change record list. Additionally, extra processing needs to be performed on the change records for addition, deletion, and modification respectively:

[0093] (1) For newly added data records, since the generation of the change records precedes the update of the database, multiple new records may be received for the same primary key. These records need to be stored at the same level to ensure that subsequent data records do not overwrite earlier ones.

[0094] (2) For modified data records, the cached records that meet the modification conditions in the local cache need to be converted into change records and placed in the change record list.

[0095] (3) For deleted data records, the cached records that meet the deletion conditions in the local cache need to be converted into change records and placed in the change record list.

[0096] (4) During the modification and deletion processes, conversion needs to be performed from the local cache. For the first time, directly convert and store them in the change record list, and then simply increase the reference count subsequently to achieve the purpose of storing data at the same level.

[0097] Finally, the cache node notifies the modified data node that the change start message has been received.

[0098] Change completion processing procedure: First, calculate the range to be updated in the local cache based on the change records, read data from the database node according to this range, and update the local cache. The method for calculating the range is as follows:

[0099] For newly added records, the range is the primary key of the record.

[0100] For modified and deleted records, the range is the primary key with a reference count of 1 for the records generated in the local cache during the change start processing procedure.

[0101] Finally, delete the current completed change records and the associated records with a reference count of 1.

[0102] Change record inspection procedure: The change record inspection can be triggered periodically. First, calculate the target change records that have timed out based on the time when the change records are added to the local change record list, and determine whether to trigger the change record inspection procedure according to different strategies.

[0103] The present invention provides two strategies:

[0104] (1) Direct trigger, that is, trigger according to the regular time.

[0105] (2) Decide by asking the modified data node. There are three situations that will trigger:

[0106] a. When the modified data node has been removed from the node list.

[0107] b. No feedback is obtained after repeatedly querying the modified data node.

[0108] c. The feedback change record of the modified data node has been completed.

[0109] Finally, it is judged whether to end the inspection. If so, the processing process is exited; if not, the timed-out change records are processed again.

[0110] For querying data:

[0111] When querying data, it is determined whether to use the local cache or the database for querying by judging whether the query condition hits the local change record, where the change record is generated during the process of modifying data. The specific process is as Figure 4 shown:

[0112] During the process of querying data, if the change record in the local cache snapshot does not hit the query condition, it means that the data to be queried has not been modified by other nodes at this moment. Therefore, this part of the cached data to be queried is strongly consistent. Even if a modification processing process can start instantaneously and successfully commit the transaction after judging that the local cache change record is hit, since it is carried out after the query processing process starts, it has no impact on the result.

[0113] If the change record in the local cache snapshot hits the query condition, it means that other nodes are modifying the data to be queried, and the transaction commit time is uncertain for the current node. Therefore, the query is transferred to the database, and the database ensures strong consistency. The specific steps are as follows:

[0114] During the process of querying data, if it is determined that the query condition exists in the change record in the local cache, then the database row version is queried;

[0115] If it is determined that the database row version is inconsistent with the row version in the local cache, then the database is queried in a KV manner, and the local cache is updated according to the query result.

[0116] If it is determined that the database row version is consistent with the row version in the local cache, then the query is based on the cache snapshot.

[0117] The query process 1->2->3->4 is a common processing process for cache hits and will not be described in detail here. When the query condition hits the change record, the processing process of step 5 will start.

[0118] The 5-step process can avoid unnecessary duplicate data access by row version checking when querying the database, which can relieve the database pressure. By filtering the data being modified through the local cache snapshot based on the query conditions, only this small portion of data needs to go through the 6-step process, avoiding the possibility of a cache avalanche caused by a large-scale cache penetration.

[0119] In the 6-step process, based on the result of filtering the local cache snapshot in the 5-step process, the primary key of the data being modified is obtained from the change record, and the database is queried in a KV manner. The database primary key index can be directly utilized to implement data query in a high-performance manner. After the database query in the 6-step process is completed, the queried data can be handed over to the 7-step process for processing asynchronously.

[0120] In the 7-step process, during the local cache update, the local cache can be quickly updated in a cas lock-free manner, enabling subsequent cache queries to determine that the database version is consistent with the local cache version after the 5-step process, thus avoiding excessive processing in the 6-step process.

[0121] The data change processing method further includes:

[0122] Adding an out-of-cluster node to the local cache cluster to obtain the service node includes:

[0123] The out-of-cluster node broadcasts the registration information of the out-of-cluster node to the in-cluster nodes;

[0124] The in-cluster nodes receive the registration information, record the registration information in the local cache, and send a reply message for the registration information to the out-of-cluster node;

[0125] If the out-of-cluster node determines that it has received the reply messages sent by all in-cluster nodes within a preset time period, it updates the local cache according to the out-of-cluster node information and adds the out-of-cluster node to the local cache cluster to obtain the service node.

[0126] That is, the node addition process:

[0127] Before constructing the local cache, the service node needs to be added to the local cache cluster (i.e., the centralized cache cluster). The cache nodes adopt a leaderless mechanism to equally update the cluster node information. When all existing in-cluster nodes have received the new node registration message, it is considered that the new node has successfully joined the cluster. To ensure that node information is not lost, the cache node information is stored using the centralized cache service. After the node joins the local cache cluster, it can provide cache services internally.

[0128] Such as Figure 5As shown in the figure, during the process of a new node joining the cluster, if only some of the nodes within the cluster reply to the new node's registration message, for these nodes within the cluster, the new node is already regarded as a node within the cluster. Therefore, during the data modification process, the change record will be notified to the new node, and the new node will reply that it has received the notification even if it has not yet joined the cluster, thus not affecting the data modification process of these nodes within the cluster. For the nodes within the cluster that have not received the registration message, although the change record has not been notified to the new node, since the new node has not completed the registration yet, it will not affect the cache correctness of the entire system either.

[0129] 1->2->3->4->5->6 is a processing process for an external node of the cluster to join the cluster. Steps 1 and 6 are the start and end nodes, which will not be described here.

[0130] Step 2 mainly writes the information of the external node of the cluster into the centralized cache service. Before writing, it first reads out the existing cluster node information and writes its own node information in a locked manner.

[0131] Step 3 broadcasts the registration information to all nodes within the cluster. After the broadcast is completed, the processing of Step 4 can be carried out.

[0132] Step 4 is to wait for all nodes within the cluster to reply. If all nodes within the cluster have not replied for a long time, it is considered that the attempt to join the cluster this time has failed and it is necessary to return to Step 2. If successful, the processing of Step 5 will start.

[0133] In Step 5, it has already successfully joined the cluster, so the local cache can be enabled. Here, cache preheating operations can be carried out to load the data that needs to be cached in the database into the memory. Before cache preheating, the change record is synchronized from the centralized cache service to the local cache. In this way, the data in the local cache is either the latest or marked as being modified by the change record.

[0134] 7->8->9 is the processing process for each node within the cluster. In Step 7, after receiving the message, the new node registration information is obtained.

[0135] Step 8 records the registration information in the local cache.

[0136] Step 9 notifies the new node that it is aware of the registration information. After this process is completed, the nodes within the cluster already regard the new node as a node within the cluster.

[0137] The data change processing method further includes:

[0138] Deleting a node within the cluster from the local cache cluster includes:

[0139] The nodes in the cluster regularly update the timestamp of the registration information of the nodes in the cluster in the centralized caching service. If it is determined that the update fails, the status of the nodes in the cluster is determined to be unavailable, and a notification message carrying the unavailable status is sent to other nodes in the local caching cluster;

[0140] One of the other nodes in the local caching cluster deletes the nodes in the cluster with the status of unavailable from the local caching cluster.

[0141] The process of a node leaving the cluster can include two parts:

[0142] The first part is that the current node itself detects that the node information has expired and actively notifies to leave the cluster; the second part is that other nodes detect that the current node information has expired and delete the node registration information. After a node leaves the cluster, it will no longer provide caching services.

[0143] As Figure 6 shown, the description of the above first part is as follows:

[0144] It includes the node keep-alive process 1->2->3->4->5 and the message processing process 6->7.

[0145] Among them, in step 2, the timestamp of its own registration information is updated in the centralized caching service. If the update is successful, after waiting for a period of time, the process of step 2 is repeated. If the update fails, it is considered that the current node has expired, and the process of step 3 is carried out.

[0146] In step 3, the status of the local node is switched to unavailable, and at this time, caching services are no longer provided.

[0147] In step 4, the current node actively notifies other nodes in the cluster that it has left the cluster, so that the information that the current node has expired can be synchronized to the cluster in time, preventing other nodes from having too long a waiting time before submitting a data modification transaction.

[0148] In step 6, after receiving the message, the information of the node leaving the cluster is extracted.

[0149] In step 7, the corresponding node in the local cache is deleted.

[0150] The data change processing method further includes:

[0151] Deleting the nodes in the cluster from the local caching cluster includes:

[0152] The nodes in the cluster regularly query for active nodes in the centralized caching service. If it is determined that the time difference between the current query time of an active node and its previous query time is less than a preset duration threshold, then the registered nodes in the local cache are compared with the active nodes, and the active nodes that are different from the registered nodes in the local cache are regarded as inactive nodes, and the inactive nodes are deleted from the centralized caching service and the local cache cluster.

[0153] As Figure 6 shown, the second part above is described as follows:

[0154] It includes the node liveness detection process of 8 -> 9 -> 10 -> 11 -> 7.

[0155] Among them, in step 9, the registered nodes are queried in the centralized caching service, the timestamp of the node is recorded as the current node activity time, and it is compared with the previous node activity time. If the time difference is less than the threshold, it is marked as an active node.

[0156] If the current node is not an active node, exit the liveness detection process.

[0157] If the current node is an active node, proceed to the next step of processing.

[0158] In step 10, the registered nodes in the local cache are compared with the active nodes, and the nodes that are not active nodes are filtered out. In step 11, the inactive nodes are deleted from the centralized caching service, and at the same time, step 7 is called to delete the nodes in the local cache as well.

[0159] The data change processing method provided by the embodiments of the present invention has the following beneficial technical effects:

[0160] 1. Eliminate the delay in data update, ensure that the data read at any time and any node is the latest and consistent, and overcome the consistency deviation caused by data synchronization delay in the prior art.

[0161] 2. Solve the single point of failure problem, avoid the interruption of data writing and consistency maintenance of the entire system due to the failure of a single node, and improve the reliability and stability of the system.

[0162] 3. Effectively handle conflicts in high-concurrency write operations, prevent data overwriting, loss or incorrect update, and ensure strong consistency of data in high-concurrency scenarios.

[0163] 4. Enhance the adaptability to node errors. In the case of node errors, the consistency state of the entire system can still be maintained, and the consistency state of the node can be quickly and accurately restored after the faulty node recovers.

[0164] 5. Reduce the complexity of the system and the development and maintenance costs, and provide a relatively simple and easy-to-understand and implement consistency algorithm without sacrificing the ability to guarantee consistency.

[0165] 6. Optimize the performance and scalability of the system so that the algorithm can still maintain efficient operation and good consistency maintenance ability under large-scale data and an increasing number of nodes.

[0166] In the data change processing method provided by the embodiment of the present invention, during the process of modifying data, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for a change start notification message sent by the cache node; wherein, the data modification node is a service node that executes the modification processing process, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs; the cache node processes the data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node; if the data modification node receives the change start notification messages corresponding to all other service nodes respectively, it submits a data modification transaction and publishes a change completion message to the cache node; the cache node updates the local cache according to the change completion message and the change record to complete the change processing, which can ensure the data consistency between the local cache and the database and reduce the data change processing response time.

[0167] Further, the change record includes a new data record, a modified data record, and a deleted data record; correspondingly, the cache node processes the data according to the change record, including:

[0168] For the new data record, if it is determined that multiple new data records are received for the same primary key, the multiple new data records are stored at the same level; reference can be made to the above embodiment for description and will not be repeated here.

[0169] For the modified data record, the cached record in the local cache that meets the modification condition is converted into a change record and placed in the change record list; reference can be made to the above embodiment for description and will not be repeated here.

[0170] For the deleted data record, the cached record in the local cache that meets the deletion condition is converted into a change record and placed in the change record list. Reference can be made to the above embodiment for description and will not be repeated here.

[0171] Further, the cache node updates the local cache according to the change completion message and the change record to complete the change processing, including:

[0172] Calculate the range to be updated in the local cache according to the change record, and read data from the database node according to the range and update it to the local cache. For details, reference may be made to the above embodiments and will not be elaborated here.

[0173] Further, the data change processing method further includes:

[0174] Calculate the target change records that have timed out according to the time when the change records are added to the change record list; for details, reference may be made to the above embodiments and will not be elaborated here.

[0175] If the change record check process for all target change records is completed after traversal, the change record check is completed. For details, reference may be made to the above embodiments and will not be elaborated here.

[0176] Further, the data change processing method further includes:

[0177] After each target change record check is completed, update the local cache according to the target change record for which the check is completed, and complete the change processing. For details, reference may be made to the above embodiments and will not be elaborated here.

[0178] Further, the data change processing method further includes:

[0179] During the data query process, if it is determined that the query condition exists in the change records in the local cache, query the database row version; for details, reference may be made to the above embodiments and will not be elaborated here.

[0180] If it is determined that the database row version is inconsistent with the row version in the local cache, query the database in a KV manner and update the local cache according to the query result. For details, reference may be made to the above embodiments and will not be elaborated here.

[0181] Figure 7 It is a schematic structural diagram of a data change processing device provided by an embodiment of the present invention. As Figure 7 shown, the data change processing device provided by the embodiment of the present invention is applied to a distributed framework; the distributed framework includes service nodes corresponding to each business service respectively, each service node is connected to a cache node corresponding to the centralized cache service, and each service node is connected to a database node corresponding to the database service; the data change processing device includes a publishing unit 701, a caching unit 702, a submission unit 703, and an updating unit 704, where:

[0182] The publishing unit 701 is used to publish a change record to the cache node by modifying a data node during the data modification process, update the database storage table in the database node, and wait for the cache node to send a change start notification message; wherein, the data modification node is a service node that executes the modification process, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs; the caching unit 702 is used to process data by the cache node according to the change record, cache the change record after data processing, and send a change start notification message corresponding to each other service node except the data modification node to the data modification node; the submission unit 703 is used to submit a data modification transaction and send a change completion message to the cache node if all change start notification messages corresponding to each other service node are received by the data modification node; the update unit 704 is used to update the local cache by the cache node according to the change completion message and the change record to complete the change process.

[0183] Specifically, the publishing unit 701 in the device is used to publish a change record to the cache node by modifying a data node during the data modification process, update the database storage table in the database node, and wait for the cache node to send a change start notification message; wherein, the data modification node is a service node that executes the modification process, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs; the caching unit 702 is used to process data by the cache node according to the change record, cache the change record after data processing, and send a change start notification message corresponding to each other service node except the data modification node to the data modification node; the submission unit 703 is used to submit a data modification transaction and send a change completion message to the cache node if all change start notification messages corresponding to each other service node are received by the data modification node; the update unit 704 is used to update the local cache by the cache node according to the change completion message and the change record to complete the change process.

[0184] The data change processing device provided by the embodiment of the present invention publishes a change record from a data modification node to the cache node during the data modification process, updates the database storage table in the database node, and waits for a change start notification message sent by the cache node; wherein, the data modification node is a service node that executes the modification process, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs; the cache node processes the data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node; if the data modification node receives the change start notification messages corresponding to all other service nodes respectively, it submits a data modification transaction and publishes a change completion message to the cache node; the cache node updates the local cache according to the change completion message and the change record to complete the change process, which can ensure the data consistency between the local cache and the database and reduce the data change processing response time.

[0185] The embodiment of the data change processing device provided by the embodiment of the present invention can be specifically used to execute the processing flow of the above method embodiments, and its functions will not be elaborated here. For details, please refer to the detailed description of the above method embodiments.

[0186] Figure 8 The schematic physical structure diagram of the computer device provided by the embodiment of the present invention is as Figure 8 shown. The computer device includes: a memory 801, a processor 802, and a computer program stored on the memory 801 and executable on the processor 802. When the processor 802 executes the computer program, the following method is implemented:

[0187] During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for a change start notification message sent by the cache node;

[0188] Wherein, the data modification node is a service node that executes the modification process, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs;

[0189] The cache node processes the data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node;

[0190] If the data modification node receives the change start notification messages corresponding to all other service nodes respectively, it submits a data modification transaction and publishes a change completion message to the cache node;

[0191] The cache node updates the local cache according to the change completion message and the change record, and completes the change processing.

[0192] This embodiment discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, the following method is implemented:

[0193] During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0194] Among them, the data modification node is a service node that executes the modification process, and the cache node is the node of the table to which the data to be modified cached locally by all service nodes belongs;

[0195] The cache node processes the data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node;

[0196] If the data modification node receives the change start notification messages corresponding to all other service nodes respectively, it submits a data modification transaction and publishes a change completion message to the cache node;

[0197] The cache node updates the local cache according to the change completion message and the change record, and completes the change processing.

[0198] This embodiment provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the following method is implemented:

[0199] During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0200] Among them, the data modification node is a service node that executes the modification process, and the cache node is the node of the table to which the data to be modified cached locally by all service nodes belongs;

[0201] The cache node processes the data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node;

[0202] If the data modification node receives the change start notification messages corresponding to all other service nodes respectively, it submits a data modification transaction and publishes a change completion message to the cache node;

[0203] The cache node updates the local cache according to the change completion message and the change record, and completes the change processing.

[0204] Compared with the technical solutions in the prior art, the data change processing method provided by the embodiment of the present invention publishes a change record from a data modification node to the cache node during the data modification process, updates the database storage table in the database node, and waits for a change start notification message sent by the cache node; wherein, the data modification node is a service node that executes the modification process, and the cache node is a node of the table to which the data to be modified cached locally by all service nodes belongs; the cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node; if the data modification node receives the change start notification messages corresponding to all other service nodes, it submits a data modification transaction and publishes a change completion message to the cache node; the cache node updates the local cache according to the change completion message and the change record, and completes the change processing, which can ensure the data consistency between the local cache and the database and reduce the data change processing response time.

[0205] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0207] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 specified in one block or multiple blocks.

[0208] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 specified in one block or multiple blocks.

[0209] In the description of this specification, the descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0210] The specific embodiments described above further elaborate on the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data change processing method, characterized in that The described data change processing method is applied to a distributed framework; the distributed framework includes service nodes corresponding to respective business services, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; The described data change processing method includes: During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for a change start notification message sent by the cache node; Wherein, the data modification node is the service node executing the modification process, and the cache node is the node of the table to which the data to be modified cached locally by all service nodes belongs; The cache node processes the data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node; If the data modification node receives the change start notification messages corresponding to all other service nodes respectively, it submits a data modification transaction and publishes a change completion message to the cache node; The cache node updates the local cache according to the change completion message and the change record to complete the change processing.

2. The data change processing method according to claim 1, wherein The change record includes a new data record, a modified data record, and a deleted data record; correspondingly, the cache node processes the data according to the change record, including: For the new data record, if it is determined that multiple new data records are received for the same primary key, the multiple new data records are stored at the same level; For the modified data record, the cached record in the local cache that meets the modification condition is converted into a change record and placed in the change record list; For the deleted data record, the cached record in the local cache that meets the deletion condition is converted into a change record and placed in the change record list.

3. The data change processing method according to claim 1, characterized in that The cache node updates the local cache according to the change completion message and the change record to complete the change processing, including: Calculating the range to be updated in the local cache according to the change record, and reading data from the database node according to the range and updating it to the local cache.

4. The data change processing method according to claim 1, wherein The data change processing method further includes: Calculating the target change records that have timed out according to the time when the change records are added to the change record list; If the change record check process for all target change records is completed after traversal, the change record check is completed.

5. The data change processing method according to claim 4, wherein The data change processing method further includes: After each target change record check is completed, the local cache is updated according to the target change record for which the check is completed to complete the change processing.

6. The data change processing method according to claim 1, characterized in that, The data change processing method further includes: During the data query process, if it is determined that the query condition exists in the change records in the local cache, the database row version is queried; If it is determined that the database row version is inconsistent with the row version in the local cache, the database is queried in a KV manner and the local cache is updated according to the query result.

7. A data change processing device, characterized in that The data change processing device is applied to a distributed framework; the distributed framework includes service nodes corresponding to respective business services, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; The data change processing device includes: A publishing unit, configured to publish a change record to the cache node by modifying a data node during the data modification process, update a database storage table in the database node, and wait for a change start notification message sent by the cache node; Wherein, the data modification node is a service node that executes a modification process, and the cache node is a node of a table to which data to be modified cached locally by all service nodes belongs; A caching unit, configured to perform data processing by the cache node according to the change record, cache the change record after data processing, and send a change start notification message corresponding to each other service node except the data modification node to the data modification node; A submission unit, configured to, if receiving change start notification messages corresponding to all other service nodes respectively through the data modification node, submit a data modification transaction and publish a change completion message to the cache node; An updating unit, configured to update a local cache by the cache node according to the change completion message and the change record to complete the change processing.

8. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images