Data change processing method and device

Abstract

The present invention provides a data change processing method and device, which relate to the field of data processing technology. The method includes publishing a change record to a cache node, updating a database storage table in a database node, and waiting for the cache node to send a change start notification message; the cache node sends a change start notification message to the data modification node; if the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node; the cache node updates the local cache according to the change completion message and the change record, and completes the change processing. The device executes the above method. The data change processing method and device provided by the embodiment of the present invention can ensure the data consistency between the local cache and the database, and reduce the data change processing response time.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data change processing method and device. Background Art

[0002] In distributed system architectures, or distributed frameworks, caching technology has become a key means of improving system performance as business scale and data access volume grow. Maintaining consistency between cached data and persistent database data is a key evaluation metric for caching frameworks.

[0003] Traditional local cache frameworks (such as Ehcache and GuavaCache) achieve extremely low-latency responses by directly accessing cached data in memory. This approach tends to achieve strong cache consistency within a single node. However, this single-node nature leads to data silos in distributed environments, making cross-node cache consistency impossible.

[0004] Mainstream centralized caching systems (such as Redis and Memcached) can partially address cache data consistency issues through centralized storage. However, the additional network communication overhead significantly increases system response time. Summary of the Invention

[0005] In response to the problems in the prior art, embodiments of the present invention provide a data change processing method and apparatus, which can at least partially solve the problems in the prior art.

[0006] In one aspect, the present invention provides a data change processing method, which is applied to a distributed framework; the distributed framework includes service nodes corresponding to respective business services, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; the data change processing method includes:

[0007] During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0008] The data modification node is a service node that performs the modification process, and the cache node is a node of the table to which the data to be modified belongs that is cached locally by all service nodes;

[0009] The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modifying node to the data modifying node;

[0010] If the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node;

[0011] The cache node updates the local cache according to the change completion message and the change record, completing the change processing.

[0012] The change records include newly added data records, modified data records, and deleted data records. Accordingly, the cache node performs data processing according to the change records, including:

[0013] For the newly added data record, if it is determined that multiple newly added data records are received with the same primary key, the multiple newly added data records are stored in the same level;

[0014] For the modified data record, convert the cache record in the local cache that meets the modification condition into a change record and put it into a change record list;

[0015] For the deleted data records, cache records in the local cache that meet the deletion conditions are converted into change records and put into a change record list.

[0016] The cache node updates the local cache according to the change completion message and the change record to complete the change processing, including:

[0017] The range of the local cache to be updated is calculated according to the change record, and data is read from the database node according to the range and updated to the local cache.

[0018] The data change processing method further includes:

[0019] Calculate the timed-out target change record based on the time when the change record was added to the change record list;

[0020] If the change record check process for all target change records is completed, the change record check is completed.

[0021] The data change processing method further includes:

[0022] Each time a target change record check is completed, the local cache is updated according to the target change record that has been checked to complete the change processing.

[0023] The data change processing method further includes:

[0024] If, during the data query process, it is determined that the query condition exists in the change record in the local cache, querying the database row version;

[0025] If it is determined that the database row version is inconsistent with the row version in the local cache, the database is queried in a KV manner, and the local cache is updated according to the query result.

[0026] In one aspect, the present invention provides a data change processing device, which is applied to a distributed framework; the distributed framework includes service nodes corresponding to respective business services, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; the data change processing device includes:

[0027] a publishing unit, configured to publish a change record to the cache node by modifying the data node during the data modification process, update the database storage table in the database node, and wait for the cache node to send a change start notification message;

[0028] The data modification node is a service node that performs the modification process, and the cache node is a node of the table to which the data to be modified belongs that is cached locally by all service nodes;

[0029] a cache unit, configured to process data according to the change record through the cache node, cache the change record after data processing, and send a change start notification message corresponding to each other service node except the data modifying node to the data modifying node;

[0030] a committing unit, configured to commit the data modification transaction and publish a change completion message to the cache node upon receiving change start notification messages corresponding to all other service nodes respectively through the data modification node;

[0031] An updating unit is configured to update a local cache through the cache node according to the change completion message and the change record, thereby completing the change processing.

[0032] In another aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following method is implemented:

[0033] During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0034] The data modification node is a service node that performs the modification process, and the cache node is a node of the table to which the data to be modified belongs that is cached locally by all service nodes;

[0035] The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modifying node to the data modifying node;

[0036] If the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node;

[0037] The cache node updates the local cache according to the change completion message and the change record, completing the change processing.

[0038] An embodiment of the present invention provides a computer-readable storage medium, including:

[0039] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following method is implemented:

[0040] During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0041] The data modification node is a service node that performs the modification process, and the cache node is a node of the table to which the data to be modified belongs that is cached locally by all service nodes;

[0042] The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modifying node to the data modifying node;

[0043] If the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node;

[0044] The cache node updates the local cache according to the change completion message and the change record, completing the change processing.

[0045] An embodiment of the present invention further provides a computer program product, comprising a computer program. When the computer program is executed by a processor, the computer program implements the following method:

[0046] During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0047] The data modification node is a service node that performs the modification process, and the cache node is a node of the table to which the data to be modified belongs that is cached locally by all service nodes;

[0048] The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modifying node to the data modifying node;

[0049] If the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node;

[0050] The cache node updates the local cache according to the change completion message and the change record, completing the change processing.

[0051] The data change processing method and device provided by the embodiment of the present invention are as follows: during the data modification process, the modified data node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message; wherein, the modified data node is a service node that executes the modification processing process, and the cache node is a node of the table to which the data to be modified belongs that is locally cached by all service nodes; the cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the modified data node to the modified data node; if the modified data node receives the change start notification message corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node; the cache node updates the local cache according to the change completion message and the change record, completes the change processing, can ensure the data consistency between the local cache and the database, and reduce the data change processing response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0053] Figure 1 It is a flowchart of a data change processing method provided by an embodiment of the present invention.

[0054] Figure 2 It is a structural diagram of the distributed framework provided by an embodiment of the present invention.

[0055] Figure 3 It is a flowchart of a data change processing method provided by another embodiment of the present invention.

[0056] Figure 4 It is a flowchart of a data change processing method provided by another embodiment of the present invention.

[0057] Figure 5 It is a flowchart of a data change processing method provided by another embodiment of the present invention.

[0058] Figure 6 It is a flowchart of a data change processing method provided by another embodiment of the present invention.

[0059] Figure 7 It is a structural diagram of a data change processing device provided by an embodiment of the present invention.

[0060] Figure 8 A schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other in any manner.

[0062] Explanation of related terms:

[0063] Distributed cache: A caching mechanism that stores data across multiple nodes to improve data access performance and availability.

[0064] Strong consistency algorithm: A computing method used to ensure that in a distributed environment, no matter which node the data is accessed from, the latest and consistent results can be obtained.

[0065] Row version: records the time of data. Whenever data is updated, the time of the recorded data is updated, and the time of the updated recorded data is guaranteed to be globally unique.

[0066] Figure 1 FIG. 1 is a flow chart of a data change processing method provided by an embodiment of the present invention. Figure 1As shown, the data change processing method provided in the embodiment of the present invention is applied to a distributed framework; the distributed framework includes service nodes corresponding to respective business services, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; the data change processing method includes:

[0067] Step S1: During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0068] The data modification node is a service node that performs a modification process, and the cache node is a node of a table to which the data to be modified belongs that is cached locally by all service nodes.

[0069] Step S2: the cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modifying node to the data modifying node.

[0070] Step S3: If the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node.

[0071] Step S4: the cache node updates the local cache according to the change completion message and the change record to complete the change processing.

[0072] In the above step S1, the device modifies the data node to publish the change record to the cache node during the data modification process, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0073] The data modification node is a service node that performs the modification process, and the cache node is a node that locally caches all service nodes for the table to which the data to be modified belongs. The apparatus can be a computer device that performs the method. The acquisition, storage, use, and processing of data in the technical solution of this application comply with relevant regulations.

[0074] In the above step S2, the cache node of the device processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node.

[0075] In the above step S3, if the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node.

[0076] In the above step S4, the cache node of the device updates the local cache according to the change completion message and the change record to complete the change processing.

[0077] like Figure 2 As shown, the distributed framework is described as follows:

[0078] The distributed framework includes gateway services, business services, centralized cache services, and database services, including:

[0079] The gateway service is responsible for request distribution, does not cache data, and does not need to worry about data consistency.

[0080] The business service is responsible for executing specific business processes, and locally caches a list of service nodes and business-related data that needs to be cached, as well as corresponding change records. Each business service can handle different businesses, so the data cached in the service nodes can have both the same part and different parts, and the minimum granularity of cached data is a table. Taking business 1 service as an example, the business service can include multiple service nodes executed in parallel, and the local cache node list in each service node is a node list of all service nodes included in the business service; the local cache table 1 in each service node records the data that needs to be cached related to the business service; the local cache table 1 change record in each service node can record the change record related to the business service in the form of a change record list. The method of the present invention is explained by taking the example of each business service including one service node.

[0081] The centralized cache service is responsible for storing all service node information, and divides the service node information into three parts: centralized cache cluster version, centralized cache node list, and centralized cache message list.

[0082] Among them, the centralized cache cluster version is used to mark the changes in nodes in the distributed framework (specifically, service nodes added and service nodes deleted in the centralized cache cluster). The centralized cache node list records all service nodes and cache-related information, specifically the sum of local cache tables of multiple business services such as Business 1 Service and Business 2 Service; the centralized cache message list records the historical messages generated by all service nodes, including change record historical messages. The messages in the centralized cache message list can be processed in the order of first-in-last-out added to the centralized cache message list and through preset trigger conditions. Therefore, the centralized cache message list records the historical messages recently generated by all service nodes.

[0083] The database service is responsible for persisting data generated during business processing and achieving strong consistency in data storage by controlling database transactions. It also provides an interface for querying the latest strongly consistent persistent data.

[0084] This section describes the process of maintaining strong consistency with the database when a service node accesses the local cache, including four main processes: modifying data, querying data, node joining, and node leaving.

[0085] The data modification process is divided into two parts: the first part is the processing of modifying the data node, and the second part is the processing of the cache node. The two parts are executed in parallel. The specific process is as follows:

[0086] Strong consistency of data updates during the data modification process is guaranteed by the database. During the modification process, a change record is generated for the modified data, and two asynchronous notifications are used to notify each node of the data change as much as possible. If other nodes do not receive feedback on the change record for an extended period of time, strong consistency is achieved by rolling back the data update transaction.

[0087] like Figure 3 As shown in the figure, the process of modifying data nodes, steps 1->3->4->6->8, is a common data modification process. In this process, steps 2, 5, and 7 are inserted:

[0088] Step 2 analyzes the data to be updated (added, deleted, or modified) in step 4, generates a change record, and notifies all cached data nodes of the data being updated. This notifies other nodes of the data being updated. To reduce transaction time, step 2 is typically performed before step 3. In practice, step 2 can be performed after step 3 and before step 4 without affecting the correctness of the method.

[0089] Step 5 waits for all nodes to acknowledge receipt of the change start message, indicating that other nodes know which data the current node will update. Only after all nodes have acknowledged this can the current node safely commit the transaction. Therefore, it is executed after step 4 and before step 6.

[0090] Step 7 notifies all cached data nodes that the previous data update has been completed. After the transaction that modified the data is committed, the current node notifies other nodes that the data modification is complete, giving other nodes a chance to process the updated data.

[0091] The cache node is responsible for processing the two data change messages sent by the modified data node, extracting the change records in the message and synchronizing them to the local change record list, and triggering the change completion processing when the preset trigger conditions are met.

[0092] Change processing begins: First, extract the change records in the message and add them to the local change record list. In addition, additional processing is required for the added, deleted, and modified change records:

[0093] (1) For newly added data records, since the generation of change records precedes the update of the database, multiple newly added records may be received for the same primary key. These records need to be stored at the same level to ensure that the later data records do not overwrite the earlier data records.

[0094] (2) For modified data records, the cache records in the local cache that meet the modification conditions need to be converted into change records and placed in the change record list.

[0095] (3) For deleting data records, the cache records in the local cache that meet the deletion conditions need to be converted into change records and placed in the change record list.

[0096] (4) When modifying and deleting data, it is necessary to convert it from the local cache. The data is directly converted and stored in the change record list for the first time, and the reference count can be directly increased subsequently to achieve the above-mentioned purpose of storing data at the same level.

[0097] Finally, the cache node notifies the data modification node that it has received the change start message.

[0098] Change completion processing: First, calculate the range of the local cache to be updated based on the change record. Then, read the data from the database node based on the range and update it to the local cache. The range is calculated as follows:

[0099] For newly added records, the scope is the primary key of the record.

[0100] For modified and deleted records, the scope is the primary key of the record with a reference count of 1 generated by the local cache at the start of the change processing.

[0101] Finally, delete the currently completed change record and the associated record with a reference count of 1.

[0102] Change record check process: Change record check can be triggered periodically. First, the target change record that has timed out is calculated based on the time when the change record was added to the local change record list. Then, different policies are used to determine whether to trigger the change record check process.

[0103] The present invention provides two strategies:

[0104] (1) Direct triggering, that is, triggering at a regular time.

[0105] (2) The decision is made by querying the modified data node. There are three situations in which this is triggered:

[0106] a. When the data is modified, the node is removed from the node list.

[0107] b. Repeated inquiries to modify data nodes did not receive any feedback.

[0108] c. Modify the data node feedback change record has been completed.

[0109] Finally, determine whether the check is completed. If so, exit the processing process; if not, reprocess the timed-out change record.

[0110] For querying data:

[0111] When querying data, the decision of whether to use the local cache or the database is made by judging whether the query conditions hit the local change record, where the change record is generated during the data modification process. Figure 4 As shown:

[0112] During the query process, if the change record in the local cache snapshot doesn't match the query criteria, it means that no other nodes have modified the data being queried at that moment, and therefore the cached data being queried is strongly consistent. Even if a modification process can be started and successfully committed immediately after the local cache change record is determined to match, it will not affect the result because it is performed after the query process begins.

[0113] If the change record in the local cache snapshot matches the query condition, it means that other nodes are modifying the data to be queried. The transaction commit time is uncertain for the current node. Therefore, the query is transferred to the database, which ensures strong consistency. The specific steps are as follows:

[0114] If, during the data query process, it is determined that the query condition exists in the change record in the local cache, querying the database row version;

[0115] If it is determined that the database row version is inconsistent with the row version in the local cache, the database is queried in a KV manner, and the local cache is updated according to the query result.

[0116] If it is determined that the database row version is consistent with the row version in the local cache, a query is performed based on the cache snapshot.

[0117] The query process 1->2->3->4 is a common cache hit process and will not be described in detail here. When the query condition hits the change record, the 5-step process will begin.

[0118] Step 5 reduces database pressure by avoiding unnecessary duplicate data accesses through row version checks during database queries. By using query conditions to filter out the currently modified data in the local cache snapshot, only this small amount of data can be processed in step 6, avoiding the possibility of cache avalanche caused by large-scale cache penetration.

[0119] In step 6, based on the results of filtering the local cache snapshot in step 5, the primary key of the data being modified is retrieved from the change log. This is then used to query the database using a key-value (KV) format. This allows for high-performance data queries by directly leveraging the database primary key index. After the database query in step 6 is complete, the resulting data is asynchronously processed in step 7.

[0120] In the process of updating the local cache in step 7, the local cache update can be completed quickly in a lock-free manner using CAS, so that subsequent cache queries can determine that the database version is consistent with the local cache version after processing in step 5, thus avoiding excessive processing in step 6.

[0121] The data change processing method further includes:

[0122] Adding the node outside the cluster to the local cache cluster to obtain the service node includes:

[0123] The node outside the cluster broadcasts the registration information of the node outside the cluster to the nodes inside the cluster;

[0124] The node within the cluster receives the registration information, records the registration information into the local cache, and sends a reply message for the registration information to the node outside the cluster;

[0125] If the node outside the cluster determines that it has received reply messages from all nodes within the cluster within a preset time period, it updates the local cache according to the node outside the cluster information and adds the node outside the cluster to the local cache cluster to obtain the service node.

[0126] That is, the node joining process:

[0127] Before building a local cache, a service node must join a local cache cluster (also known as a centralized cache cluster). Cache nodes use a masterless mechanism, updating cluster node information equally. When all existing nodes in the cluster have received the new node registration message, the new node is considered to have successfully joined the cluster. To ensure node information is not lost, cache node information is stored using the centralized cache service. Once a node joins the local cache cluster, it can provide cache services internally.

[0128] like Figure 5As shown in the figure, when a new node joins a cluster, if only some of the cluster's nodes respond to the new node's registration message, these nodes will already be aware of the new node as a cluster member. Therefore, during data modifications, the new node will be notified of the change. Even if the new node has not yet joined the cluster, it will still respond with a notification, thus preventing these nodes from modifying their data. Cluster nodes that do not receive the registration message will not notify the new node of the change, but because the new node has not yet completed registration, this does not affect the cache correctness of the entire system.

[0129] Steps 1->2->3->4->5->6 are the process of joining an external node to the cluster. Steps 1 and 6 are the start and end nodes and are not described here.

[0130] Step 2 mainly writes the information of nodes outside the cluster into the centralized cache service. Before writing, it reads the existing cluster node information and writes its own node information in a locked manner.

[0131] Step 3 broadcasts the registration information to all nodes in the cluster. Once the broadcast is complete, step 4 can be performed.

[0132] Step 4 is to wait for responses from all nodes in the cluster. If no responses are received from all nodes in the cluster for a long time, the attempt to join the cluster is considered to have failed and you need to go back to step 2. If successful, the process starts in step 5.

[0133] In step 5, you've successfully joined the cluster, so the local cache is now enabled. You can now perform a cache warmup to load the data you want to cache from the database into memory. Before warming the cache, synchronize the change log from the centralized cache service to the local cache. This ensures that the locally cached data is either up-to-date or marked as being modified by the change log.

[0134] Steps 7->8->9 are the processing steps for each node in the cluster. After receiving the message in step 7, the new node registration information is obtained.

[0135] Step 8: Record the registration information into the local cache.

[0136] Step 9 notifies the new node of the registration information. After this process is completed, the nodes in the cluster will already treat the new node as a node in the cluster.

[0137] The data change processing method further includes:

[0138] Deleting a node in the cluster from the local cache cluster includes:

[0139] The node in the cluster periodically updates the timestamp of the registration information of the node in the cluster in the centralized cache service. If it is determined that the update fails, the state of the node in the cluster is determined to be unavailable, and a notification message carrying the unavailable state is sent to other nodes in the local cache cluster;

[0140] One of the other nodes in the local cache cluster deletes the cluster node that is in an unavailable state from the local cache cluster.

[0141] The process of a node leaving the cluster can be divided into two parts:

[0142] The first part is when the current node detects that its node information has expired and proactively notifies its departure from the cluster. The second part is when other nodes detect that the current node information has expired and delete the node registration information. After a node leaves the cluster, it will no longer provide cache services.

[0143] like Figure 6 As shown, the first part above is explained as follows:

[0144] It includes the node keepalive process 1->2->3->4->5 and the message processing process 6->7.

[0145] In step 2, the timestamp of the node's registration information is updated in the centralized cache service. If the update is successful, the node will wait for a while and then repeat step 2. If the update fails, the node is considered invalid and the process of step 3 is carried out.

[0146] Step 3 switches the local node status to unavailable, and no cache service is provided at this time.

[0147] Step 4 proactively notifies the cluster that other nodes have left the cluster, so that the information that the current node has failed can be synchronized to the cluster in a timely manner, preventing other nodes from having to wait too long before submitting data modification transactions.

[0148] Step 6: After receiving the message, extract the information of the node leaving the cluster.

[0149] Step 7: Delete the corresponding node in the local cache.

[0150] The data change processing method further includes:

[0151] Deleting a node in the cluster from the local cache cluster includes:

[0152] The nodes in the cluster periodically query the active nodes in the centralized cache service. If it is determined that the time difference between the current query time of the active node and the last query time of the active node is less than a preset duration threshold, the registered nodes in the local cache are compared with the active nodes, and the active nodes different from the registered nodes in the local cache are regarded as inactive nodes, and the inactive nodes are deleted from the centralized cache service and the local cache cluster.

[0153] like Figure 6 As shown, the second part is explained as follows:

[0154] This includes the node keepalive detection process from 8->9->10->11->7.

[0155] Among them, step 9 is to query the registered nodes in the centralized cache service, record the timestamp of the node as the current node activity time, and compare it with the last node activity time. If the time difference is less than the threshold, it is marked as an active node.

[0156] If the current node is not an active node, exit the keepalive detection process.

[0157] If the current node is an active node, proceed to the next step.

[0158] Step 10 compares the registered nodes in the local cache with the active nodes and filters out the inactive nodes. Step 11 deletes the inactive nodes from the centralized cache service and simultaneously calls step 7 to delete the nodes in the local cache as well.

[0159] The data change processing method provided by the embodiment of the present invention has the following beneficial technical effects:

[0160] 1. Eliminate data update delays, ensuring that data read at any time and on any node is up-to-date and consistent, overcoming the consistency deviation caused by data synchronization delays in existing technologies.

[0161] 2. Solve the single point failure problem, avoid the interruption of data writing and consistency maintenance of the entire system due to the failure of a single node, and improve the reliability and stability of the system.

[0162] 3. Effectively handle conflicts in high-concurrency write operations, prevent data overwriting, loss, or incorrect updates, and ensure strong data consistency in high-concurrency scenarios.

[0163] 4. Enhanced adaptability to node errors. In the event of a node error, the system can still maintain a consistent state, and the node consistency state can be quickly and accurately restored after the error node recovers.

[0164] 5. Reduce system complexity and development and maintenance costs, and provide a relatively simple, easy-to-understand and implement consistency algorithm without sacrificing consistency assurance capabilities.

[0165] 6. Optimize the system’s performance and scalability so that the algorithm can maintain efficient operation and good consistency maintenance capabilities under large-scale data and a growing number of nodes.

[0166] The data change processing method provided by an embodiment of the present invention comprises the following steps: during the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message; wherein, the data modification node is a service node that executes the modification processing process, and the cache node is a node of the table to which the data to be modified belongs that is locally cached by all service nodes; the cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node; if the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node; the cache node updates the local cache according to the change completion message and the change record, completes the change processing, can ensure the data consistency between the local cache and the database, and reduces the data change processing response time.

[0167] Furthermore, the change record includes newly added data records, modified data records, and deleted data records; accordingly, the cache node performs data processing according to the change record, including:

[0168] For the newly added data record, if it is determined that multiple newly added data records are received with the same primary key, the multiple newly added data records are stored at the same level; please refer to the above embodiment for description and will not be repeated here.

[0169] For the modified data record, the cache records in the local cache that meet the modification conditions are converted into change records and put into the change record list; the above embodiment can be referred to for description and will not be repeated here.

[0170] For the deleted data records, the cache records in the local cache that meet the deletion conditions are converted into change records and put into the change record list.

[0171] Furthermore, the cache node updates the local cache according to the change completion message and the change record to complete the change processing, including:

[0172] The range of the local cache to be updated is calculated based on the change record, and data is read from the database node based on the range and updated to the local cache.

[0173] Furthermore, the data change processing method further includes:

[0174] The target change record that has timed out is calculated based on the time when the change record is added to the change record list; the description can be made with reference to the above embodiment and will not be repeated here.

[0175] If the change record checking process of all target change records is completed, the change record checking is completed.

[0176] Furthermore, the data change processing method further includes:

[0177] Each time a target change record check is completed, the local cache is updated according to the target change record that has been checked, and the change processing is completed.

[0178] Furthermore, the data change processing method further includes:

[0179] During the data query process, if it is determined that the query condition exists in the change record in the local cache, the database row version is queried; reference may be made to the above embodiment for explanation, which will not be repeated here.

[0180] If it is determined that the database row version is inconsistent with the row version in the local cache, the database is queried in a KV manner, and the local cache is updated according to the query result.

[0181] Figure 7 FIG. 1 is a structural diagram of a data change processing device provided by an embodiment of the present invention. Figure 7 As shown, the data change processing device provided by the embodiment of the present invention is applied to a distributed framework; the distributed framework includes service nodes corresponding to each business service, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; the data change processing device includes a publishing unit 701, a cache unit 702, a commit unit 703, and an update unit 704, wherein:

[0182] The publishing unit 701 is used to publish the change record to the cache node by modifying the data node during the data modification process, update the database storage table in the database node, and wait for the cache node to send a change start notification message; wherein, the data modification node is a service node that performs the modification processing process, and the cache node is a node of the table to which the data to be modified belongs that is locally cached by all service nodes; the cache unit 702 is used to process data according to the change record through the cache node, cache the change record after data processing, and send a change start notification message corresponding to each other service node except the data modification node to the data modification node; the submission unit 703 is used to submit the data modification transaction and publish a change completion message to the cache node if the change start notification message corresponding to all other service nodes is received through the data modification node; the update unit 704 is used to update the local cache according to the change completion message and the change record through the cache node to complete the change processing.

[0183] Specifically, the publishing unit 701 in the device is used to publish the change record to the cache node by modifying the data node during the data modification process, update the database storage table in the database node, and wait for the cache node to send a change start notification message; wherein, the modified data node is a service node that performs the modification processing process, and the cache node is a node of the table to which the data to be modified belongs that is locally cached by all service nodes; the cache unit 702 is used to process data according to the change record through the cache node, cache the change record after data processing, and send a change start notification message corresponding to each other service node except the modified data node to the modified data node; the submission unit 703 is used to submit the data modification transaction and publish a change completion message to the cache node if the change start notification message corresponding to all other service nodes is received through the modified data node; the update unit 704 is used to update the local cache according to the change completion message and the change record through the cache node to complete the change processing.

[0184] The data change processing device provided by an embodiment of the present invention is configured such that during the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message; wherein the data modification node is a service node that executes the modification processing process, and the cache node is a node of a table to which the data to be modified belongs that is locally cached by all service nodes; the cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node; if the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node; the cache node updates the local cache according to the change completion message and the change record to complete the change processing, thereby ensuring data consistency between the local cache and the database and reducing the data change processing response time.

[0185] The data change processing device provided in the embodiment of the present invention can be used to execute the processing flow of the above-mentioned method embodiments. Its functions are not described in detail here, and reference can be made to the detailed description of the above-mentioned method embodiments.

[0186] Figure 8 A schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention is shown in FIG. Figure 8 As shown, the computer device includes: a memory 801, a processor 802, and a computer program stored in the memory 801 and executable on the processor 802. When the processor 802 executes the computer program, the following method is implemented:

[0187] During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0188] The data modification node is a service node that performs the modification process, and the cache node is a node of the table to which the data to be modified belongs that is cached locally by all service nodes;

[0189] The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modifying node to the data modifying node;

[0190] If the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node;

[0191] The cache node updates the local cache according to the change completion message and the change record, completing the change processing.

[0192] This embodiment discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, the following method is implemented:

[0193] During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0194] The data modification node is a service node that performs the modification process, and the cache node is a node of the table to which the data to be modified belongs that is cached locally by all service nodes;

[0195] The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modifying node to the data modifying node;

[0196] If the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node;

[0197] The cache node updates the local cache according to the change completion message and the change record, completing the change processing.

[0198] This embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following method is implemented:

[0199] During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message;

[0200] The data modification node is a service node that performs the modification process, and the cache node is a node of the table to which the data to be modified belongs that is cached locally by all service nodes;

[0201] The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modifying node to the data modifying node;

[0202] If the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node;

[0203] The cache node updates the local cache according to the change completion message and the change record, completing the change processing.

[0204] Compared with the technical solutions in the prior art, the embodiments of the present invention provide a data change processing method, in which, during the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message; wherein, the data modification node is a service node that executes the modification processing process, and the cache node is a node of the table to which the data to be modified belongs that is locally cached by all service nodes; the cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modification node to the data modification node; if the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node; the cache node updates the local cache according to the change completion message and the change record to complete the change processing, thereby ensuring data consistency between the local cache and the database and reducing the data change processing response time.

[0205] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0207] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0208] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0209] Throughout this specification, reference to terms such as "one embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0210] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data change processing method, characterized in that: The data change processing method is applied to a distributed framework; the distributed framework includes service nodes corresponding to each business service, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; The data change processing method includes: During the data modification process, the data modification node publishes a change record to the cache node, updates the database storage table in the database node, and waits for the cache node to send a change start notification message; The data modification node is a service node that performs the modification process, and the cache node is a node of the table to which the data to be modified belongs that is cached locally by all service nodes; The cache node processes data according to the change record, caches the change record after data processing, and sends a change start notification message corresponding to each other service node except the data modifying node to the data modifying node; If the data modification node receives the change start notification messages corresponding to all other service nodes, it commits the data modification transaction and publishes a change completion message to the cache node; The cache node updates the local cache according to the change completion message and the change record to complete the change processing; The change record includes newly added data records, modified data records, and deleted data records. Accordingly, the cache node performs data processing according to the change record, including: For the newly added data record, if it is determined that multiple newly added data records are received with the same primary key, the multiple newly added data records are stored in the same level; For the modified data record, convert the cache record in the local cache that meets the modification condition into a change record and put it into a change record list; For the deleted data records, cache records in the local cache that meet the deletion conditions are converted into change records and put into a change record list.

2. The data change processing method according to claim 1, characterized in that: The cache node updates the local cache according to the change completion message and the change record to complete the change processing, including: The range of the local cache to be updated is calculated according to the change record, and data is read from the database node according to the range and updated to the local cache.

3. The data change processing method according to claim 1, wherein: The data change processing method further includes: Calculate the timed-out target change record based on the time when the change record was added to the change record list; If the change record check process for all target change records is completed, the change record check is completed.

4. The data change processing method according to claim 3, wherein: The data change processing method further includes: Each time a target change record check is completed, the local cache is updated according to the target change record that has been checked to complete the change processing.

5. The data change processing method according to claim 1, wherein: The data change processing method further includes: If, during the data query process, it is determined that the query condition exists in the change record in the local cache, querying the database row version; If it is determined that the database row version is inconsistent with the row version in the local cache, the database is queried in a KV manner, and the local cache is updated according to the query result.

6. A data change processing device, characterized in that: The data change processing device is applied to a distributed framework; the distributed framework includes service nodes corresponding to each business service, each service node is connected to a cache node corresponding to a centralized cache service, and each service node is connected to a database node corresponding to a database service; The data change processing device includes: a publishing unit, configured to publish a change record to the cache node by modifying the data node during the data modification process, update the database storage table in the database node, and wait for the cache node to send a change start notification message; The data modification node is a service node that performs the modification process, and the cache node is a node of the table to which the data to be modified belongs that is cached locally by all service nodes; a cache unit, configured to process data according to the change record through the cache node, cache the change record after data processing, and send a change start notification message corresponding to each other service node except the data modifying node to the data modifying node; a committing unit, configured to commit the data modification transaction and publish a change completion message to the cache node upon receiving change start notification messages corresponding to all other service nodes respectively through the data modification node; An updating unit, configured to update a local cache through the cache node according to the change completion message and the change record, thereby completing the change processing; The change records include newly added data records, modified data records, and deleted data records; accordingly, the cache unit is specifically used to: For the newly added data record, if it is determined that multiple newly added data records are received with the same primary key, the multiple newly added data records are stored in the same level; For the modified data record, convert the cache record in the local cache that meets the modification condition into a change record and put it into a change record list; For the deleted data records, cache records in the local cache that meet the deletion conditions are converted into change records and put into a change record list.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images