Data processing method and device, electronic equipment and storage medium

Abstract

This application discloses a data processing method, apparatus, electronic device, and storage medium, comprising: receiving a data update notification message, which is sent by a target device storing a consensus protocol log when a consensus protocol log update is detected; generating an update data read request based on a historical read position, wherein the historical read position is the end position of the data read in the previous read of the consensus protocol log; sending the update data read request to the target device, wherein the update data read request instructs the target device to perform the operation of reading updated data based on the historical read position; receiving the updated data sent by the target device, and sending the updated data to other devices. This application can detect a data update as soon as the consensus protocol log is updated, and promptly generate and send an update data read request to read the updated data in the consensus protocol log, making the data synchronization process faster and more timely.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Change Data Capture (CDC) is an important capability in database systems. CDC refers to the process by which a database system detects data changes, acquires the changed data, and synchronizes it with other data processing systems.

[0003] Existing change data capture technologies typically achieve this by reading the Write-Ahead Log (WAL) of a single-machine storage engine. Before writing new data, the single-machine storage engine first writes the data to a log file on the persistent storage device; this log file is called the write-ahead log. Current solutions can start a background process on each machine. This background process can periodically read and parse the contents of the write-ahead log, and then send the parsed change data to the downstream data processing system.

[0004] However, periodic data updates cause a certain delay in the process of capturing changed data, resulting in data updates not being implemented in a timely manner. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and storage medium, which can improve the problem of untimely data updates in the prior art.

[0006] This application provides a data processing method, the method comprising: receiving a data update notification message, wherein the data update notification message is sent by a target device storing a consistency protocol log when an update to the consistency protocol log is detected; generating an update data read request based on a historical read position, wherein the historical read position is the end position of the data read in the previous read of the consistency protocol log; and sending the update data read request to the target device, the update data read request being used to instruct the target device to perform an operation of reading update data based on the historical read position.

[0007] This application embodiment also provides a data processing method, the method comprising: when a consensus protocol log update is detected, sending a data update prompt message to an update device; receiving an update data read request sent by the update device, wherein the update data read request includes a historical read position, the historical read position being the end position of the data read in the previous reading of the consensus protocol log; reading update data from the consensus protocol log based on the historical read position; and sending the update data to the update device.

[0008] This application provides another data processing method applied to a routing center, which is communicatively connected to multiple devices, including an update device.

[0009] The method includes: if no heartbeat information is received from the updating device within the heartbeat time period, determining a new updating device from among multiple devices that normally send heartbeat information; sending an update priority identifier to the new updating device, the update priority identifier being used to instruct the new updating device to send the update data along with the update priority identifier to other devices when it receives update data. This application embodiment provides a data processing apparatus, the apparatus including:

[0010] An information receiving unit is used to receive data update notification information, wherein the data update notification information is sent by a target device storing a consistency protocol log when it detects an update to the consistency protocol log;

[0011] The request generation unit is used to generate an update data read request based on the historical read position, wherein the historical read position is the end position of the data read when the consistency protocol log was read last time;

[0012] A request sending unit is configured to send the update data read request to the target device, wherein the update data read request is used to instruct the target device to perform an operation of reading update data based on the historical read position;

[0013] The update synchronization unit is used to receive update data sent by the target device and send the update data to other devices.

[0014] In one embodiment, the device further includes:

[0015] A timed request generation unit is used to generate a timed update data read request based on the historical read position when no data update prompt information is received within a set time period. The timed update data read request is an update data read request generated when no data update prompt information is received within the set time period.

[0016] A timed request sending unit is used to send the update data timed read request to the target device, the update data read request being used to instruct the target device to perform the operation of reading update data based on the historical read position.

[0017] In one embodiment, the device further includes:

[0018] An information feedback unit is configured to receive non-updated feedback information sent by the target device, wherein the non-updated feedback information is sent by the target device when no updated data exists after detecting the historical reading position; or to receive updated data sent by the target device, wherein the updated data is sent by the target device when updated data exists after detecting the historical reading position.

[0019] In one embodiment, the device further includes:

[0020] The periodic read unit is used to periodically read the write-ahead log from the persistent storage device;

[0021] The log parsing unit is used to parse the read write-ahead log to obtain the changed data;

[0022] Correspondingly, the update synchronization unit includes:

[0023] An update receiving subunit is used to receive update data sent by the target device;

[0024] A data comparison subunit is used to compare the updated data with the changed data;

[0025] The data sending subunit is used to send the updated data to other devices when the updated data is consistent with the changed data, so as to realize the synchronization of the updated data between devices.

[0026] In one implementation, the update data sent by the target device is unparsed update data;

[0027] Correspondingly, the update receiving subunit is specifically used to: receive update data sent by the target device, and parse the update data to obtain parsed update data;

[0028] A data comparison subunit is specifically used to compare the parsed updated data with the changed data. This application also provides a data processing apparatus, the apparatus comprising:

[0029] The information sending unit is used to send a data update prompt message to the updating device when an update to the consistency protocol log is detected;

[0030] The request receiving unit is used to receive the update data read request sent by the update device, wherein the update data read request includes a historical read position, which is the end position of the data read when the consistency protocol log was read last time;

[0031] A data reading unit is used to read updated data from the consistency protocol log based on the historical read position;

[0032] A data sending unit is used to send update data to the update device.

[0033] In one embodiment, the device further includes:

[0034] A timed request receiving unit is used to receive a timed update data read request, wherein the timed update data read request includes the historical read position; the timed update data read request is an update data read request generated by the update device when it has not received the data update prompt information within a set time period.

[0035] The feedback sending unit is used to generate no-update feedback information when the consistency protocol log has no updated data after the historical read position, and send the no-update feedback information to the updating device;

[0036] The update sending unit is used to send the update data to the update device when the consistency protocol log has updated data after the historical read position.

[0037] In one embodiment, the device further includes:

[0038] The device determination unit is used to determine a new update device from multiple devices that normally send heartbeat information when no heartbeat information is received from the update device within the heartbeat time period.

[0039] The priority identifier sending unit is used to send an update priority identifier to the new update device. The update priority identifier is used to instruct the new update device to send the update data along with the update priority identifier to other devices when it receives update data.

[0040] This application embodiment also provides a data processing apparatus applied to a routing center, the routing center being communicatively connected to multiple devices, the multiple devices including an update device; the apparatus includes:

[0041] An update device determination unit is used to determine a new update device from multiple devices that normally send heartbeat information if no heartbeat information is received from the update device within the heartbeat time period.

[0042] The identifier sending unit is used to send an update priority identifier to the new update device. The update priority identifier is used to instruct the new update device to send the update data along with the update priority identifier to other devices when it receives update data.

[0043] This application also provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the data processing methods provided in this application.

[0044] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the data processing methods provided in this application.

[0045] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in any of the data processing methods provided in this application.

[0046] In the data processing method provided in this application embodiment, a data update notification message is received, which is sent by a target device storing a consistency protocol log when an update to the consistency protocol log is detected. An update data read request is generated based on the historical read position, where the historical read position is the end position of the data read during the previous read of the consistency protocol log. The update data read request is sent to the target device so that the target device can read the updated data according to the historical read position. The updated data sent by the target device is received, and the updated data is sent to other devices to achieve synchronization of the updated data among the devices.

[0047] In this embodiment, when the target device storing the consensus protocol log detects an update to the consensus protocol log, it sends a data update notification to the updating device executing the data processing method of this application. Upon receiving the data update notification, the updating device generates an update data read request based on the historical read position and sends the update data read request to the target device to obtain data from the target device after the historical read position (i.e., data updated after the previous read of the consensus protocol log) and synchronize it with other devices. This embodiment allows the device to detect a data update as soon as the consensus protocol log is updated, promptly generating and sending an update data read request to read the updated data in the consensus protocol log, making the data synchronization process faster and more timely. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1a This is a schematic diagram illustrating an application scenario of the data processing method provided in this application;

[0050] Figure 1b This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0051] Figure 1c A schematic flowchart illustrating the process of the updating device obtaining update data from the target device is shown.

[0052] Figure 2a This is a schematic flowchart of a data processing method provided in another embodiment of this application;

[0053] Figure 2b A schematic structural block diagram of the equipment being changed and updated is shown;

[0054] Figure 3 This is a flowchart illustrating a data processing method provided in a specific embodiment of this application;

[0055] Figure 4 This is a schematic diagram of a data processing apparatus provided in one embodiment of this application;

[0056] Figure 5 This is a schematic diagram of a data processing apparatus provided in another embodiment of this application;

[0057] Figure 6 This is a schematic diagram of a data processing apparatus provided in another embodiment of this application;

[0058] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] This application provides a data processing method, apparatus, electronic device, and storage medium.

[0061] Specifically, the data processing device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, desktop computer, etc. The server can be a single server, a server cluster consisting of multiple servers, or a cloud server, etc.

[0062] In some embodiments, the data processing apparatus may also be integrated into multiple electronic devices, such as multiple servers, with the data processing method of this application being implemented by the multiple servers.

[0063] In some embodiments, the terminal can also be used as a server to implement some or all of the functions of a server.

[0064] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0065] Please see details Figure 1a The method provided in this application embodiment may include: receiving a data update notification message, wherein the data update notification message is sent by a target device storing a consistency protocol log when it detects an update to the consistency protocol log; generating an update data read request based on a historical read position, wherein the historical read position is the end position of the data read in the previous read of the consistency protocol log; sending the update data read request to the target device, the update data read request being used to instruct the target device to perform an operation of reading update data based on the historical read position; receiving the update data sent by the target device, and sending the update data to other devices.

[0066] The above method allows the system to detect data updates as soon as the consistency protocol log is updated, and promptly generate and send update data read requests to read the updated data from the consistency protocol log, making the data synchronization process faster and more timely.

[0067] The data processing method provided in this application can be applied to distributed database systems. A distributed database system is a logically unified database formed by connecting multiple physically dispersed database system units. A distributed database system typically includes multiple physical machine nodes, which are responsible for storing data and running the management software of the distributed database system.

[0068] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0069] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0070] In this embodiment, a data processing method is provided. For example... Figure 1b As shown, the specific process of this method may include the following steps 110 to 140:

[0071] 110. Receive data update notification information, wherein the data update notification information is sent by the target device storing the consistency protocol log when it detects an update to the consistency protocol log.

[0072] A consistency protocol is an algorithm used in distributed database systems to ensure data consistency among multiple data replicas. Consistency protocols coordinate data replicas distributed across different nodes, ensuring they are updated and synchronized in a controlled manner, thereby avoiding data inconsistency problems.

[0073] Alternatively, the consensus protocol can be implemented in any of the following ways:

[0074] In one specific implementation, the implementation process of the consensus protocol may include:

[0075] The proposer selects a unique number N and sends a "prepare(N)" request to a majority of acceptors. The number N is unique and increments throughout the system.

[0076] If the number N in the Prepare request received by the recipient is greater than the number in all previously received Prepare requests, then the recipient will no longer accept any proposals with numbers less than N. Specifically, after receiving a proposal, the recipient first obtains the number of the received proposal and compares the proposal number with the value N. If the proposal number is less than the value N, the proposal is ignored.

[0077] After receiving a Prepare request with number N, the recipient can send feedback to the proposer indicating that it has received the Prepare request with number N.

[0078] If the proposer receives positive responses to the Prepare request from a majority of the receivers, it can send an "accept" request (Accept(N,V)) to those receivers. Here, V represents the data to be synchronized.

[0079] Upon receiving an Accept request, the receiver compares the number N in the request with its own stored number. If the number N is not less than its own stored number, the receiver accepts the data V to be synchronized and records it. If the number N is less than its own stored number, the receiver ignores the Accept request.

[0080] Once the data V to be synchronized is accepted by a majority of receivers, this value is considered finalized. Learners can find out the accepted value by asking the receivers, or the receivers can proactively notify the learners.

[0081] The proposer is the logical entity that initiates the proposal. It can be a client request handler, a dedicated service module, or any program capable of generating data items that need to be agreed upon.

[0082] An acceptor is a logical entity that receives and may accept proposals. It is typically part of a storage system and is responsible for persisting proposal information.

[0083] A learner is a logical entity that obtains information about accepted proposals from the recipients. It can be the system's state machine, another service, or other component that needs to know the eventual consistent value.

[0084] Learners, acceptors, and proposers can run on the same physical machine or be distributed across different servers or computing nodes.

[0085] In another implementation, the specific implementation process of the consensus protocol may include:

[0086] When a follower does not receive any messages for a period of time, the follower will be converted to a candidate state and a new round of election will begin.

[0087] The candidate first increments its own term number and then sends a vote request (RequestVote RPC) to other nodes in the cluster.

[0088] A node that receives a voting request can decide whether to vote for the candidate based on preset rules (e.g., whether the candidate's term number is greater than or equal to the current node's term number).

[0089] If a candidate receives a majority of votes from the nodes, it can become the new leader. If multiple candidates run for office simultaneously, resulting in no single candidate receiving a majority of votes, the process will wait until the next election cycle to try again.

[0090] Once a leader is elected, it begins receiving client requests, adding these requests as log entries to its own log, and then synchronizing them with all followers. Each log entry contains a command (specified by the client), the term number that created the entry, and an index position.

[0091] The leader synchronizes log entries by sending AppendEntries RPC requests to followers. Upon receiving an AppendEntries RPC request, a follower first checks its own log for entries with the same index and term number. If an entry exists, it means the log entries before that position are matched. If no entry exists, or if an entry exists but the index and term number do not match, it indicates a discrepancy between the follower's and leader's logs. In cases of discrepancies, the leader can identify the point of divergence and truncate its log at that point, removing subsequent entries that differ from the leader's. Next, the follower updates its own log based on the leader's log, filling in missing entries, until the follower's and leader's logs are fully synchronized. The AppendEntries RPC request is used to achieve log synchronization between the leader and followers.

[0092] Let's illustrate this with an example. Suppose the leader has a log entry sequence [Term 2, Index 5] -> [Term 3, Index 6] -> [Term 3, Index 7], while the follower has [Term 2, Index 5] -> [Term 2, Index 6] -> [Term 2, Index 7]. In this scenario: When the leader sends the AppendEntries RPC, it will indicate in the request that its last log entry is [Term 3, Index 7]. The follower checks its own log and finds that while the log entry at index 5 (i.e., Index 5) matches, the log entries starting from index 6 (i.e., Index 6) do not match (the follower's term number is lower). Therefore, the follower can delete all log entries from index 6 onwards and request the leader to provide the correct log entries [Term 3, Index 6] and [Term 3, Index 7]. After receiving these log entries, the follower adds them to its own log, thus reaching agreement with the leader.

[0093] The leader is the core node responsible for managing the entire cluster operation. It handles all requests from clients and adds these requests as log entries to its own log.

[0094] Followers are passive roles; they do not initiate requests but respond to requests from leaders or candidates.

[0095] A candidate is a node that attempts to become the new leader through election when there is no effective leader.

[0096] Leader, Follower, and Candidate are not specific physical devices, but rather nodes that play different roles or states in a distributed system. Leader, Follower, and Candidate can be implemented by different processes or threads on the same physical server, or they can be distributed across different physical servers, depending on the system's architecture and requirements.

[0097] In another implementation, the specific implementation process of the consensus protocol may include:

[0098] The coordinator initiates a transaction and sends a "prepare request" to all involved participants, asking them if they are ready to commit the transaction.

[0099] Upon receiving a "ready" request, each participant performs the following actions: checks if it has sufficient resources to complete the transaction (e.g., checks disk space, locks, etc.). If it can complete the transaction, it locks the relevant resources, logs the transaction to indicate that it is ready to commit, and returns a "Prepared" message to the coordinator. If it cannot complete the transaction, it returns an error message.

[0100] The coordinator waits for responses from all participants. If all participants return "Ready," the process proceeds to the Commit Phase. If any participant is not ready, the coordinator will decide to abort the transaction.

[0101] Specifically, if all participants have completed their preparations:

[0102] The coordinator sends a "commit" request to all participants, instructing them to formally commit the transaction. Upon receiving the "commit" request, each participant executes the transaction commit operation, releases previously locked resources, and returns a "committed" confirmation to the coordinator. Once the coordinator receives confirmations from all participants, it marks the transaction as successfully completed. The transaction commit operation is used to implement modifications to data based on business needs. For example, in an e-commerce application, this might involve data updates across multiple steps, such as order creation, inventory reduction, and payment processing.

[0103] If at least one participant fails to prepare:

[0104] The coordinator sends an "Abort request" to all participants, instructing them to cancel the transaction. Upon receiving the "Abort request," each participant reverts its changes, unlocks resources, and returns an "Aborted" confirmation to the coordinator. Once the coordinator receives confirmations from all participants, it marks the transaction as failed.

[0105] The coordinator is a separate service or process responsible for managing the entire distributed transaction process. It does not directly execute the operations within the transaction, but rather ensures that all participants consistently decide whether to commit or rollback the transaction through interaction with the participants. Participants are the nodes that actually execute the transaction operations, such as database servers or other storage services. Each participant is responsible for managing its local resources and performing corresponding operations according to the coordinator's instructions. The coordinator and participants can run as different processes or threads on the same physical machine, or they can be distributed across multiple different physical devices, depending on the system architecture and requirements.

[0106] It should be understood that the specific implementation of the consensus protocol should not be construed as a limitation on this application.

[0107] Consistency protocols can maintain a logical log, known as the consistency protocol log. When data is updated, the updated data is written to the consistency protocol log. Specifically, the consistency protocol log is a special type of log maintained by the consistency protocol in a distributed database system to ensure consistency among multiple data replicas. This log records all operations that modify the data (such as inserts, updates, or deletions) so that each data replica can apply these operations in the same order, thereby ensuring data consistency. The consistency protocol log can use a uniform log format and is not bound to a specific single-machine storage engine. In contrast, the parsing method of the write-ahead log in existing technologies depends on the single-machine storage engine, and the path and format of the write-ahead log can be different for each single-machine storage engine. Therefore, compared with the existing technology of using write-ahead logs, using the consistency protocol log allows change data capture to only support parsing the format of the consistency protocol log, without having to support multiple write-ahead log formats.

[0108] The target device is the device that stores the consensus protocol log. The consensus protocol log can be stored on one device or multiple devices. If the consensus protocol log is stored on multiple devices, when the consensus protocol log of one device is updated, that device broadcasts the log update data to the other devices storing the consensus protocol log. Optionally, the target device storing the consensus protocol log may be different from, or may be the same as, the update device performing steps 110 to 140. It should be understood that the specific number of devices storing the consensus protocol log and the identity of the devices should not be considered a limitation of this application.

[0109] For ease of description, let's take the example of a target device storing the consensus protocol log and the update device performing steps 110 to 140 being different devices. When the target device detects an update to its stored consensus protocol log, it can send a data update notification to the update device.

[0110] The update device is the device running a Change Data Capture (CDC) replica. During CDC replication, the following processes are performed: reading and parsing the consistency protocol log of the distributed database, capturing updated data, and sending the updated data to other devices. A CDC replica can essentially be viewed as a specific type of database replica or service instance. Anomaly recovery for CDC replicas can employ the same anomaly recovery mechanism as other data replicas, eliminating the need to develop a dedicated anomaly recovery management module for CDC replicas.

[0111] Change data capture technology can be used in various scenarios. Optionally, changed data can be synchronized to a message queue, allowing subscribers to process the data accordingly. For example, in an e-commerce system, when an order status is updated, the database system's change data capture function can send the new order status to a message queue. After receiving the new order status, subscribers can send it to the buyer's device. Data can also be synchronized in real-time from the database system to a data analysis system, enabling data analysts to perform large-scale real-time data queries. Furthermore, data can be synchronized to a backup database, allowing users to switch to the backup database if the primary database fails.

[0112] Data update notifications are messages indicating that a data update has occurred in the consistency protocol log. These notifications remind the updating device that a new set of data changes is ready to be synchronized. Optionally, the data update notification may include basic information such as the timestamp or version number of the change, but importantly, it triggers the updating device to begin a new round of data synchronization.

[0113] 120. Generate an update data read request based on the historical read position, wherein the historical read position is the end position of the data read when the consensus protocol log was read last time.

[0114] After receiving a data update notification, the updating device can obtain the end position of the data read during the previous read of the consistency protocol log (i.e., the historical read position). Optionally, the historical read position is usually stored in a local variable or configuration file for later use.

[0115] Using a defined historical read position, the updating device constructs a specific update data read request. This request explicitly specifies that all subsequent update data should be read starting from that historical read position. This ensures that no intermediate changes are missed and that all new data since the last synchronization is efficiently retrieved. Optionally, in addition to specifying the starting position, the update data read request may include other parameters, such as a maximum number of return entries limit and a timeout period, to optimize performance and adapt to different application scenarios.

[0116] 130. Send the update data read request to the target device, the update data read request being used to instruct the target device to perform the operation of reading update data based on the historical read position.

[0117] The updating device sends the constructed update data read request to the target device responsible for managing the consistency protocol log. Upon receiving the update data read request, the target device locates the corresponding position based on the historical read positions included in the request and begins reading all change records from that position onwards until the termination condition specified in the update data read request is met (such as reaching the maximum number of entries or encountering the end of the log). Then, the target device packages the collected update data into a response message and returns it to the updating device.

[0118] 140. Receive update data sent by the target device and send the update data to other devices.

[0119] The updating device receives the response message sent by the target device, parses the update data from the response message, and then sends the update data to other devices, thereby realizing the synchronization process of update data between devices.

[0120] In the above implementation, when the consistency protocol log stored in the target device is updated, the updating device can be notified that a data update has occurred, and an update data read request can be generated and sent in a timely manner to read the updated data in the consistency protocol log of the target device, which can make the data synchronization process faster and more timely.

[0121] Optionally, in one embodiment, prior to step 140, the present application embodiment may further include the following steps A1 to A2:

[0122] A1. Periodically read the write-ahead log from the persistent storage device.

[0123] The specific duration of the cycle can be set by the developers based on their experience; for example, it could be 10 seconds or 5 seconds. It should be understood that the specific cycle duration should not be construed as a limitation of this application.

[0124] Persistent storage devices are storage media that retain data even after power is turned off or the system restarts. Common persistent storage devices include:

[0125] A hard disk drive (HDD) is a traditional type of storage device that uses a rotating disk and a moving read / write head to store and retrieve data.

[0126] Solid State Drives (SSDs) are storage devices based on flash memory technology that are faster, more durable, and consume less power than HDDs.

[0127] Optical discs (CD, DVD, Blu-ray, etc.) are optical media that use lasers to read and write data.

[0128] In this embodiment, the persistent storage device corresponds to the update device. Therefore, the update device can periodically read the write-ahead log from its corresponding HDD, SSD, or optical disk.

[0129] Optionally, the write-ahead log stored on the persistent storage device can be updated by a standalone storage engine. Before writing new data, the standalone storage engine first writes the data to the write-ahead log of the persistent storage device. The standalone storage engine is responsible for storing and managing the data copies stored on the standalone machine.

[0130] Write-ahead log (WAL) is a technique used to ensure data integrity and durability, widely applied in databases and file systems. Optionally, before data is modified, these modifications are recorded in a log file, typically stored on persistent storage devices such as hard drives or solid-state drives (SSDs). The actual data is only modified after this log information has been successfully written. If a system failure occurs during data writing, incomplete transactions can be recovered by replaying the contents of the log file, thus ensuring data consistency and durability.

[0131] Specifically, when the system recovers from a failure, it checks the last successful checkpoint, which is a snapshot of the system state. Then, the system reads all incomplete transactions from the WAL (Write-Ahead Log) after that checkpoint and redoes committed data changes that may not have been fully written to disk, or undoes uncommitted transaction changes based on these log records.

[0132] A2. Parse the read write-ahead log to obtain the changed data.

[0133] Optionally, the format of the write-ahead log (WAL) can be specified beforehand. The WAL typically contains a series of log records, each representing a specific database operation (such as insert, update, or delete). These log records may contain the following information:

[0134] Transaction ID: A unique number that identifies each transaction.

[0135] Operation type: Indicates the type of database operation corresponding to this record (e.g., INSERT, UPDATE, DELETE, etc.).

[0136] Operation target: Specify the location of the affected data row or object (such as table name, row ID, etc.).

[0137] New and old values: For UPDATE operations, the values ​​before and after the modification may be included; for INSERT operations, the values ​​may only be the new values; for DELETE operations, the values ​​may be the old values ​​that were deleted.

[0138] Based on the structure and format of the write-ahead log file, locate and identify each individual log record. Extract necessary fields from each log record, such as transaction ID, operation type, operation target, and related values.

[0139] During the parsing process, the integrity of each record also needs to be checked, for example, by verifying the checksum to ensure that no data is corrupted. If problems are found, appropriate measures can be taken to improve the situation, such as ignoring problematic records, attempting to repair them, or reporting the errors.

[0140] Once the log records are successfully parsed, the next step is to transform this raw information into change data that can be directly applied to the system state:

[0141] The parsed fields are mapped to data structures used internally by the system. For example, table names and row IDs are mapped to in-memory data structures for fast lookup and updates. Corresponding change commands are generated based on the parsed results. For instance, if it's an update record, an "update" command is created, containing the columns to be modified and their new values. If there are multiple related log records (such as multiple updates within the same transaction), they can be sorted and merged according to transaction order or other logical rules to ensure consistency in the final application.

[0142] Accordingly, step 140 may specifically include the following steps 141 to 143:

[0143] 141. Receive update data sent by the target device.

[0144] 142. Compare the updated data with the changed data.

[0145] Optionally, in one implementation, the update data sent by the target device is unparsed update data. Accordingly, step 141 may include: receiving the update data sent by the target device and parsing the update data to obtain parsed update data.

[0146] Step 142 may include: comparing the parsed updated data with the changed data.

[0147] The update data sent by the target device comes from the consistency protocol log. The consistency protocol log can use a uniform log format and is not bound to a specific single-machine storage engine. In contrast, existing write-ahead log parsing methods rely on a single-machine storage engine, and the path and format of the write-ahead log can differ for each storage engine. Therefore, compared to existing technologies using write-ahead logs, using the consistency protocol log allows change data capture to only support parsing the consistency protocol log format, without needing to support multiple write-ahead log formats. Optionally, the consistency protocol log format can be JSON. Since all devices follow the same consistency protocol log format, only one set of parsing logic is needed to process log entries from different devices, simplifying the change data capture process.

[0148] Optionally, the parsed updated data can be compared item by item with the change data parsed from the write-ahead log. Optionally, this step can include not only checking whether the content of each record is the same, but also verifying the consistency of metadata such as transaction IDs and timestamps.

[0149] If inconsistencies are detected, a notification will pop up to help developers analyze the cause. Possible causes include network transmission errors, parsing errors, or synchronization delays between the two data sources. Developers can then take appropriate action, such as re-requesting data, correcting the parsing logic, or adjusting the synchronization strategy.

[0150] 143. If the updated data is consistent with the changed data, the updated data is sent to other devices to achieve synchronization of the updated data among devices.

[0151] Optionally, a synchronization request can be constructed that includes all necessary information. In addition to the actual updated data, auxiliary information such as version numbers and source identifiers can be included to help the receiver identify and process the data. Different synchronization strategies can be selected depending on the application scenario. For example, for scenarios with high real-time requirements, a push mechanism can be used to immediately notify all relevant devices; while for scenarios with higher fault tolerance requirements, a batch synchronization approach can be used to periodically check and synchronize data.

[0152] Synchronization requests are sent to all relevant device nodes via a reliable messaging mechanism (such as AMQP, MQTT, etc.) or by directly calling the API interface. Each receiver can process these updated data according to a predetermined process, thereby ensuring that the local copy remains consistent with the global state.

[0153] After synchronization is complete, the updating device can collect feedback from each device to confirm whether each updated data has been successfully applied. If a failure occurs, it can retry promptly or take remedial measures to avoid data inconsistencies.

[0154] In the above implementation, the updating device can obtain update data through two channels: receiving update data sent by the target device, and reading and parsing the write-ahead log from persistent storage to obtain the changed data. Then, the data obtained from different channels is compared: the update data is compared with the changed data. If they match, it indicates that the update data is accurate. Therefore, the verified update data can be sent to other devices to achieve update data synchronization. This method not only improves data consistency and reliability but also enhances the robustness and availability of the system.

[0155] Optionally, in one embodiment, the present application may further include the following steps 150 to 170:

[0156] 150. If the data update prompt information is not received within the set time period, an update data timed read request is generated based on the historical read position, wherein the update data timed read request is an update data read request generated when the data update prompt information is not received within the set time period.

[0157] The set time period is a predefined specific time window. The specific duration of the set time period can be set by the developers based on their experience and should not be construed as a limitation of this application. If no data update notification is received during the set time period, step 150 can be executed. Optionally, the failure to receive a data update notification from the target device within the set time period may be due to network latency, device malfunction, or other reasons.

[0158] If no data update notification is received within the set time period, the update device will automatically generate a new update data read request (i.e., a timed update data read request) based on the historical read position. This indicates that even without receiving a notification, the update device will proactively check for new data updates.

[0159] 160. Send the update data timed read request to the target device, the update data read request being used to instruct the target device to perform the operation of reading update data based on the historical read position.

[0160] The updating device sends a timed read request for the updated data generated in step 150 to the target device. This request includes the historical read position, instructing the target device to search for and return all subsequent updated data starting from that position. These steps enable the timely acquisition of the latest data changes even without receiving data update notifications, thereby maintaining system data consistency and real-time performance.

[0161] 170. Receive no-update feedback information sent by the target device, wherein the no-update feedback information is sent by the target device when no updated data exists after detecting the historical read position; or receive updated data sent by the target device, wherein the updated data is sent by the target device when detecting updated data exists after detecting the historical read position.

[0162] Since the update data timed read request is actively generated by the updating device, when the target device receives the update data timed read request, there are two possibilities: i. there is no update data after the historical read position; ii. there is update data after the historical read position.

[0163] For i. No updated data exists after the historical read position:

[0164] If the target device checks its logs or database for all records since the historical read position but finds no new data updates, it will send a "No Update" feedback message to the updating device, informing it that there is currently no new data to synchronize. This feedback lets the updating device know that no further action is needed and that it can wait for the next data update notification or the scheduled read cycle.

[0165] For ii. where updated data exists after the historical read position:

[0166] If the target device discovers new data updates in its logs or database that have existed since a historical read position, it packages these updates into a response message and sends it to the updating device. Upon receiving this update data, the updating device can process it according to the previous procedure, thereby achieving data synchronization between devices.

[0167] In the above implementation, the system can not only passively receive data update notifications but also actively check and synchronize data, enhancing its robustness and reliability. This method is particularly suitable for application scenarios that have high requirements for data consistency and cannot tolerate long-term data inconsistency.

[0168] Please see details Figure 1c , Figure 1c A schematic flowchart illustrating the process of an updating device obtaining update data from a target device is shown. Users can write data to a data copy on any device; please refer to [link to relevant documentation]. Figure 1c For example, a user writes data to a data replica on device 1. The data replica on device 1 appends the written data to the end of the consistency protocol log in the format specified in the consistency protocol log. Please refer to [link to relevant documentation]. Figure 1c Let's assume data 3 represents the end of the consensus protocol log. Then, the data to be written can be placed at position 3 in the consensus protocol log. The updating device can read the data at the end of the consensus protocol log by running a CDC replica. Optionally, the updating device can generate an update data read request upon receiving a data update notification and read the data at the end of the consensus protocol log by sending the update data read request to the target device; alternatively, if no data update notification is received within a set time period, it can generate a timed update data read request and read the data at the end of the consensus protocol log by sending the timed update data read request to the target device. It should be understood that the specific method by which the updating device reads the data at the end of the consensus protocol log should not be construed as a limitation of this application. The updating device will then send the read update data to other devices. Optionally, please refer to... Figure 1c Specifically, the update device can send the read update data to device 2, and the update device can also send the read update data to other devices running the downstream data system.

[0169] Other devices in the downstream data system can be various types of data processing or storage systems. They receive updated data sent from the updating device and further perform data processing, analysis, storage, and other operations. Optionally, in one specific embodiment, other devices in the downstream data system may include:

[0170] Data analytics platforms: These systems can perform complex analyses on updated data, such as trend analysis and predictive analysis. For example, a data analytics platform may include business intelligence (BI) tools and data warehouses.

[0171] Data storage systems: Updated data may be sent to various types of databases or data lakes for long-term storage, for later querying and use. Data storage systems can include relational databases, NoSQL databases, distributed file systems, etc.

[0172] Detection and Alarm System: In operation and maintenance scenarios, updated data may be used to detect the health status of the system. When an anomaly is detected, an alarm can be issued in a timely manner to help operation and maintenance personnel quickly locate and resolve problems. In the data processing method provided in this application embodiment, a data update prompt message is received. This message is sent by the target device storing the consistency protocol log when it detects an update to the consistency protocol log. An update data read request is generated based on the historical read position, where the historical read position is the end position of the data read during the previous read of the consistency protocol log. The update data read request is sent to the target device so that the target device can read the updated data according to the historical read position. The update data sent by the target device is received and then sent to other devices to achieve synchronization of the update data between devices. In this application embodiment, when the target device storing the consistency protocol log detects an update to the consistency protocol log, it sends a data update prompt message to the updating device executing the data processing method of this application. After receiving a data update notification, the updating device generates an update data read request based on the historical read position and sends the request to the target device to obtain data after the historical read position (i.e., data updated after the previous read of the consistency protocol log) and synchronize it with other devices. In this embodiment, the device is aware of the data update as soon as the consistency protocol log is updated, promptly generates and sends an update data read request to read the updated data from the consistency protocol log.

[0173] The embodiments of this application can make the data synchronization process faster and more timely.

[0174] In this embodiment, a data processing method is also provided, which is executed by the target device. For example... Figure 2a As shown, the specific process of this method may include the following steps 210 to 240:

[0175] 210. When a consistency protocol log update is detected, send a data update notification message to the updating device.

[0176] A consistency protocol is an algorithm used in distributed database systems to ensure data consistency among multiple data replicas. Consistency protocols coordinate data replicas distributed across different nodes, ensuring they are updated and synchronized in a controlled manner, thereby avoiding data inconsistency problems.

[0177] Alternatively, the consensus protocol can be implemented in any of the following ways:

[0178] In one specific implementation, the implementation process of the consensus protocol may include:

[0179] The proposer selects a unique number N and sends a "prepare(N)" request to a majority of acceptors. The number N is unique and increments throughout the system.

[0180] If the number N in the Prepare request received by the recipient is greater than the number in all previously received Prepare requests, then the recipient will no longer accept any proposals with numbers less than N. Specifically, after receiving a proposal, the recipient first obtains the number of the received proposal and compares the proposal number with the value N. If the proposal number is less than the value N, the proposal is ignored.

[0181] After receiving a Prepare request with number N, the recipient can send feedback to the proposer, indicating that it has received the Prepare request with number N.

[0182] If the proposer receives positive responses to the Prepare request from a majority of the receivers, it can send an "Accept" request (Accept(N,V)) to those receivers. Here, V represents the data to be synchronized.

[0183] Upon receiving an Accept request, the receiver compares the number N in the request with its own stored number. If the number N is not less than its own stored number, the receiver accepts the data V to be synchronized and records it. If the number N is less than its own stored number, the receiver ignores the Accept request.

[0184] Once the data V to be synchronized is accepted by a majority of receivers, this value is considered finalized. Learners can find out the accepted value by asking the receivers, or the receivers can proactively notify the learners.

[0185] The proposer is the logical entity that initiates the proposal. It can be a client request handler, a dedicated service module, or any program capable of generating data items that need to be agreed upon.

[0186] An acceptor is a logical entity that receives and may accept proposals. It is typically part of a storage system and is responsible for persisting proposal information.

[0187] A learner is a logical entity that obtains information about accepted proposals from the recipients. It can be the system's state machine, another service, or other component that needs to know the eventual consistent value.

[0188] Learners, acceptors, and proposers can run on the same physical machine or be distributed across different servers or computing nodes.

[0189] In another implementation, the specific implementation process of the consensus protocol may include:

[0190] When a follower does not receive any messages for a period of time, the follower will be converted to a candidate state and a new round of election will begin.

[0191] The candidate first increments its own term number and then sends a vote request (RequestVote RPC) to other nodes in the cluster.

[0192] A node that receives a voting request can decide whether to vote for the candidate based on preset rules (e.g., whether the candidate's term number is greater than or equal to the current node's term number).

[0193] If a candidate receives a majority of votes from the nodes, it can become the new leader. If multiple candidates run for office simultaneously, resulting in no single candidate receiving a majority of votes, the process will wait until the next election cycle to try again.

[0194] Once a leader is elected, it begins receiving client requests, adding these requests as log entries to its own log, and then synchronizing them with all followers. Each log entry contains a command (specified by the client), the term number that created the entry, and an index position.

[0195] The leader synchronizes log entries by sending AppendEntries RPC requests to followers. Upon receiving an AppendEntries RPC request, a follower first checks its own log for entries with the same index and term number. If an entry exists, it means the log entries before that position are matched. If no entry exists, or if an entry exists but the index and term number do not match, it indicates a discrepancy between the follower's and leader's logs. In cases of discrepancies, the leader can identify the point of divergence and truncate its log at that point, removing subsequent entries that differ from the leader's. Next, the follower updates its own log based on the leader's log, filling in missing entries, until the follower's and leader's logs are fully synchronized. The AppendEntries RPC request is used to achieve log synchronization between the leader and followers.

[0196] The leader is the core node responsible for managing the entire cluster operation. It handles all requests from clients and adds these requests as log entries to its own log.

[0197] Followers are passive roles; they do not initiate requests but respond to requests from leaders or candidates.

[0198] A candidate is a node that attempts to become the new leader through election when there is no effective leader.

[0199] Leader, Follower, and Candidate are not specific physical devices, but rather nodes that play different roles or states in a distributed system. Leader, Follower, and Candidate can be implemented by different processes or threads on the same physical server, or they can be distributed across different physical servers, depending on the system's architecture and requirements.

[0200] In another implementation, the specific implementation process of the consensus protocol may include:

[0201] The coordinator initiates a transaction and sends a "prepare request" to all involved participants, asking them if they are ready to commit the transaction.

[0202] Upon receiving a "ready" request, each participant performs the following actions: checks if it has sufficient resources to complete the transaction (e.g., checks disk space, locks, etc.). If it can complete the transaction, it locks the relevant resources, logs the transaction to indicate that it is ready to commit, and returns a "Prepared" message to the coordinator. If it cannot complete the transaction, it returns an error message.

[0203] The coordinator waits for responses from all participants. If all participants return "Ready," the process proceeds to the Commit Phase. If any participant is not ready, the coordinator will decide to abort the transaction.

[0204] Specifically, if all participants have completed their preparations:

[0205] The coordinator sends a "commit" request to all participants, instructing them to formally commit the transaction. Upon receiving the "commit" request, each participant executes the transaction commit operation, releases previously locked resources, and returns a "committed" confirmation to the coordinator. Once the coordinator receives confirmations from all participants, it marks the transaction as successfully completed. The transaction commit operation is used to implement modifications to data based on business needs. For example, in an e-commerce application, this might involve data updates across multiple steps, such as order creation, inventory reduction, and payment processing.

[0206] If at least one participant fails to prepare:

[0207] The coordinator sends an "Abort request" to all participants, instructing them to cancel the transaction. Upon receiving the "Abort request," each participant reverts its changes, unlocks resources, and returns an "Aborted" confirmation to the coordinator. Once the coordinator receives confirmations from all participants, it marks the transaction as failed.

[0208] The coordinator is a separate service or process responsible for managing the entire distributed transaction process. It does not directly execute the operations within the transaction, but rather ensures that all participants consistently decide whether to commit or rollback the transaction through interaction with the participants. Participants are the nodes that actually execute the transaction operations, such as database servers or other storage services. Each participant is responsible for managing its local resources and performing corresponding operations according to the coordinator's instructions. The coordinator and participants can run as different processes or threads on the same physical machine, or they can be distributed across multiple different physical devices, depending on the system architecture and requirements.

[0209] It should be understood that the specific implementation of the consensus protocol should not be construed as a limitation on this application.

[0210] Consistency protocols can maintain a logical log, known as the consistency protocol log. When data is updated, the updated data is written to the consistency protocol log. Specifically, the consistency protocol log is a special type of log maintained by the consistency protocol in a distributed database system to ensure consistency among multiple data replicas. This log records all operations that modify the data (such as inserts, updates, or deletions) so that each data replica can apply these operations in the same order, thereby guaranteeing data consistency.

[0211] The target device is the device that stores the consensus protocol log. The consensus protocol log can be stored on one device or multiple devices. If the consensus protocol log is stored on multiple devices, when the consensus protocol log of one device is updated, that device broadcasts the log update data to the other devices that store the consensus protocol log. Optionally, the target device storing the consensus protocol log may be different from the update device performing steps 110 to 140, or it may be the same device.

[0212] For ease of description, let's take the example that the target device storing the consistency protocol log is not the same device as the update device performing steps 110 to 140.

[0213] When a target device detects an update to its stored consistency protocol log, it can send a data update notification to the updating device. The data update notification is a message indicating that a data update has occurred in the consistency protocol log. It serves to remind the updating device that a new set of data changes is ready to be synchronized. Optionally, the data update notification may include some basic information, such as the timestamp or version number of the change, but importantly, it triggers the updating device to begin a new round of data synchronization.

[0214] 220. Receive an update data read request sent by the update device, wherein the update data read request includes a historical read position, which is the end position of the data read when the consistency protocol log was read last time.

[0215] 230. Based on the historical read position, read the updated data from the consistency protocol log.

[0216] After receiving an update data read request, the target device locates the corresponding position based on the historical read position included in the update data read request, and begins reading all change records from that position onwards until the termination condition specified in the update data read request is met (such as reaching the maximum number of entries or encountering the end of the log). Then, the target device packages the collected update data into a response message and returns it to the update device.

[0217] 240. Send update data to the update device.

[0218] The updating device receives the response message sent by the target device, parses the update data from the response message, and then sends the update data to other devices, thereby realizing the synchronization process of update data between devices.

[0219] In the above implementation, when the consistency protocol log stored in the target device is updated, the updating device can be notified that a data update has occurred, and an update data read request can be generated and sent in a timely manner to read the updated data in the consistency protocol log of the target device, which can make the data synchronization process faster and more timely.

[0220] Optionally, in one embodiment, the present application may further include the following steps 250 to 290:

[0221] 250. Receive an update data timed read request, wherein the update data timed read request includes the historical read position; the update data timed read request is an update data read request generated by the update device when it has not received the data update prompt information within a set time period.

[0222] The set time period is a predefined specific time window. The specific duration of the set time period can be set by the developers based on their experience and should not be construed as a limitation of this application. If no data update notification is received during the set time period, step 150 can be executed. Optionally, the failure to receive a data update notification from the target device within the set time period may be due to network latency, device malfunction, or other reasons.

[0223] If no data update notification is received within the set time period, the update device will automatically generate a new update data read request (i.e., a timed update data read request) based on the historical read position. This indicates that even without receiving a notification, the update device will proactively check for new data updates.

[0224] 260. If no updated data exists in the consistency protocol log after the historical read position, generate a non-updated feedback message and send the non-updated feedback message to the updating device.

[0225] Since the update data timed read request is actively generated by the updating device, when the target device receives the update data timed read request, there are two possibilities: i. there is no update data after the historical read position; ii. there is update data after the historical read position.

[0226] For i. No updated data exists after the historical read position:

[0227] If the target device checks its logs or database for all records since the historical read position but finds no new data updates, it will send a "No Update" feedback message to the updating device, informing it that there is currently no new data to synchronize. This feedback lets the updating device know that no further action is needed and that it can wait for the next data update notification or the scheduled read cycle.

[0228] 270. If there is updated data in the consistency protocol log after the historical read position, the updated data is sent to the updating device.

[0229] Continuing with the example above, for ii. where updated data exists after a historical read position:

[0230] If the target device discovers new data updates in its logs or database that have existed since a historical read position, it packages these updates into a response message and sends it to the updating device. Upon receiving this update data, the updating device can process it according to the previous procedure, thereby achieving data synchronization between devices.

[0231] In the above implementation, the system can not only passively receive data update notifications but also actively check and synchronize data, enhancing its robustness and reliability. This method is particularly suitable for application scenarios that have high requirements for data consistency and cannot tolerate long-term data inconsistency.

[0232] 280. If no heartbeat information is received from the updating device within the heartbeat time period, a new updating device is determined from among the multiple devices that normally send heartbeat information.

[0233] In distributed systems, a heartbeat mechanism can be used to detect the status of each node. Each node, including update devices, can periodically send heartbeat messages to the target device to indicate that it is still online and working normally.

[0234] If no heartbeat is received from the updating device within the predetermined heartbeat period, the target device can determine that the updating device may be faulty or unreachable. In this case, the target device can select a new updating device from among the devices that are still able to send heartbeats normally. The method for determining a new updating device from multiple devices that are still sending heartbeats normally can be: selecting the one with the best performance, the one with the lowest load, or randomly selecting an available device.

[0235] Optionally, in one implementation, if the target device does not receive heartbeat information from the updating device within the heartbeat time period, the target device may disqualify the updating device from being an updating device.

[0236] 290. Send an update priority identifier to the new update device, the update priority identifier being used to instruct the new update device, upon receiving update data, to send the update data along with the update priority identifier to other devices.

[0237] The update priority identifier is used to indicate the priority of the updated data. Optionally, when the target device generates the update priority identifier, the priority of the generated update priority identifier increases with the generation time: that is, the later the generation time of the update priority identifier, the higher the priority.

[0238] Update priority identifiers help the receiver determine the actual data to be synchronized when receiving update data. For example, if a receiver receives messages representing update data from two updating devices, and one message carries an update priority identifier while the other does not, the update data in the message carrying the update priority identifier will prevail. If both messages carry update priority identifiers, the one with the higher priority identifier will be used.

[0239] In the above implementation, it is possible to quickly switch to a backup device when a failure of the updating device is detected, and to ensure that the importance and urgency of the data are taken into account during the data synchronization process, thereby optimizing overall performance and user experience.

[0240] Please see details Figure 2b , Figure 2b A schematic block diagram of the update device is shown. If the routing center does not receive heartbeat information from the update device within the heartbeat time period, it determines that the update device is faulty and selects a new update device from among the multiple devices that are normally sending heartbeat information. Please refer to [link to relevant documentation]. Figure 2b The routing center can disqualify a faulty update device from becoming an update device by sending a delete CDC replica command to the faulty update device; alternatively, it can identify a new update device by sending an add CDC replica command to the new update device. It is understood that the routing center can be the same device as the target device running the consistency protocol log, or it can be a different device. In this embodiment, we will use the example of the routing center and the target device being the same device for description.

[0241] Optionally, in one implementation, a communication failure may occur between the fault update device and the routing center. In the current situation, even if the routing center sends an instruction to the fault update device to delete the CDC copy, the fault update device may fail to delete the CDC copy. Therefore, the downstream data system may receive update data from two update devices. In this case, the routing center can send an update priority identifier when adding a CDC copy to the new update device. The new update device can then send the update priority identifier along with subsequent update data to the downstream data system. For the downstream data system, although it receives two sets of update data, one set carries the update priority identifier, while the other does not. Therefore, the downstream data system can update its data based on the update data carrying the update priority identifier.

[0242] Optionally, in one embodiment, taking the routing center and the target device as examples where they are not the same device, the embodiments of this application may further include the following data processing method. This data processing method is applied to the routing center, which is communicatively connected to multiple devices, including an update device. For details, please refer to [link to relevant documentation]. Figure 2b ;

[0243] The method may include the following steps B1 to B2:

[0244] B1. If no heartbeat information is received from the updating device within the heartbeat time period, a new updating device is determined from among the multiple devices that normally send heartbeat information.

[0245] B2. Send an update priority identifier to the new update device. The update priority identifier is used to instruct the new update device to send the update data along with the update priority identifier to other devices when it receives update data.

[0246] It should be understood that steps B1 to B2 above correspond to steps 280 to 290, and will not be repeated here.

[0247] In this embodiment, the method of this application embodiment is described in detail using the interaction between the target device and the updating device as an example. Figure 3 As shown, the specific process of a data processing method is as follows:

[0248] 301. When the target device detects an update to the consistency protocol log, it sends a data update notification to the updating device.

[0249] 302. Update device data reception update prompt information.

[0250] 303. The updating device generates an update data read request based on the historical read position, wherein the historical read position is the end position of the data read when the consistency protocol log was read last time.

[0251] 304. The updating device sends an update data read request to the target device.

[0252] 305. The target device receives an update data read request sent by the updating device, wherein the update data read request includes the historical read position.

[0253] 306. The target device reads updated data from the consistency protocol log based on the historical read position.

[0254] 307. The target device sends update data to the updating device.

[0255] 308. The updating device receives the update data sent by the target device and sends the update data to other devices to achieve synchronization of the update data between devices.

[0256] Optionally, in one embodiment, prior to step 308, the present application embodiment may further include:

[0257] The update device periodically reads the write-ahead log from the persistent storage device; the read write-ahead log is parsed to obtain the change data.

[0258] Accordingly, step 308 may specifically include the following steps 3081 to 3083:

[0259] 3081. Receive update data sent by the target device.

[0260] 3082. Compare the updated data with the changed data.

[0261] 3083. If the updated data is consistent with the changed data, the updated data is sent to other devices to achieve synchronization of the updated data between devices.

[0262] 309. If the updating device does not receive the data update prompt information within a set time period, it generates a timed update data read request based on the historical read position, wherein the timed update data read request is an update data read request generated when the data update prompt information is not received within the set time period.

[0263] 310. The updating device sends the update data timed read request to the target device.

[0264] 311. The target device receives a timed read request for updated data.

[0265] 312. If the consistency protocol log does not contain updated data after the historical read position, the target device generates a non-updated feedback message and sends the non-updated feedback message to the updating device.

[0266] 313. If there is updated data in the consistency protocol log after the historical read position, the target device sends the updated data to the updating device.

[0267] 314. If no heartbeat information is received from the updating device within the heartbeat time period, the target device determines a new updating device from among the multiple devices that normally send heartbeat information.

[0268] 315. The target device sends an update priority identifier to the new updating device. The update priority identifier is used to instruct the new updating device to send the update data along with the update priority identifier to other devices when it receives update data.

[0269] The specific execution process of steps 301 to 315 has been explained in detail above, and will not be repeated here.

[0270] In the data processing method provided in this application embodiment, a data update notification message is received. This message is sent by a target device storing a consistency protocol log when it detects an update to the consistency protocol log. An update data read request is generated based on the historical read position, where the historical read position is the end position of the data read during the previous read of the consistency protocol log. The update data read request is sent to the target device so that the target device can read the updated data based on the historical read position. The updated data sent by the target device is received and then sent to other devices to achieve synchronization of the updated data among the devices. In this application embodiment, when the target device storing the consistency protocol log detects an update to the consistency protocol log, it sends a data update notification message to the updating device executing the data processing method of this application. After receiving the data update notification message, the updating device generates an update data read request based on the historical read position and sends the request to the target device to obtain the data after the historical read position (i.e., the data updated after the previous read of the consistency protocol log) from the target device and synchronize it with other devices. In this embodiment of the application, when the consistency protocol log is updated, it can know that a data update has occurred, and promptly generate and send an update data read request to read the updated data in the consistency protocol log.

[0271] The embodiments of this application can make the data synchronization process faster and more timely.

[0272] To better implement the above methods, embodiments of this application also provide a data processing apparatus, such as... Figure 4 As shown, the device includes:

[0273] The information receiving unit 401 is used to receive data update prompt information, wherein the data update prompt information is sent by the target device storing the consistency protocol log when the consistency protocol log is detected to be updated;

[0274] The request generation unit 402 is used to generate an update data read request based on the historical read position, wherein the historical read position is the end position of the data read when the consistency protocol log was read last time;

[0275] Request sending unit 403 is used to send the update data read request to the target device, the update data read request being used to instruct the target device to perform the operation of reading update data based on the historical read position;

[0276] The update synchronization unit 404 is used to receive update data sent by the target device and send the update data to other devices to realize the synchronization of the update data between devices.

[0277] In one embodiment, the device further includes:

[0278] A timed request generation unit is used to generate a timed update data read request based on the historical read position when no data update prompt information is received within a set time period. The timed update data read request is an update data read request generated when no data update prompt information is received within the set time period.

[0279] A timed request sending unit is used to send the update data timed read request to the target device, the update data read request being used to instruct the target device to perform the operation of reading update data based on the historical read position.

[0280] In one embodiment, the device further includes:

[0281] An information feedback unit is configured to receive non-updated feedback information sent by the target device, wherein the non-updated feedback information is sent by the target device when no updated data exists after detecting the historical reading position; or to receive updated data sent by the target device, wherein the updated data is sent by the target device when updated data exists after detecting the historical reading position.

[0282] In one embodiment, the device further includes:

[0283] The periodic read unit is used to periodically read the write-ahead log from the persistent storage device;

[0284] The log parsing unit is used to parse the read write-ahead log to obtain the changed data;

[0285] Accordingly, update synchronization unit 404, including:

[0286] An update receiving subunit is used to receive update data sent by the target device;

[0287] A data comparison subunit is used to compare the updated data with the changed data;

[0288] The data sending subunit is used to send the updated data to other devices when the updated data is consistent with the changed data, so as to realize the synchronization of the updated data between devices.

[0289] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0290] In the data processing method provided in this application embodiment, a data update notification message is received. This message is sent by a target device storing a consistency protocol log when it detects an update to the consistency protocol log. An update data read request is generated based on the historical read position, where the historical read position is the end position of the data read during the previous read of the consistency protocol log. The update data read request is sent to the target device so that the target device can read the updated data based on the historical read position. The updated data sent by the target device is received and then sent to other devices to achieve synchronization of the updated data among the devices. In this application embodiment, when the target device storing the consistency protocol log detects an update to the consistency protocol log, it sends a data update notification message to the updating device executing the data processing method of this application. After receiving the data update notification message, the updating device generates an update data read request based on the historical read position and sends the request to the target device to obtain the data after the historical read position (i.e., the data updated after the previous read of the consistency protocol log) from the target device and synchronize it with other devices. In this embodiment of the application, when the consistency protocol log is updated, it can know that a data update has occurred, and promptly generate and send an update data read request to read the updated data in the consistency protocol log.

[0291] The embodiments of this application can make the data synchronization process faster and more timely.

[0292] To better implement the above methods, embodiments of this application also provide a data processing apparatus, such as... Figure 5 As shown, the device includes:

[0293] The information sending unit 501 is used to send a data update prompt message to the updating device when a consistency protocol log update is detected;

[0294] The request receiving unit 502 is used to receive the update data reading request sent by the updating device, wherein the update data reading request includes a historical reading position, which is the end position of the data read when the consistency protocol log was read last time;

[0295] Data reading unit 503 is used to read updated data from the consistency protocol log based on the historical read position;

[0296] The data sending unit 504 is used to send update data to the updating device to achieve synchronization of the update data between devices.

[0297] In one embodiment, the device further includes:

[0298] A timed request receiving unit is used to receive a timed update data read request, wherein the timed update data read request includes the historical read position; the timed update data read request is an update data read request generated by the update device when it has not received the data update prompt information within a set time period.

[0299] The feedback sending unit is used to generate no-update feedback information when the consistency protocol log has no updated data after the historical read position, and send the no-update feedback information to the updating device;

[0300] The update sending unit is used to send the update data to the update device when the consistency protocol log has updated data after the historical read position.

[0301] In one embodiment, the device further includes:

[0302] The device determination unit is used to determine a new update device from multiple devices that normally send heartbeat information when no heartbeat information is received from the update device within the heartbeat time period.

[0303] The priority identifier sending unit is used to send an update priority identifier to the new update device. The update priority identifier is used to instruct the new update device to send the update data along with the update priority identifier to other devices when it receives update data.

[0304] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0305] In the data processing method provided in this application embodiment, a data update notification message is received. This message is sent by a target device storing a consistency protocol log when it detects an update to the consistency protocol log. An update data read request is generated based on the historical read position, where the historical read position is the end position of the data read during the previous read of the consistency protocol log. The update data read request is sent to the target device so that the target device can read the updated data based on the historical read position. The updated data sent by the target device is received and then sent to other devices to achieve synchronization of the updated data among the devices. In this application embodiment, when the target device storing the consistency protocol log detects an update to the consistency protocol log, it sends a data update notification message to the updating device executing the data processing method of this application. After receiving the data update notification message, the updating device generates an update data read request based on the historical read position and sends the request to the target device to obtain the data after the historical read position (i.e., the data updated after the previous read of the consistency protocol log) from the target device and synchronize it with other devices. In this embodiment of the application, when the consistency protocol log is updated, it can know that a data update has occurred, and promptly generate and send an update data read request to read the updated data in the consistency protocol log.

[0306] The embodiments of this application can make the data synchronization process faster and more timely.

[0307] To better implement the above methods, this application embodiment further provides a data processing apparatus, such as... Figure 6 As shown, the device includes:

[0308] The update device determination unit 601 is used to determine a new update device from multiple devices that normally send heartbeat information if no heartbeat information is received from the update device within the heartbeat time period.

[0309] The identifier sending unit 602 is used to send an update priority identifier to the new updating device. The update priority identifier is used to instruct the new updating device to send the update data along with the update priority identifier to other devices when it receives update data. This application embodiment also provides an electronic device, which can be a terminal, server, or other devices. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0310] In some embodiments, the data processing device may also be integrated into multiple electronic devices. For example, the data processing device may be integrated into multiple servers, with multiple servers implementing the data processing method of this application. In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 7 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0311] The electronic device may include components such as a processor 701 with one or more processing cores, a memory 702 with one or more computer-readable storage media, a power supply 703, an input module 704, and a communication module 705. Those skilled in the art will understand that... Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0312] The processor 701 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes various functions and processes data by running or executing software programs and / or modules stored in the memory 702, and by calling data stored in the memory 702. In some embodiments, the processor 701 may include one or more processing cores; in some embodiments, the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 701.

[0313] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing model training by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.

[0314] The electronic device also includes a power supply 703 that supplies power to the various components. In some embodiments, the power supply 703 can be logically connected to the processor 701 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 703 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0315] The electronic device may also include an input module 704, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0316] The electronic device may also include a communication module 705. In some embodiments, the communication module 705 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 705 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0317] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 701 in the electronic device loads the executable files corresponding to the processes of one or more application programs into the memory 702 according to the following instructions, and the processor 701 runs the application programs stored in the memory 702, thereby realizing the various functions in the various method embodiments of this application, as follows:

[0318] The system receives a data update notification, which is sent by a target device storing a consistency protocol log when it detects an update to the consistency protocol log. Based on a historical read position, it generates an update data read request, where the historical read position is the end position of the data read during the previous read of the consistency protocol log. The system sends the update data read request to the target device, instructing the target device to perform the operation of reading update data based on the historical read position. It receives the update data sent by the target device and sends the update data to other devices to achieve synchronization of the update data among the devices.

[0319] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0320] Therefore, embodiments of this application provide a computer-readable storage medium storing instructions that can be loaded by a processor to perform steps in any of the data processing methods provided in embodiments of this application. For example, the instructions can perform the following steps:

[0321] The system receives a data update notification, which is sent by a target device storing a consistency protocol log when it detects an update to the consistency protocol log. Based on a historical read position, it generates an update data read request, where the historical read position is the end position of the data read during the previous read of the consistency protocol log. The system sends the update data read request to the target device, instructing the target device to perform the operation of reading update data based on the historical read position. It receives the update data sent by the target device and sends the update data to other devices to achieve synchronization of the update data among the devices.

[0322] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0323] According to one aspect of this application, a computer program product or computer program is provided, comprising instructions stored in a computer-readable storage medium. A processor of a computer device reads the instructions from the computer-readable storage medium and executes the instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0324] Since the instructions stored in the storage medium can execute the steps of any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0325] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0326] The foregoing has provided a detailed description of a data processing method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, The method includes: Receive data update notification information, wherein the data update notification information is sent by the target device storing the consistency protocol log when it detects an update to the consistency protocol log; Based on the historical read position, an update data read request is generated, wherein the historical read position is the end position of the data read when the consensus protocol log was read last time; Send the update data read request to the target device, the update data read request being used to instruct the target device to perform the operation of reading update data based on the historical read position; Receive update data sent by the target device and send the update data to other devices.

2. The method as described in claim 1, characterized in that, The method further includes: If the data update notification is not received within the set time period, an update data timed read request is generated based on the historical read position. The update data timed read request is an update data read request generated when the data update notification is not received within the set time period. The update data timed read request is sent to the target device, and the update data timed read request is used to instruct the target device to perform the operation of reading update data based on the historical read position.

3. The method as described in claim 2, characterized in that, After sending the update data timed read request to the target device, the method further includes: Receive a no-update feedback message sent by the target device, wherein the no-update feedback message is sent by the target device when it detects that no updated data exists after the historical read position; or The target device receives update data, wherein the update data is sent by the target device when it detects that update data exists after the historical read position.

4. The method as described in claim 1, characterized in that, Before receiving the update data sent by the target device and sending the update data to other devices, the method further includes: Periodically read the write-ahead log from persistent storage; The read write-ahead log is parsed to obtain the changed data; The step of receiving update data sent by the target device and sending the update data to other devices includes: Receive update data sent by the target device; Compare the updated data with the changed data; If the updated data is consistent with the changed data, the updated data is sent to other devices to achieve synchronization of the updated data among devices.

5. The method as described in claim 4, characterized in that, The update data sent by the target device is unparsed update data; The receiving of update data sent by the target device includes: Receive update data sent by the target device, and parse the update data to obtain parsed update data; The step of comparing the updated data with the changed data includes: The parsed updated data is compared with the changed data.

6. A data processing method, characterized in that, The method includes: When a consistency protocol log update is detected, a data update notification message is sent to the updating device. Receive an update data read request sent by the update device, wherein the update data read request includes a historical read position, which is the end position of the data read when the consistency protocol log was read last time; Based on the historical read position, update data is read from the consistency protocol log; Send update data to the update device.

7. The method as described in claim 6, characterized in that, The method further includes: Receive an update data timed read request, wherein the update data timed read request includes the historical read position; the update data timed read request is an update data read request generated by the update device when it has not received the data update prompt information within a set time period; If no updated data exists in the consistency protocol log after the historical read position, a non-updated feedback message is generated and sent to the updating device; If the consistency protocol log contains updated data after the historical read position, the updated data is sent to the updating device.

8. The method as described in claim 6, characterized in that, The method further includes: If no heartbeat information is received from the updating device within the heartbeat time period, a new updating device is determined from among the multiple devices that normally send heartbeat information; Send an update priority identifier to the new update device. The update priority identifier is used to instruct the new update device to send the update data along with the update priority identifier to other devices when it receives update data.

9. A data processing method, characterized in that, It is applied to a routing center that communicates with multiple devices, including update devices; The method includes: If no heartbeat information is received from the updating device within the heartbeat time period, a new updating device is determined from among the multiple devices that normally send heartbeat information; Send an update priority identifier to the new update device. The update priority identifier is used to instruct the new update device to send the update data along with the update priority identifier to other devices when it receives update data.

10. A data processing apparatus, characterized in that, The device includes: An information receiving unit is used to receive data update notification information, wherein the data update notification information is sent by a target device storing a consistency protocol log when it detects an update to the consistency protocol log; The request generation unit is used to generate an update data read request based on the historical read position, wherein the historical read position is the end position of the data read when the consistency protocol log was read last time; A request sending unit is configured to send the update data read request to the target device, wherein the update data read request is used to instruct the target device to perform an operation of reading update data based on the historical read position; The update synchronization unit is used to receive update data sent by the target device and send the update data to other devices.

11. A data processing apparatus, characterized in that, The device includes: The information sending unit is used to send a data update prompt message to the updating device when an update to the consistency protocol log is detected; The request receiving unit is used to receive the update data read request sent by the update device, wherein the update data read request includes a historical read position, which is the end position of the data read when the consistency protocol log was read last time; A data reading unit is used to read updated data from the consistency protocol log based on the historical read position; A data sending unit is used to send update data to the update device.

12. A data processing apparatus, characterized in that, It is applied to a routing center that communicates with multiple devices, including update devices; The device includes: An update device determination unit is used to determine a new update device from multiple devices that normally send heartbeat information if no heartbeat information is received from the update device within the heartbeat time period. The identifier sending unit is used to send an update priority identifier to the new update device. The update priority identifier is used to instruct the new update device to send the update data along with the update priority identifier to other devices when it receives update data.

13. An electronic device, characterized in that, It includes a processor and a memory, the memory storing instructions; the processor loads instructions from the memory to perform the steps of the data processing method as described in any one of claims 1 to 9.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions adapted for loading by a processor to perform the steps of the data processing method according to any one of claims 1 to 9.

15. A computer program product, characterized in that, It includes instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 9.

Images