Data delivery method, data processing method and corresponding device, system and medium

Abstract

The application provides a data delivery method, a data processing method, and corresponding equipment, a system, and a medium. The data delivery method comprises the following steps: acquiring a real-time consumption point in a write-ahead log system (WAL) log of a database and real-time change data of the database; the real-time change data is change data corresponding to the real-time consumption point in change data for changing the database; and the real-time consumption point and the real-time change data are synchronously sent to a message system. According to the embodiment of the application, the precise once delivery and the precise once processing can be realized, and the accuracy of stream computing is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data delivery method, a data processing method, and corresponding equipment, systems, and media. Background Technology

[0002] In the database field, modified data that alters the database is often delivered to a messaging system. The downstream stream computing system subscribes to the modified data in the messaging system and performs real-time calculations as needed. However, the modified data in the messaging system is often duplicated. When the stream computing system performs calculations on the modified data in the messaging system, it may perform the same modified data repeatedly, resulting in low accuracy of the stream computing results. Summary of the Invention

[0003] This application provides a data delivery method, a data processing method, and corresponding equipment, systems, and media to solve the problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide a data delivery method that can be applied to a data subscription system, the method comprising:

[0005] Retrieve the real-time consumption points and real-time change data of the database from the WAL (Write-Ahead Logging) log; the real-time change data of the database is the change data that corresponds to the real-time consumption points in the change data that makes changes to the database.

[0006] Real-time consumption locations and real-time change data are sent synchronously to the messaging system.

[0007] Secondly, embodiments of this application provide a data processing method that can be applied to a stream computing system, the method comprising:

[0008] Obtain changed data from the database stored in the messaging system; the changed data is delivered to the messaging system using the data delivery method provided in the first aspect of this application.

[0009] Calculate the changes to the database.

[0010] Thirdly, a data delivery device, applicable to a data subscription system, includes:

[0011] The data acquisition module is used to acquire the real-time consumption points and real-time change data of the database from the WAL log; the real-time change data of the database is the change data corresponding to the real-time consumption points in the change data of the database.

[0012] The data delivery module is used to synchronously send real-time consumption locations and real-time change data to the messaging system.

[0013] Fourthly, a data processing apparatus applicable to a stream computing system, the apparatus comprising:

[0014] The data acquisition module is used to acquire change data stored in the database of the messaging system; the change data is delivered to the messaging system by the data delivery device provided in the third aspect of this application.

[0015] The data calculation module is used to perform calculations on the changed data in the database.

[0016] Fifthly, embodiments of this application provide an electronic device that can serve as a data subscription system or a stream computing system. The electronic device may include a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it implements any of the methods provided in the embodiments of this application.

[0017] Sixthly, embodiments of this application provide a data subscription system, including: a master node and worker nodes connected by communication;

[0018] The master node is used to: assign WAL log consumption tasks to worker nodes;

[0019] Worker nodes are used to: respond to consumption tasks by reading the current offset of the WAL log from the master node; determine the real-time consumption point based on the current offset and read the real-time change data from the database storage system; and synchronously send the real-time consumption point and real-time change data to the message system.

[0020] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods provided in embodiments of this application.

[0021] Compared with the prior art, this application has the following advantages:

[0022] The data subscription system can synchronously send the real-time consumption points in the database's WAL logs and their corresponding real-time changes to the database to the messaging system. The real-time consumption points can be used to determine the location where the data subscription system consumes the database's changed data. By synchronously sending the real-time consumption points and real-time changes to the messaging system, downstream systems will not read duplicate changed data when consuming data from the messaging system, thus achieving exact-once delivery. Based on this, subsequent stream computing can achieve exact-once processing, ensuring that each changed data in the messaging system is calculated only once, thereby avoiding the impact of duplicate calculations on the accuracy of the calculation and ensuring the accuracy of the stream computing results.

[0023] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0024] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0025] Figure 1 A schematic diagram illustrating an application scenario for implementing the method of this application embodiment;

[0026] Figure 2 A flowchart illustrating a data delivery method provided in an embodiment of this application;

[0027] Figure 3 A partial flowchart illustrating another data delivery method provided in this application embodiment;

[0028] Figure 4 An exemplary flowchart illustrating the allocation of consumption tasks and request data between master nodes and worker nodes in this application embodiment;

[0029] Figure 5 This is a schematic diagram illustrating the principle of an optional implementation of the data delivery method according to an embodiment of this application;

[0030] Figure 6 This is a schematic diagram illustrating the principle of parsing the latest changed data and submitting data to the messaging system in the embodiments of this application;

[0031] Figure 7 This is a schematic diagram illustrating the principle of persisting data to the coordination service system according to an embodiment of this application;

[0032] Figure 8 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0033] Figure 9 A schematic diagram of the structural framework of a data delivery device provided in an embodiment of this application;

[0034] Figure 10 A schematic diagram of the structural framework of a data processing device provided in this application embodiment; and

[0035] Figure 11 This is a schematic diagram of the structural framework of an electronic device provided in an embodiment of this application. Detailed Implementation

[0036] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0037] First, the following is a brief introduction to several technical terms involved in this application:

[0038] CDC (Change Data Capture): refers to the process of identifying and capturing changes made to data in a database, and then transmitting the resulting change data to downstream processes or systems in real time.

[0039] Exactly Once Delivery: This refers to the semantics of delivering changed data from the database to the messaging system. Exactly Once Delivery ensures that downstream systems will not read duplicate changed data when consuming data from the messaging system.

[0040] Exactly once processing: refers to the process in which each piece of changed data is ultimately processed only once during the process of reading data from the message system and processing streaming changed data in a stream computing system.

[0041] WAL log: The incremental change log of the database. Data is written to the incremental change log before it is actually written to the database.

[0042] Offset Topic: A collection of data in a messaging system used to store consumption point data of the WAL logs consumed by the data subscription system.

[0043] Data Topic: A collection of data in a messaging system used to store change data in the database WAL log.

[0044] In the database field, there are several solutions for delivering or synchronizing database changes. For example, a primary cluster parses changes in the WAL log and sends them to a backup cluster, which then replays the data. Another approach is to call a coprocessor to send the changes to a messaging system each time changes are written to the database. However, the first solution generates duplicate data, hindering accurate stream processing results. The second solution adds a synchronization logic to the messaging system with each write, increasing latency and reducing throughput, ultimately impacting online business operations.

[0045] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0046] Figure 1 This is a schematic diagram illustrating an application scenario for implementing a method according to an embodiment of this application. (Refer to...) Figure 1 In the database domain, the following data delivery process is involved: a data subscription system can retrieve real-time change data from the database storage system and deliver this data to a message system. Downstream stream computing systems can subscribe to the change data in the message system and perform real-time calculations on the change data according to actual needs, such as calculating the average statistic over a time period. The database can be HBase (an open-source database) or a database further developed from HBase.

[0047] A data subscription system can be an electronic device or a master-slave architecture system. In one example, refer to... Figure 1 The data subscription system can include a master node and worker nodes connected by communication. The master node is used to: assign WAL log consumption tasks to worker nodes; worker nodes are used to: respond to consumption tasks by reading the starting offset of the WAL log corresponding to the current moment from the master node; determine the real-time consumption point based on the starting offset and read the real-time change data from the database storage system; and synchronously send the real-time consumption point and real-time change data to the messaging system. There can be multiple worker nodes, thus forming a distributed data subscription system. Each node (master node or worker node) can be any device such as a physical machine or a cloud server ECS (Elastic Compute Service). The specific working principle of the data subscription system will be described in detail in subsequent embodiments. The messaging system can be a message queue, such as Kafka (a high-throughput distributed publish-subscribe messaging system).

[0048] Reference Figure 2 This application provides a data delivery method that can be applied to a data subscription system. The method may include: S201, obtaining the real-time consumption point and the real-time change data of the database from the WAL log of the database, wherein the real-time change data is the change data corresponding to the real-time consumption point in the change data of the database; S202, synchronously sending the real-time consumption point and the real-time change data to the message system.

[0049] Based on the above data delivery method, the data subscription system can synchronously send the real-time consumption points in the database's WAL log and their corresponding real-time changes to the database to the message system. The real-time consumption point data can be used to determine the location where the data subscription system consumes the database's changes. By synchronously sending the real-time consumption points and the target real-time changes to the message system, the downstream system will not read duplicate changes when consuming data from the message system, thus achieving precise one-time delivery and making subsequent stream computing more accurate.

[0050] Consumer points can be pre-stored in a point set. This set can store information such as the log file name, log file offset, and synchronization time. The log file name, offset, and time information can be stored as key-value pairs, with the log file name as the key and the offset and synchronization time as the values. The values ​​in the point set can include multiple data blocks, each containing modified data. Each data block can include a start offset and an end offset. A consumer point can include the log file name and a start offset (startOffset) for that log file. In one example, a consumer point can also include the end offset (nextOffset) of the same data block. The modified data in the data block formed by the start and end offsets is the modified data corresponding to that consumer point. The end offset of each data block can also serve as the start offset of the next data block.

[0051] Database changes can be pre-stored in a data set. The data set can store information such as row keys and change data. The row keys and change data can be stored in key-value pairs, where the row key can be used as the key and the change data can be used as the value.

[0052] Real-time consumption points can be read from the messaging system or determined based on a starting offset in the messaging system. The acquisition of real-time consumption points is achieved by pulling data from the background, which does not require a long synchronization operation during delivery, thus reducing the impact on business operations.

[0053] In one implementation, in step S202 above, synchronously sending real-time consumption points and real-time change data to the messaging system may include: submitting a transaction to the messaging system; the transaction carries the real-time consumption points and real-time change data. The messaging system has transaction characteristics, and based on these characteristics, a transaction can be submitted to the messaging system, submitting the real-time consumption points and real-time change data within the transaction. When the transaction is successfully submitted, it is guaranteed that the real-time consumption points and real-time change data are simultaneously submitted to the messaging system, thus making the consumption point data and target change data visible simultaneously when read. When the transaction fails to submit, the real-time consumption points and real-time change data are not visible simultaneously when read.

[0054] In one implementation, such as Figure 1 As shown, a data subscription system can include master nodes and worker nodes. Correspondingly, as... Figure 3 As shown, in step S201 above, obtaining the real-time consumption point and real-time change data in the database's WAL log can include: S301, the master node assigns WAL log consumption tasks to the worker nodes; S302, the worker nodes respond to the consumption tasks by reading the current offset of the WAL log from the master node; S303, the worker nodes periodically determine the real-time consumption point based on the current offset and periodically read the real-time change data from the database storage system. In step S202 above, synchronously sending the real-time consumption point and real-time change data to the message system can include: the worker nodes periodically sending the real-time consumption point and real-time change data to the message system. Through the cooperation between the master node and worker nodes, the capture and delivery of consumption points and change data can be achieved. This is a system-level data delivery scheme with minimal impact on business read / write operations on the database.

[0055] When a worker node reads real-time change data, it can parse the real-time change data through multiple parsers, and then synchronously send the real-time consumption point and the parsed real-time change data to the message system. Parsing the real-time change data can include converting the data format of the real-time change data, for example, converting the original file format of the real-time change data into a format that is easy for users to understand.

[0056] Real-time consumption points sent to the messaging system can be stored in the point set of the messaging system, and real-time change data sent to the messaging system can be stored in the data set of the messaging system.

[0057] In one implementation, such as Figure 1As shown, there can be multiple worker nodes. Correspondingly, in step S302 above, the worker node responds to the consumption task by reading the current offset of the WAL log from the master node, which may include: the worker node responds to the consumption task by executing an isolation process and reading the current offset of the WAL log from the master node; the isolation process can be used to prevent other worker nodes from sending consumption points and change data to the message system.

[0058] In a distributed architecture with multiple worker nodes, a split-brain problem may occur, where a consumption task might be simultaneously assigned to multiple worker nodes and executed on multiple nodes. To resolve this, an isolation process can be implemented within each worker node. After the isolation process is completed, the consumption point and changed data are synchronously sent to the messaging system to prevent other worker nodes from executing the consumption task. This ensures that a consumption task is executed on only one worker node. The isolation process can be implemented based on the isolation inherent in the messaging system's transactions; for example, when a new transaction starts, the old transaction is closed.

[0059] In one implementation, the data delivery method provided in this application may further include: the master node reading the consumption points of the WAL logs stored in the message system, replaying the consumption points, and continuously updating the consumption points in the log metadata of the WAL logs stored in the master node during the replay process. For example, the replayed consumption points can be used to overwrite the consumption points in the log metadata of the same WAL log, thereby enabling the maintenance of the log metadata of the WAL logs in the master node.

[0060] In one implementation, step S302 above, reading the current offset of the WAL log from the master node, may include: the worker node sending a task initialization request to the master node; the master node responding to the task initialization request, determining whether the playback position for replaying the consumption point has reached the specified position; if the playback position has reached the specified position, the master node returning a starting offset of the WAL log corresponding to the specified position to the worker node as the current offset. The specified position can be determined based on the request time of the task initialization request (e.g., the time when the worker node sends the task initialization request or the time when the master node receives the task initialization request), and the current time can be that request time.

[0061] Based on the task initialization request sent by the worker node, the request time can be determined, and then the specified position can be determined. Based on the specified position, the playback position of the consumption point can be determined. When the playback position reaches the specified position, the master node and the worker node can return the corresponding starting offset, so that the worker node can periodically determine the real-time consumption point based on the starting offset, and can also periodically read the corresponding change data based on the starting offset as real-time change data.

[0062] In step S303 above, the worker node periodically determines the real-time consumption point and reads real-time changed data from the database storage system based on the current offset. This can include: at the beginning of the period, the worker node determines the real-time consumption point based on the current offset, for example, using the current offset and its corresponding end offset as the real-time consumption point, and reads the changed data of one data block from the database storage system starting from the current offset, as the real-time changed data; in each period after the beginning of the period, the worker node updates the current offset, for example, using the start offset of the next data block after the already read data block as the new current offset, and the worker node determines the new real-time consumption point based on the new current offset, and reads the changed data of the next data block from the database storage system starting from the new current offset, as the new real-time changed data. Through this periodic offset update and data reading, the changed data of each data block of a complete log file can be read.

[0063] Figure 4 This illustrates an exemplary flowchart of the allocation of consumption tasks and data requests between master nodes and worker nodes. Figure 4 The process includes the following steps: S401, the master node assigns a WAL log consumption task to the worker nodes; S402, the worker nodes execute an isolation procedure to prevent other worker nodes from executing the consumption task; S403, the worker nodes generate a task initialization request; S404, the worker nodes send a task initialization request to the master node, requesting to read the current offset of the WAL log; S405, the master node checks whether the playback of the consumption point has reached the specified position. If yes, it executes S406; otherwise, it executes S407; S406, the master node returns the end offset of the current data block (the data block corresponding to the specified position) of the WAL log, which is also the start offset of the next data block, as the current offset, and then executes S408 and S409; S407, the master node returns a retry flag and then executes S404; S408, based on the current offset, the worker nodes periodically determine the real-time consumption point and periodically read the real-time change data. The specific methods for periodically determining the real-time consumption point and periodically reading the real-time change data can be found in the preceding description.

[0064] In one implementation, based on the consumption points, the log metadata in the master node may further include: allocation information of WAL log consumption tasks being assigned to worker nodes. Each time a WAL log consumption task is assigned to a worker node, allocation information is generated. Based on the allocation information, it can be determined which WAL log consumption tasks are assigned to which worker nodes, and thus it can be determined which WAL logs and which worker nodes have been assigned, and which logs and which worker nodes have not yet been assigned.

[0065] Correspondingly, the master node assigns WAL log consumption tasks to worker nodes, which may include: the master node determining the target log and target worker node based on the allocation information corresponding to each WAL log; assigning the consumption tasks of the target log to the target worker node; and recording the allocation information in the log metadata of the target log. Here, the target log is an unassigned WAL log, and the target worker node is an unassigned worker node. Based on the allocation information, unassigned WAL logs and unassigned worker nodes can be quickly matched, thereby quickly assigning WAL log consumption tasks. After allocation, the allocation information can be recorded in the log metadata of the WAL log to be allocated. This allocation information can be used in the next allocation to identify whether the WAL log has already been allocated and on which worker node.

[0066] In one implementation, the data delivery method provided in this application may further include: periodically persisting the log metadata in the master node and the replay position of the master node for replaying the consumption point to the coordination service system. For example, the replay of the consumption point may be periodically paused, while persisting the current log metadata and the replay position of the consumption point, and then the replay of the consumption point may be resumed.

[0067] Regularly persisting log metadata in the coordination service system can prevent the loss of log metadata after it expires. When the log metadata expires and is needed, the persisted log metadata can be read from the coordination service system. At the same time as persisting the log metadata, the replay position can also be persisted. The persistence of the replay position can ensure the correctness of the persisted log metadata.

[0068] In one embodiment, the data delivery method provided in this application may further include: when the master node restarts, reading the most recent replay position in the coordination service system as the target replay position; the master node reading the consumption point after the target replay position from the message system, and replaying the consumption point.

[0069] After periodically persisting the replay position in the coordination service system, when the master node restarts, it can read the consumption position of the WAL log stored in the point set of the message system from the most recent replay position in the coordination service system, without having to start replaying from the starting offset of the first data block of the WAL log, which can reduce replay overhead.

[0070] The coordination service system can be Zookeeper (a distributed application coordination service software). The operation of persisting log metadata and / or replaying positions in the coordination service system can be implemented through checkpoint technology.

[0071] Figure 5 The diagram illustrates the principle of an optional implementation of the data delivery method provided in this application. (Refer to...) Figure 5 For example, the master node can read the consumption points from the message system's point set through the Relay thread of the exact-once management controller and replay the read consumption points. During the replay process, the master node can continuously update the log metadata of each WAL log in the WAL log metadata set based on the consumption points. For instance, it can look up the log metadata of the WAL log corresponding to the log file name contained in the replayed consumption point in the WAL log metadata set, and use the replayed consumption point to overwrite the consumption points in the log metadata of the same log file name in the WAL log metadata set. The master node can also use the Dispatcher thread to find the log metadata of unassigned WAL logs in the WAL log metadata set and assign the consumption task of the log metadata of the unassigned WAL logs to unassigned worker nodes. During the assignment, the log file name of the log metadata to be consumed can be sent to the worker node. After the assignment is completed, the assignment information can be recorded in the log metadata of the WAL log for the next assignment query.

[0072] Reference Figure 5 For example, after receiving a consumption task, the worker node can read the current offset from the master node through the Reader thread, and then read the real-time change data corresponding to the current offset from the database storage system. The reading process can be referred to... Figure 4 For example, multiple parsers can then be used to parse real-time change data and generate real-time consumption points based on the current offset, such as... Figure 6 As shown, real-time consumption points and parsed real-time change data can be stored in a transactional buffer. Then, an asynchronous transaction commit thread can simultaneously write the real-time consumption points and parsed real-time change data into a transaction. Finally, the transaction is committed to the message system, simultaneously submitting the real-time consumption points and parsed real-time change data from the transactional buffer to the message system. The real-time consumption points submitted to the message system can be stored in a point set within the message system, and the real-time change data submitted to the message system can be stored in a data set within the message system.

[0073] Reference Figure 5 and Figure 7For example, the master node can periodically check the log metadata and the current replay position of the replay thread in the master node through the Checkpoint Manager thread of the Exact Once Management Controller, and persist the log metadata and the current replay position to Zookeeper.

[0074] Based on the same technical concept, referring to Figure 8 This application also provides a data processing method applicable to a stream computing system. The method may include: S801, acquiring changed data from a database stored in a message system; S802, performing calculations on the changed data. The changed data may be delivered to the message system using any of the data delivery methods provided in this application.

[0075] The data delivery method provided in this application embodiment can deliver changed data from the database to the message system in a precise one-time delivery manner. Based on this, the data processing method provided in this application embodiment can achieve precise one-time processing, ensuring that each piece of data in the message system is calculated only once, thereby avoiding the impact of repeated calculations on the accuracy of the calculation and ensuring the accuracy of the stream computing results.

[0076] Based on the same technical concept, embodiments of this application provide a data delivery device that can be applied to a data subscription system, such as... Figure 9 As shown, the device may include a data acquisition module 901 and a data delivery module 902.

[0077] The data acquisition module 901 can be used to acquire the real-time consumption points and the real-time change data of the database from the WAL log; the real-time change data is the change data corresponding to the real-time consumption points in the change data of the database. The data delivery module 902 can be used to synchronously send the real-time consumption points and the real-time change data to the message system.

[0078] In one implementation, the data delivery module 902 can be used to: submit a transaction to the messaging system; the transaction carries real-time consumption points and real-time change data.

[0079] Based on the same technical concept, this application also provides a data processing method that can be applied to stream computing systems, such as... Figure 10 As shown, the device may include a data acquisition module 1001 and a data calculation module 1002.

[0080] The data acquisition module 1001 can be used to acquire change data from the database stored in the messaging system; the change data can be delivered to the messaging system through the data delivery device provided in this embodiment. The data calculation module 1002 can be used to perform calculations on the change data.

[0081] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0082] Based on the same technological concept, such as Figure 11 As shown, this application embodiment provides an electronic device that can serve as a data subscription system or a stream computing system. The electronic device includes a memory 1101 and a processor 1102. The memory 1101 stores a computer program that can run on the processor 1102. When the processor 1102 executes the computer program, it implements any of the methods provided in this application embodiment. The number of memories 1101 and processors 1102 can be one or more.

[0083] The electronic device also includes:

[0084] Communication interface 1103 is used to communicate with external devices and perform data exchange and transmission.

[0085] If the memory 1101, processor 1102, and communication interface 1103 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0086] Optionally, in a specific implementation, if the memory 1101, processor 1102, and communication interface 1103 are integrated on a single chip, then the memory 1101, processor 1102, and communication interface 1103 can communicate with each other through an internal interface.

[0087] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods provided in this application.

[0088] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device equipped with the chip to perform any of the methods provided in this application.

[0089] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute any of the methods provided in the application.

[0090] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0091] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0092] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0093] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0094] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0095] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0096] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0097] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0098] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0099] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data delivery method, characterized in that, The method is applied to a data subscription system, which includes a master node and worker nodes; the method includes: The master node assigns WAL log consumption tasks to the worker nodes; The worker node responds to the consumption task by reading the current offset of the WAL log from the master node, periodically determining the real-time consumption point in the WAL log based on the current offset, and periodically reading real-time change data from the database storage system; the real-time change data is the change data corresponding to the real-time consumption point in the change data that modifies the database. The worker node periodically sends the real-time consumption point and the real-time change data to the messaging system synchronously. The real-time consumption point and the real-time change data are carried in the transaction submitted to the messaging system. When the transaction is successfully submitted, the consumption point data and the real-time change data are simultaneously visible when read. When the transaction fails to submit, the real-time consumption point and the real-time change data are simultaneously invisible when read.

2. The data delivery method according to claim 1, characterized in that, The number of working nodes is multiple; The worker node responds to the consumption task by reading the current offset of the WAL log from the master node, including: The worker node responds to the consumption task by executing the isolation process and reading the current offset of the WAL log from the master node; The isolation process is used to prevent other worker nodes from sending consumption points and change data to the messaging system.

3. The data delivery method according to claim 1, characterized in that, Also includes: The master node reads the consumption points of the WAL logs stored in the message system, replays the consumption points, and continuously updates the consumption points in the log metadata of the WAL logs stored in the master node during the replay process.

4. The data delivery method according to claim 3, characterized in that, Also includes: The step of reading the current offset of the WAL log from the master node includes: The worker node sends a task initialization request to the master node; In response to the task initialization request, the master node determines whether the playback position for replaying the consumption point has reached the specified position; the specified position is determined based on the request time of the task initialization request. When the playback position reaches the specified position, the master node returns the starting offset of the WAL log corresponding to the specified position to the worker node as the current offset.

5. The data delivery method according to claim 3, characterized in that, Also includes: The log metadata and the replay location of the master node for replaying the consumption point are periodically persisted to the coordination service system.

6. The data delivery method according to claim 5, characterized in that, Also includes: When the master node restarts, the nearest replay location in the coordination service system is read as the target replay location; The master node reads the consumption points after the target replay position from the message system and replays those consumption points.

7. The data delivery method according to claim 3, characterized in that, The log metadata also includes: allocation information of WAL log consumption tasks being assigned to worker nodes; The master node assigns the WAL log consumption task to the worker node, including: The master node determines the target log and the target worker node based on the allocation information corresponding to each WAL log; the target log is an unassigned WAL log, and the target worker node is an unassigned worker node. Assign the consumption task of the target log to the target worker node; The allocation information for this allocation is recorded in the log metadata of the target log.

8. A data processing method, characterized in that, Applied to stream computing systems, the method includes: Obtain change data from the database stored in the messaging system; the change data is delivered to the messaging system using the data delivery method described in any one of claims 1-7. The changed data is calculated.

9. A data delivery device, characterized in that, The device is applied to a data subscription system, which includes a master node and worker nodes; the device includes: The allocation module of the master node is used to allocate WAL log consumption tasks to the worker nodes. The data acquisition module of the working node is used to respond to the consumption task by reading the current offset of the WAL log from the master node, periodically determining the real-time consumption point in the WAL log based on the current offset, and periodically reading real-time change data from the database storage system; the real-time change data is the change data corresponding to the real-time consumption point in the change data of the database. The data delivery module of the working node is used to periodically send the real-time consumption point and the real-time change data to the message system synchronously. The real-time consumption point and the real-time change data are carried in the transaction submitted to the message system. When the transaction is successfully submitted, the consumption point data and the real-time change data are simultaneously visible when read. When the transaction fails to submit, the real-time consumption point and the real-time change data are simultaneously invisible when read.

10. A data processing apparatus, characterized in that, The device, used in a stream computing system, includes: The data acquisition module is used to acquire change data stored in the database of the messaging system; the change data is delivered to the messaging system through the data delivery device as described in claim 9. The data calculation module is used to calculate the changed data.

11. An electronic device, characterized in that, As a data subscription system or stream computing system, the electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of claims 1-8 when executing the computer program.

12. A data subscription system, characterized in that, include: The master node and worker nodes are connected via communication. The master node is used to: assign WAL log consumption tasks to the worker nodes; The worker node is used to: respond to the consumption task by reading the current offset of the WAL log from the master node; periodically determine the real-time consumption point in the WAL log based on the current offset, and periodically read real-time change data from the database storage system; the real-time change data is the change data corresponding to the real-time consumption point among the change data that modifies the database; and periodically send the real-time consumption point and the real-time change data synchronously to the message system; wherein the real-time consumption point and the real-time change data are carried in the transaction submitted to the message system, and when the transaction is successfully submitted, the consumption point data and the real-time change data are simultaneously visible when read; when the transaction fails to submit, the real-time consumption point and the real-time change data are simultaneously invisible when read.

13. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the method of any one of claims 1-8.

Images