Data processing methods, devices, equipment and media

Abstract

This invention discloses a data processing method, apparatus, device, and medium, relating to the field of automatic programming technology. The method includes: reading at least one data change message streamed by a message middleware; wherein each data change message carries transaction information to which the corresponding data change operation belongs; processing a matching wide data table based on the relevant data of all data change messages corresponding to the target transaction; and synchronizing data in a downstream storage system based on the wide data table. This technical solution processes the changed data captured in the database at the granularity of the transaction to which the database change operation belongs, ensuring that the downstream storage system does not experience situations where some data is updated while others are not, thus guaranteeing data consistency in the downstream storage system at all times.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of automatic programming technology, and in particular to a data processing method, apparatus, device and medium. Background Technology

[0002] In streaming data processing, CDC (Change Data Capture) is typically used to retrieve committed change data from the database and send this change data to the downstream storage system for its use.

[0003] There is usually a correlation between multiple changes to data in a database. However, since each change is sent to the downstream storage system separately in the current technology, the downstream storage system may experience partial updates and partial omissions when these multiple changes are not fully synchronized. This can lead to data inconsistencies in the downstream storage system at certain times. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, device, and medium to ensure data consistency in downstream storage systems at all times.

[0005] In a first aspect, embodiments of the present invention provide a data processing method, including:

[0006] Read at least one data change message received by the message middleware in a streaming manner; wherein each data change message carries transaction information to which the corresponding data change operation belongs;

[0007] Process the matching wide data table based on the relevant data of all data change messages corresponding to the target transaction;

[0008] Data synchronization is performed on the downstream storage system based on the aforementioned wide data table.

[0009] Secondly, embodiments of the present invention also provide a data processing apparatus, comprising:

[0010] The data change message reading module is used to read at least one data change message received by the message middleware in a streaming manner; wherein, each data change message carries the transaction information to which the corresponding data change operation belongs;

[0011] The wide data table processing module is used to process and match the wide data table based on the relevant data of all data change messages corresponding to the target transaction.

[0012] The data synchronization module is used to synchronize data with the downstream storage system based on the data wide table.

[0013] Thirdly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data processing method as described in any embodiment of the present invention.

[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in any embodiment of the present invention.

[0015] The technical solution provided by this invention processes a wide data table based on all the change data corresponding to the target transaction, and then synchronizes the data of the downstream data storage system based on the wide data table. That is, the change data captured in the database is processed at the granularity of the transaction to which the database change operation belongs, so that there will be no situation in the downstream storage system where some data is updated and some data is not updated, thus ensuring the data consistency of the downstream storage system at all times. Attached Figure Description

[0016] Figure 1 This is a flowchart of a data processing method according to Embodiment 1 of the present invention;

[0017] Figure 2 This is a flowchart of a data processing method according to Embodiment 2 of the present invention;

[0018] Figure 3 This is a schematic diagram of the structure of a data processing device according to Embodiment 3 of the present invention;

[0019] Figure 4 This is a schematic diagram of the hardware structure of a computer device according to Embodiment 4 of the present invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0021] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it may also have additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0022] Example 1

[0023] Figure 1 This is a flowchart of a data processing method provided in Embodiment 1 of the present invention, which is applicable to the processing of streaming data. The method can be executed by the data processing device provided in the embodiment of the present invention. The device can be implemented in software and / or hardware and is generally integrated into a computer device.

[0024] like Figure 1 As shown, the data processing method provided in this embodiment specifically includes:

[0025] S110. Read at least one data change message received by the message middleware in a streaming manner; wherein each data change message carries transaction information to which the corresponding data change operation belongs.

[0026] Message middleware is a supporting software system that provides synchronous or asynchronous, reliable message transmission for application systems in a network environment, based on queue and message passing technologies. In this embodiment, the message middleware can be Kafka, RocketMQ, etc.

[0027] A data change message refers to a message corresponding to a data change operation in the database. These data change operations include INSERT, DELETE, and UPDATE operations.

[0028] In this embodiment, the data change message carries the transaction information to which the corresponding data change operation belongs. A transaction refers to a sequence of database operations that manipulate various types of data. The operations within this sequence are either all executed or none are executed; these database operations form an indivisible whole.

[0029] Transaction information refers to information that describes a transaction. Optionally, transaction information includes the transaction ID and the total number of transaction messages. The transaction ID is information that uniquely identifies a transaction; the total number of transaction messages refers to the total number of messages generated based on all the operations included in the transaction, that is, the total number of operations included in the transaction.

[0030] It is worth noting that the start and end operations included in a transaction can be counted in the total number of transaction messages (in which case corresponding data change messages will be generated for the start and end operations respectively), or they can be excluded from the total number of transaction messages (in which case corresponding data change messages will not be generated for the start and end operations respectively). The specific setting can be determined according to the actual project; this embodiment does not impose any specific limitations on this. The following explanation uses the example of excluding the start and end operations from the total number of transaction messages for illustration.

[0031] Accordingly, in this embodiment, the data change message carries the transaction number and the total number of transaction messages. Optionally, the data change message carries the transaction number and the total number of transaction messages in the message header.

[0032] As an optional implementation of this embodiment, before reading at least one data change message received streaming from the message middleware, the method may further include: obtaining the submitted data change operations from the database, generating a data change message based on each data change operation, and streaming it to the message middleware.

[0033] Retrieve committed data change operations from the database, and assemble each data change operation into a data change message according to the information of the transaction to which it belongs. The data change message carries transaction information, such as the transaction number and the total number of transaction messages in the message header. Then, send the data change message containing the data information to the message middleware for subsequent streaming processing.

[0034] S120. Process the matching wide table based on the relevant data of all data change messages corresponding to the target transaction.

[0035] A target transaction refers to any sequence of database operations.

[0036] The data related to the data change message refers to the data used to synchronize data with the downstream storage system, which corresponds to the data change message.

[0037] A wide table, literally speaking, is a database table with a large number of fields. It usually refers to a database table that links together metrics, dimensions, and attributes related to a business theme.

[0038] When synchronizing data in the downstream storage system for a target transaction, first determine all data change messages corresponding to the target transaction, then read the relevant data corresponding to these data change messages, and finally process the corresponding wide data table based on these relevant data.

[0039] As an optional implementation of this embodiment, before processing the matching wide data table based on the relevant data of all data change messages corresponding to the target transaction, the following may also be included:

[0040] Based on the transaction number and the total number of transaction messages carried by each data change message, determine all data change messages corresponding to the target transaction, and obtain the relevant data of all data change messages.

[0041] The transaction numbers carried by each data change message read in the message middleware are counted. If the number of data change messages corresponding to the same transaction number is consistent with the total number of corresponding transaction messages, it can be determined that all the changed data corresponding to that transaction can be obtained. That is, all the data change messages corresponding to that transaction have arrived. Then, the relevant data of all data change messages corresponding to that transaction number can be obtained for processing the corresponding wide data table.

[0042] As an optional implementation of this embodiment, after the message reading middleware receives at least one data change message in a streaming manner, it may further include: streaming the relevant data of the at least one data change message into the target database.

[0043] The target database refers to any database that can be used as a data storage middleware; in this embodiment, it could be, for example, HBase.

[0044] After reading one or more data change messages in the message middleware, they can be written to the target database first, so as to temporarily store the panoramic data corresponding to the data change message in the target database.

[0045] Optionally, writing the at least one data change message into the target database in a streaming manner can specifically involve adding a transaction number field to the data table in the target database on the basis of isomorphism when writing the relevant data of the at least one data change message into the target database in a streaming manner.

[0046] The transaction number field identifies the transaction to which the changed data belongs; its value is the transaction number. Adding a transaction number field to a homogeneous table in the target database enables isolation between transactions.

[0047] Furthermore, after writing the relevant data of the at least one data change message into the target database in a streaming manner, the message characteristics of the at least one data change message can also be written into the transaction buffer pool corresponding to the target database in a streaming manner; wherein, the message characteristics include: transaction number, total number of transaction messages, entity, and message primary key.

[0048] In this embodiment, after reading one or more data change messages in the message middleware, they can first be written to the target database, and then the message characteristics of these data change messages can be written to the transaction buffer pool.

[0049] The transaction buffer pool is implemented based on the target database. When the target database is HBase, the transaction buffer pool is implemented based on HBase and is used to cache the message characteristics of data change messages read from the message middleware to ensure that all data change messages included in the transaction are synchronized to the downstream data storage system after they have all arrived.

[0050] Optionally, the data structure in the transaction buffer pool includes a transaction ID (TID), a total number of transaction messages (Tcount), an entity, and a message primary key. The transaction ID (TID) and the total number of transaction messages (Tcount) are the transaction information carried in the data change message, the entity is the data table name corresponding to the data change message, and the message primary key is the primary key value of the data change message.

[0051] Furthermore, based on the message characteristics of multiple data change messages recorded in the transaction buffer pool, all data change messages corresponding to the target transaction can be determined, and relevant data of all data change messages can be obtained.

[0052] As an optional implementation, based on the transaction number carried by each data change message and the total number of transaction messages, all data change messages corresponding to the target transaction are determined, and the relevant data of all data change messages is obtained. Specifically, this can be done as follows:

[0053] Based on the message characteristics of the data change messages recorded in the transaction buffer pool, determine all data change messages corresponding to the target transaction; and read the relevant data of all data change messages from the target database based on the message characteristics of the data change messages.

[0054] The system determines whether the number of messages corresponding to the transaction number of the target transaction recorded in the transaction buffer pool is consistent with the total number of transaction messages of the target transaction. If they are consistent, all data change messages carrying the transaction number of the target transaction can be used as all data change messages corresponding to the target transaction. Then, based on the entity and message primary key corresponding to these data change messages recorded in the transaction buffer pool, the relevant data corresponding to the data change information can be read from the target database.

[0055] Because the target database adds a transaction number field to the isomorphic table, it achieves isolation between transactions. Therefore, when reading relevant data corresponding to data change information in the target database, it can be filtered by transaction number, and the latest version of the data with a transaction number less than that of the target transaction can be read. This is to prevent subsequent transactions from modifying the data corresponding to the target transaction and compromising the integrity of the target transaction.

[0056] Furthermore, as an optional implementation, the message features may also include: a timestamp and a root message primary key for associating with the master data table.

[0057] The root message primary key refers to the message primary key of the master table associated with the data table corresponding to the data change message. If the data table corresponding to the data change message is the master table, then its message primary key and the root message primary key are the same.

[0058] After determining all data change messages corresponding to the target transaction, relevant data corresponding to the data change information can be read from the target database based on the entity, message primary key, root message primary key, and timestamp recorded in the transaction buffer pool that correspond to these data change messages.

[0059] By combining the primary key of the message with the primary key of the root message, the relevant data corresponding to the data change information can be read from the target database, and the relevant data in the corresponding main data table can be queried at the same time, thereby improving the data query efficiency.

[0060] S130. Synchronize data with the downstream storage system according to the data wide table.

[0061] The data wide table is read and written to the downstream storage system for data synchronization. Therefore, when querying data through the distributed search engine (Elasticsearch), there will be no situation where some data is updated while others are not, ensuring data consistency in the downstream storage system at all times.

[0062] Based on the above technical solution, as an optional implementation, before processing the matching wide table according to the relevant data of all data change messages corresponding to the target transaction, it may further include: determining that all data change messages corresponding to all other transactions before the target transaction have been synchronized to the downstream storage system.

[0063] All other transactions preceding the target transaction refer to all transactions that occurred before the target transaction. If the transaction numbers are ordered chronologically, with earlier transactions having smaller transaction numbers, then all other transactions preceding the target transaction refer to all other transactions with transaction numbers less than the target transaction's transaction number.

[0064] In this embodiment, before processing the matching data wide table based on the relevant data of all data change messages corresponding to the target transaction, in addition to ensuring that all data change messages corresponding to the target transaction have been received, it should also be ensured that all data change messages corresponding to all other transactions preceding the target transaction have been synchronized to the downstream storage system. This can be either that the synchronization of all data change messages corresponding to all other transactions preceding the target transaction has begun to the downstream storage system, or that the synchronization of all data change messages corresponding to all other transactions preceding the target transaction has been completed. "Beginning synchronization to the downstream storage system" can refer to the operation of reading the data wide table and starting to write data to the downstream storage system after processing the data wide table.

[0065] For example, a variable can be set to describe the maximum transaction number of the transaction currently synchronized with the downstream storage system. Furthermore, the variable value can be used to determine whether all data change messages for all other transactions preceding the target transaction have been synchronized with the downstream storage system.

[0066] As an optional implementation, after synchronizing data with the downstream storage system according to the data wide table, the method may further include: deleting the message characteristics of all data change messages corresponding to the target transaction in the transaction buffer pool.

[0067] In this implementation, the transaction buffer pool will no longer record the transaction numbers of transactions that have completed data synchronization with the downstream storage system.

[0068] Furthermore, when writing the message characteristics of the data change message into the transaction buffer pool, determining that all data change messages corresponding to all other transactions preceding the target transaction have been synchronized to the downstream storage system can specifically involve:

[0069] If the message characteristics recorded in the transaction buffer pool do not include the transaction numbers of all other transactions preceding the target transaction, then it is determined that all data change messages corresponding to all other transactions preceding the target transaction have been synchronized to the downstream storage system.

[0070] When determining whether all data change messages corresponding to all other transactions preceding the target transaction have been synchronized to the downstream storage system, it can be determined whether the message characteristics recorded in the transaction buffer pool also include the transaction numbers of all other transactions preceding the target transaction. If they are included, it is determined that all data change messages corresponding to all other transactions preceding the target transaction have not been synchronized to the downstream storage system; if they are not included, it is determined that all data change messages corresponding to all other transactions preceding the target transaction have been synchronized to the downstream storage system.

[0071] The technical solution provided by this invention processes a wide data table based on all the change data corresponding to the target transaction, and then synchronizes the data of the downstream data storage system based on the wide data table. That is, the change data captured in the database is processed at the granularity of the transaction to which the database change operation belongs, so that there will be no situation in the downstream storage system where some data is updated and some data is not updated, thus ensuring the data consistency of the downstream storage system at all times.

[0072] Example 2

[0073] Figure 2 This is a flowchart of a data processing method provided in Embodiment 2 of the present invention. This embodiment provides a specific implementation method based on the foregoing embodiments. Specifically, the message middleware can be Kafka, and the target database can be HBase.

[0074] like Figure 2 As shown, the data processing method provided in this embodiment specifically includes:

[0075] S210. Use CDC to retrieve committed data change operations from the database.

[0076] Retrieving changed data from a database requires capturing these changes. The easiest way to do this is through triggers, but this can negatively impact database performance and increase overhead. A better and more universal approach is to export the database logs, such as MySQL's Binlog or Oracle's GoldenGate.

[0077] S220. Each data change operation is combined into a data change message, and the data change message carries the transaction number and the total number of transaction messages.

[0078] Each data change message can be preceded by a message header, which includes the transaction number and the total number of transaction messages.

[0079] S230. The data change message is streamed to Kafka.

[0080] The transaction information contained in a Kafka message includes the transaction number and the total number of transaction messages.

[0081] S240. Read at least one data change message from Kafka and write the at least one data message into HBase.

[0082] When writing at least one data message to HBase, a transaction number field (the value of the transaction number field is the transaction number carried by the data change message) is added to the HBase homogeneous data table to perform transaction isolation.

[0083] S250. After completing the HBase write, write the message characteristics of the at least one data change message into the transaction buffer pool.

[0084] The transaction buffer pool is implemented based on HBase. It is set up to ensure that all data change messages contained in the target transaction are received before being written to Elasticsearch.

[0085] The message characteristics include: transaction number, total number of transaction messages, entity, and message primary key. Correspondingly, the transaction buffer pool data structure also includes transaction number, total number of transaction messages, entity, and message primary key.

[0086] Furthermore, the message characteristics may also include: a timestamp and a root message primary key for associating with the master data table. Correspondingly, the transaction buffer pool data structure also includes a timestamp and a root message primary key.

[0087] S260. Determine whether all data change messages corresponding to the target transaction have arrived in the transaction buffer pool. If not, execute S260; if yes, execute S270.

[0088] When determining whether all data change messages corresponding to the target transaction have arrived in the transaction buffer pool, it can be specifically determined whether the number of messages corresponding to the transaction number of the target transaction in the transaction buffer pool is equal to the total number of transaction messages of the target transaction.

[0089] S270. Determine whether all data change messages corresponding to all other transactions before the target transaction have arrived. If not, execute S270; if yes, execute S280.

[0090] S280. Read HBase data based on the entity and message primary key corresponding to the transaction number of the target transaction recorded in the transaction buffer pool, and process the corresponding wide data table.

[0091] Specifically, when reading HBase data based on the message primary key, filtering is performed according to the transaction number, and the latest version of the data that is less than or equal to the transaction number is read. This is to prevent subsequent transactions from modifying the data being assembled (i.e., the data being assembled for the target transaction) and compromising the data integrity of the target transaction, thus achieving a similar effect to transaction isolation based on the transaction number.

[0092] S290. Read the data from the wide data table and write it to the downstream storage system.

[0093] S2100. After completing the data writing to the downstream storage system, delete the message characteristics corresponding to the target transaction recorded in the transaction buffer pool.

[0094] Correspondingly, when determining whether all data change messages corresponding to all other transactions preceding the target transaction have arrived, this can be done based on the transaction number of the data change message recorded in the transaction buffer pool.

[0095] For any details not explained in this embodiment, please refer to the foregoing embodiments; they will not be repeated here.

[0096] In the above technical solution, the changed data is obtained at the transaction level. Each data change message generated from the changed data contains the transaction number and the total number of transaction messages. Then, the transaction cache pool is used to perform data synchronization processing with the downstream storage system at the transaction level during streaming data processing, thereby ensuring the data consistency of the downstream storage system.

[0097] Example 3

[0098] Figure 3 This is a schematic diagram of a data processing device provided in Embodiment 3 of the present invention. It is applicable to the processing of streaming data. The device can be implemented in software and / or hardware and is generally integrated into a computer device.

[0099] like Figure 3 As shown, the data processing device specifically includes: a data change message reading module 310, a data wide table processing module 320, and a data synchronization module 330. Among them,

[0100] The data change message reading module 310 is used to read at least one data change message received by the message middleware in a streaming manner; wherein, each data change message carries transaction information to which the corresponding data change operation belongs;

[0101] The data wide table processing module 320 is used to process the matching data wide table based on the relevant data of all data change messages corresponding to the target transaction;

[0102] The data synchronization module 330 is used to synchronize data with the downstream storage system based on the data wide table.

[0103] The technical solution provided by this invention processes a wide data table based on all the change data corresponding to the target transaction, and then synchronizes the data of the downstream data storage system based on the wide data table. That is, the change data captured in the database is processed at the granularity of the transaction to which the database change operation belongs, so that there will be no situation in the downstream storage system where some data is updated and some data is not updated, thus ensuring the data consistency of the downstream storage system at all times.

[0104] Optionally, the data change message carries the transaction number and the total number of transaction messages.

[0105] Optionally, the above apparatus further includes: a change data acquisition module, used to determine all data change messages corresponding to the target transaction based on the transaction number carried by each data change message and the total number of transaction messages before processing the matching wide table according to the relevant data of all data change messages corresponding to the target transaction, and to acquire the relevant data of all data change messages.

[0106] Optionally, the above apparatus further includes: a synchronized change confirmation module, used to determine, before processing the matching data wide table based on the relevant data of all data change messages corresponding to the target transaction, that all data change messages corresponding to all other transactions prior to the target transaction have been synchronized to the downstream storage system.

[0107] Optionally, the above apparatus further includes: a data change message generation and sending module, configured to retrieve submitted data change operations from the database before the message reading middleware receives at least one data change message in a streaming manner, and generate a data change message according to each data change operation, and stream the message to the message middleware.

[0108] Optionally, the above apparatus further includes: a change data temporary storage module, used to stream-write the relevant data of the at least one data change message into the target database after the message reading middleware has received at least one data change message in a streaming manner.

[0109] Optionally, the change data temporary storage module is specifically used to add a transaction number field to the data table in the target database on the basis of isomorphism when writing the relevant data of the at least one data change message into the target database in a streaming manner.

[0110] Optionally, the above apparatus further includes: a transaction temporary storage statistics module, used to stream the message features of the at least one data change message into a transaction buffer pool corresponding to the target database after the relevant data of the at least one data change message is streamed into the target database;

[0111] The message characteristics include: transaction number, total number of transaction messages, entity, and message primary key.

[0112] Optionally, the change data acquisition module is specifically used to determine all data change messages corresponding to the target transaction based on the message characteristics of the data change messages recorded in the transaction buffer pool; and to read the relevant data of all data change messages in the target database based on the message characteristics of the data change messages.

[0113] Optionally, the message features may also include: a timestamp and a root message primary key used to associate the data master table.

[0114] Optionally, the above apparatus further includes: a transaction temporary storage update module, used to delete the message characteristics of all data change messages corresponding to the target transaction in the transaction buffer pool after synchronizing data with the downstream storage system according to the data wide table.

[0115] Optionally, the synchronized change confirmation module is specifically used to determine that all data change messages corresponding to all other transactions before the target transaction have been synchronized to the downstream storage system if the message characteristics recorded in the transaction buffer pool do not include the transaction numbers of all other transactions before the target transaction.

[0116] The above-described data processing device can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data processing method.

[0117] Since the data processing apparatus described above is capable of executing the data processing method in the embodiments of the present invention, those skilled in the art can understand the specific implementation methods and various variations of the data processing apparatus in this embodiment based on the data processing method described in the embodiments of the present invention. Therefore, how the data processing apparatus implements the data processing method in the embodiments of the present invention will not be described in detail here. Any apparatus used by those skilled in the art to implement the data processing method in the embodiments of the present invention falls within the scope of protection of this application.

[0118] Example 4

[0119] Figure 4 This is a schematic diagram of the hardware structure of a computer device provided in Embodiment 4 of the present invention. Figure 4 A block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present invention is shown. Figure 4 The computer device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0120] like Figure 4 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0121] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0122] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0123] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0124] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0125] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although... Figure 4 As not shown, it can be used in conjunction with computer device 12 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0126] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a data processing method provided in an embodiment of the present invention.

[0127] That is, when the processing unit executes the program, it performs the following: reading at least one data change message received by the message middleware in a streaming manner; wherein each data change message carries transaction information to which the corresponding data change operation belongs; processing a matching data wide table based on the relevant data of all data change messages corresponding to the target transaction; and synchronizing data with the downstream storage system based on the data wide table.

[0128] Example 5

[0129] Embodiment 5 of the present invention provides a computer-readable storage medium storing a computer program. When executed by a processor, the program implements a data processing method as provided in all embodiments of the present application: that is, when the program is executed by the processor, it implements: reading at least one data change message received in a streaming manner by a message middleware; wherein each data change message carries transaction information to which the corresponding data change operation belongs; processing a matching data wide table based on the relevant data of all data change messages corresponding to the target transaction; and synchronizing data with the downstream storage system based on the data wide table.

[0130] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0131] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0132] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0133] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​(such as Java, Smalltalk, C++) and conventional procedural programming languages ​​(such as the "C" language or similar programming languages). The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized by, The method comprises: reading at least one data change message received by a message middleware in a streaming manner; wherein each data change message carries transaction information to which a corresponding data change operation belongs; processing a matched data wide table according to related data of all data change messages corresponding to a target transaction; performing data synchronization on a downstream storage system according to the data wide table; after reading at least one data change message received by the message middleware in a streaming manner, the method further comprises: writing related data of the at least one data change message to a target database in a streaming manner, and adding a transaction number field to a data table in the target database on a homogeneous basis; wherein the transaction number field is used to identify transaction information to which changed data belongs; after writing related data of the at least one data change message to the target database in a streaming manner, the method further comprises: writing message features of the at least one data change message to a transaction buffer pool corresponding to the target database in a streaming manner; wherein the message features include a transaction number, a total number of transaction messages, an entity, and a message primary key; before processing a matched data wide table according to related data of all data change messages corresponding to a target transaction, the method comprises: determining all data change messages corresponding to the target transaction according to a transaction number and a total number of transaction messages carried by each data change message, and obtaining related data of the all data change messages, including: determining all data change messages corresponding to the target transaction according to message features of the data change messages recorded in the transaction buffer pool; and reading related data of the all data change messages in the target database according to the message features of the data change messages.

2. The method of claim 1, wherein, The data change message carries a transaction number and a total number of transaction messages.

3. The method of claim 1, wherein, before processing a matched data wide table according to related data of all data change messages corresponding to a target transaction, the method further comprises: determining that all data change messages corresponding to all other transactions before the target transaction have been synchronized to the downstream storage system.

4. The method according to claim 1 or 3, characterized in that, before reading at least one data change message received by the message middleware in a streaming manner, the method further comprises: obtaining committed data change operations in a database, and generating a data change message according to each data change operation and sending the data change message to the message middleware in a streaming manner.

5. The method of claim 1, wherein, The message features further include a timestamp and a root message primary key used to associate a data master table.

6. The method of claim 1, wherein, after performing data synchronization on a downstream storage system according to the data wide table, the method further comprises: deleting message features of all data change messages corresponding to the target transaction in the transaction buffer pool.

7. The method of claim 6, wherein, determining that all data change messages corresponding to all other transactions before the target transaction have been synchronized to the downstream storage system, including: if transaction numbers of all other transactions before the target transaction are not included in message features recorded in the transaction buffer pool, it is determined that all data change messages corresponding to all other transactions before the target transaction have been synchronized to the downstream storage system.

8. A data processing apparatus, characterized by, The method comprises: The data change message reading module is configured to read at least one data change message received by the message middleware in a streaming manner, wherein each data change message carries transaction information to which a corresponding data change operation belongs. The data wide table processing module is configured to process a matched data wide table according to related data of all data change messages corresponding to a target transaction. The data synchronization module is configured to perform data synchronization on a downstream storage system according to the data wide table. The apparatus further comprises: The change data staging module is configured to write related data of the at least one data change message to a target database in a streaming manner, and add a transaction number field to a data table in the target database on a homogeneous basis, wherein the transaction number field is used to identify transaction information to which the change data belongs. The transaction staging statistics module is configured to, after writing the related data of the at least one data change message to the target database in a streaming manner, write message features of the at least one data change message to a transaction buffer pool corresponding to the target database in a streaming manner, wherein the message features include a transaction number, a total number of transaction messages, an entity, and a message primary key. The change data obtaining module is configured to, before processing a matched data wide table according to related data of all data change messages corresponding to a target transaction, determine all data change messages corresponding to the target transaction according to message features of the data change messages recorded in the transaction buffer pool, and read related data of the all data change messages in the target database according to the message features of the data change messages.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1-7 when executing the program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-7.

Images