Database archiving log merging method and device, server, medium and product

Abstract

The embodiment of the invention provides a database archiving log merging method and device, a server, a medium and a product. The method comprises the steps that database archiving logs of all physical nodes are collected through a stream-oriented computing framework, the multiple database archiving logs are combined, and all physical library operation data in the combined database archiving logs are partitioned; in the partitioning process, if any physical library operation data is fragmented key data, judging whether a library table primary key which is the same as any physical library operation data exists in a cache queue or not; if yes, determining the physical library operation data corresponding to the same library table primary key in the cache queue as preposed data, determining any physical library operation data as postposed data, and merging the preposed data and the postposed data to obtain logic library operation data; and outputting the logic library operation data to a final merged database archiving log through a message queue. According to the method provided by the invention, the real-time performance and the accuracy of merging the archived logs of the database are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, and particularly relates to a database archive log merging method and device, a server, a medium and a product. BACKGROUND

[0002] The database archive log is a log file recording database changes in the database. In a database with sharding, logical database operation data is split into multiple physical database operation data and distributed to multiple underlying physical nodes. Each physical node maintains its own database archive log, and the database archive logs between the physical nodes are not shared or synchronized, which causes certain difficulties in database archive log analysis of the database with sharding. Therefore, the database archive logs of each physical node need to be merged first.

[0003] At present, the prior art pulls the database archive logs of each physical node once according to a preset time period, and then merges the multiple database archive logs.

[0004] However, if a large number of database archive logs are suddenly generated within a preset time period, the large number of database archive logs may not be processed in time, thereby reducing the real-time performance and accuracy of the database archive log merging. SUMMARY

[0005] The database archive log merging method, device, server, medium and product provided by the embodiments of the present application can improve the real-time performance and accuracy of the database archive log merging.

[0006] In a first aspect, the embodiments of the present application provide a database archive log merging method, comprising: collecting database archive logs of each physical node by a log collection component of a stream computing framework to obtain a plurality of database archive logs; wherein each database archive log comprises a plurality of physical library operation data; merging the physical library operation data in the plurality of database archive logs to obtain a merged database archive log; partitioning each physical library operation data in the merged database archive log; in the partitioning process, performing the following operations on any physical library operation data: determining whether any physical library operation data is shard key data; wherein the shard key data comprises pre-data and post-data, and the library table primary key of the pre-data and the library table primary key of the post-data of the same shard key data are the same; if it is determined that any physical library operation data is shard key data, determining whether there is a library table primary key identical to any physical library operation data in a cache queue; if it is determined that there is a library table primary key identical to any physical library operation data in the cache queue, determining any physical library operation data as post-data; determining the physical library operation data corresponding to the library table primary key identical to any physical library operation data in the cache queue as pre-data; taking the pre-data from the cache queue, merging the pre-data and the post-data to convert the physical library operation data that is shard key data into logical library operation data; and delivering the logical library operation data to a message queue to output the final merged database archive log.

[0007] In a possible implementation, determining whether any physical library operation data is shard key data comprises: determining whether any physical library operation data is shard key data by determining whether the foreign key of any physical library operation data is consistent with the library table primary key of any physical library operation data; accordingly, if it is determined that any physical library operation data is shard key data, it comprises: if it is determined that the foreign key of any physical library operation data is not consistent with the library table primary key of any physical library operation data, determining that any physical library operation data is shard key data.

[0008] In a possible implementation, after delivering the logical library operation data to the message queue, the method further comprises: obtaining a timestamp of the logical library operation data; and adding an expiration label to the logical library operation data cached in the cache queue earlier than the timestamp.

[0009] In a possible implementation, after determining whether there is a library table primary key identical to any logical data in the cache queue, the method further comprises: if it is determined that there is no library table primary key identical to any logical data in the cache queue, determining any logical data as pre-data; and caching the pre-data to the cache queue according to the library table primary key of the pre-data.

[0010] In a possible implementation, after the pre-data is cached to the cache queue, the method further includes: if the post-data corresponding to the pre-data is not acquired within a preset expiration time threshold, adding an expiration label to the pre-data; and removing the pre-data from the cache queue and delivering the pre-data to the message queue.

[0011] In a possible implementation, after it is judged whether any physical library operation data is shard key data, the method further includes: if it is judged that any physical library operation data is not shard key data, adding an expiration label to any physical library operation data according to a timestamp of the physical library operation data; mapping the physical library operation data to which the expiration label is added to logical library operation data, and delivering the logical library operation data to the message queue.

[0012] In a second aspect, an embodiment of the present application provides a database archive log merging apparatus, including:

[0013] The collection module is configured to collect database archive logs of each physical node through a log collection component of a stream computing framework, to obtain a plurality of database archive logs; each database archive log includes a plurality of physical library operation data.

[0014] The merging module is configured to merge the physical library operation data in the plurality of database archive logs, to obtain a merged database archive log.

[0015] The partitioning module is configured to partition each physical library operation data in the merged database archive log; and in the partitioning process, the following operations are performed on any physical library operation data.

[0016] Specifically, the partitioning module includes:

[0017] The judging unit is configured to judge whether any physical library operation data is shard key data; the shard key data includes pre-data and post-data, and the library table primary key of the pre-data and the library table primary key of the post-data of the same shard key data are the same.

[0018] The determining unit is configured to, if it is judged that any physical library operation data is shard key data, judge whether the cache queue includes the same library table primary key as the physical library operation data.

[0019] The determining unit is configured to, if it is judged that the cache queue includes the same library table primary key as the physical library operation data, determine any physical library operation data as post-data, and determine the physical library operation data in the cache queue corresponding to the same library table primary key as the physical library operation data as pre-data.

[0020] The conversion unit is configured to take the front data from the cache queue, combine the front data and the rear data, and convert the physical library operation data which is the shard key data into logical library operation data.

[0021] The output unit is configured to deliver the logical library operation data to the message queue, and output the final combined database archive log.

[0022] In a third aspect, an embodiment of the present application provides a server, comprising: a memory, a processor;

[0023] The memory stores computer-executable instructions.

[0024] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0026] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0027] The database archive log merging method, apparatus, server, medium, and product provided in this application, through the log acquisition component of a streaming computing framework, can collect database archive logs from each physical node in real time, improving the real-time performance of database archive log merging. Since the merged database archive logs contain physical database operation data, it needs to be converted into logical database data. Each physical database operation data in the merged database archive logs is partitioned. During the partitioning process, the preceding data for the same shard key is stored in a cache queue. When the following data corresponding to the preceding data is obtained, the preceding data is retrieved from the cache queue and merged with the corresponding following data. This converts the physical database operation data of the shard key data into logical database operation data. The logical database data is then saved to a message queue, and the final merged database archive logs are output via the message queue. This ensures the order of the logical database operation data in the final merged database archive logs, thereby improving the accuracy of database archive log merging. In addition, on the one hand, the order of operation data in each physical library is ensured by using a cache queue, which further improves accuracy; on the other hand, by adding an expiration tag, the operation data of physical libraries with expired tags are no longer processed, which further improves real-time performance; and on the other hand, by setting an expiration time threshold for the cache queue, it will not occupy storage for a long time without being released, which reduces storage costs. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0029] Figure 1 This is a schematic diagram of the server structure provided in an embodiment of this application;

[0030] Figure 2 A flowchart illustrating the database archive log merging method provided in this application embodiment;

[0031] Figure 3a This is a schematic diagram illustrating the operation sequence of physical library data for different physical nodes at the same transmission rate, as provided in this embodiment.

[0032] Figure 3b This embodiment illustrates the operation sequence of physical library data for different physical nodes at different transmission rates. Figure 1 ;

[0033] Figure 3c This embodiment illustrates the operation sequence of physical library data for different physical nodes at different transmission rates. Figure 2 ;

[0034] Figure 4A schematic diagram of the structure of the database archive log merging device provided in the embodiments of this application;

[0035] Figure 5 This is a schematic diagram of the hardware structure of the server provided in an embodiment of this application.

[0036] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0038] Database archive logs are log files that record database changes. In sharded databases, logical database operations are split into multiple physical database operations, distributed across multiple underlying physical nodes. Each physical node maintains its own database archive log, and these logs are neither shared nor synchronized, making database archive log parsing in sharded databases challenging. Therefore, it's necessary to merge the database archive logs from each physical node. Currently, existing technology retrieves the database archive logs from each physical node at preset time intervals and then merges them. However, if a large number of database archive logs suddenly appear within a preset time period, it may be impossible to process them in a timely manner, thus reducing the real-time performance and accuracy of the database archive log merging process.

[0039] To address the aforementioned technical issues, this application proposes the following technical approach: By utilizing the log collection component of a streaming computing framework, database archive logs from each physical node can be collected in real time, improving the real-time performance of database archive log merging. Since the merged database archive logs contain physical database operation data, it needs to be converted into logical database data. Each physical database operation data point in the merged database archive logs is converted into logical database operation data, and this logical database data is saved to a message queue. The final merged database archive logs are then output via the message queue, ensuring the order of the logical database operation data in the final merged database archive logs, thereby improving the accuracy of database archive log merging.

[0040] Figure 1 This is a schematic diagram of the server structure provided in an embodiment of this application. Figure 1 As shown, the server includes: a log collection component 101, a processor 102, and a display device 103. It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the method for merging database archived logs. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, as can be determined according to the actual application scenario, and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0041] In the specific implementation process, the log collection component 101 can be an input / output interface or a communication interface, and can collect the database archive logs of each physical node.

[0042] The processor 102 can merge physical database operation data in multiple database archive logs based on the database archive logs of each physical node to obtain merged database archive logs; process each physical database operation data in the merged database archive logs to convert each physical database operation data into logical database operation data; and deliver the logical database operation data to a message queue to output the final merged database archive logs.

[0043] Display device 103 can be used to display the final merged database archive logs.

[0044] The display device can also be a touch screen, used to receive user commands while displaying the above content, so as to realize the operation interaction with the user.

[0045] It should be understood that the aforementioned processor can be implemented by reading instructions from memory and executing those instructions, or it can be implemented through chip circuitry.

[0046] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0047] Figure 2 A flowchart illustrating the database archive log merging method provided in this application embodiment is shown below. Figure 2 As shown, the method includes:

[0048] S201: Collect database archive logs from each physical node using the log collection component of the streaming computing framework to obtain multiple database archive logs; each database archive log contains multiple physical database operation data.

[0049] Streaming computing frameworks are processes that perform real-time computation on streaming data. Streaming data refers to a dynamic collection of data that is infinitely distributed in time and quantity. The value of this data decreases over time, therefore real-time computation with sub-second response times is essential. The task of a streaming computing framework is to continuously process streaming data and provide results in real time.

[0050] In this embodiment, the database archive logs of each physical node are a single stream of data.

[0051] In this embodiment, after a database is sharded and partitioned, the logical database operation data is split into multiple physical database operation data, distributed across multiple underlying physical nodes. Each physical node maintains its own database archive log, and the database archive logs between physical nodes are neither shared nor synchronized.

[0052] Optionally, statistical analysis of the database archive logs of each physical node reveals two phenomena: First, the unilateral timestamps are monotonically increasing. The physical database operation data in a single database archive log is sequential and monotonically increasing according to the operation record time. Second, when changes to logical database operation data involve physical operation data across multiple physical nodes, the change times on each physical node are relatively consistent. Therefore, we can use this condition to perform time anchoring between multiple physical nodes to determine the relative time displacement of physical database operation data on each physical node.

[0053] Optionally, the physical library operation data in each physical node can be preprocessed using the FlatMap operator, including: blacklist / whitelist filtering, physical logical table mapping, and data normalization transformation.

[0054] S202: Merge the physical database operation data in multiple database archive logs to obtain a merged database archive log.

[0055] In this embodiment, the Union operator is used to merge the physical library operation data of multiple physical nodes in real time.

[0056] S203: Partition the physical database operation data in the merged database archive log; during the partitioning process, perform the following operations on any physical database operation data:

[0057] In this embodiment, the Process operator is used to partition each physical database operation data in the merged database archive log, so that physical database operation data with the same primary key of the same database table are processed in the same partition.

[0058] Specifically, step S203 includes S2031 to S2035:

[0059] S2031: Determine whether any physical database operation data is shard key data; where shard key data includes pre-data and post-data, and the primary key of the database table for the pre-data and the primary key of the database table for the same shard key data are the same;

[0060] Specifically, by determining whether the foreign key of any physical database operation data is consistent with the primary key of the database table of any physical database operation data, it can be determined whether any physical database operation data is a sharding key data.

[0061] For example, after performing database sharding and table partitioning on a database, a logical database data operation Update is split into physical database operations Delete and Insert. The physical database operations Delete and Insert are shard key data, and the primary key of the database table for Delete is the same as that for Insert.

[0062] S2032: If any physical library operation data is determined to be shard key data, then determine whether there is a library table primary key in the cache queue that is the same as any physical library operation data;

[0063] Specifically, if it is determined that the foreign key of any physical database operation data is inconsistent with the primary key of any physical database operation data table, then any physical database operation data is determined to be shard key data.

[0064] In this embodiment, any physical database operation data has a foreign key. If any physical database operation data has a foreign key that is inconsistent with the primary key of the database table of any physical database operation data, it indicates that the primary key of the database table of any physical database operation data has changed, and thus any physical database operation data is determined to be shard key data.

[0065] S2033: If it is determined that there is a primary key of a database table in the cache queue that is the same as any physical database operation data, then the physical database operation data is determined as the subsequent data; the physical database operation data corresponding to the primary key of the database table in the cache queue that is the same as any physical database operation data is determined as the preceding data.

[0066] Optionally, if it is determined that there is no database table primary key in the cache queue that is the same as any logical data, then that logical data is determined as the preceding data; the preceding data is cached in the cache queue according to the database table primary key of the preceding data. Optionally, we cache the preceding data in the cache queue and wait for the subsequent data to be merged.

[0067] Optionally, if the subsequent data corresponding to the preceding data is not obtained within the preset expiration time threshold, an expiration tag is added to the preceding data; the preceding data is removed from the cache queue and delivered to the message queue.

[0068] The preset expiration time threshold is the expiration time threshold that is set in advance for the cache queue.

[0069] Optionally, if the preceding data matching the subsequent data in the cache queue contains an expired tag, it indicates that updated physical library operation data has already been passed to the downstream operator. Therefore, both the preceding and subsequent data are expired, and expired tags are added to them before they are passed to the downstream operator. If the preceding data matching the subsequent data in the cache queue does not contain an expired tag, it indicates that neither the preceding nor the subsequent data in this pair of shard key operation data has expired, and therefore a merge operation is performed.

[0070] In this embodiment, due to the different transmission rates of different physical nodes, some physical library operation data arrives earlier. Therefore, it is necessary to reasonably add expiration tags to the later-arriving physical library operation data, based on the principle that the timestamp of each physical node is monotonically increasing and the local time of shard key data changes is consistent. Figure 3a This is a schematic diagram illustrating the operation sequence of physical library data on different physical nodes at the same transmission rate, as provided in this embodiment. Figure 3a As shown, physical node A and physical node B have the same transmission rate. Figure 3b This embodiment illustrates the operation sequence of physical library data for different physical nodes at different transmission rates. Figure 1 ; Figure 3c This embodiment illustrates the operation sequence of physical library data for different physical nodes at different transmission rates. Figure 2 ;like Figure 3b As shown, according to the time anchoring principle, the relative time displacement is determined by insert1 and delete1. Since there is earlier physical library operation data U after insert1 of physical node B, an expiration tag is added to the later physical library operation data U that exists before delete1 of physical node A.

[0071] S2034: Retrieve the preceding data from the cache queue, merge the preceding data and the following data, and convert the physical library operation data of the shard key data into logical library operation data.

[0072] Optionally, obtain the timestamp of the logic library operation data; add an expiration tag to the logic library operation data cached in the cache queue earlier than the timestamp.

[0073] S2035: Deliver the logical library operation data to the message queue to output the final merged database archive log.

[0074] In this embodiment, the Sink operator delivers the logic library operation data to the message queue.

[0075] Optionally, the message queue can be Kafka.

[0076] In summary, the log collection component of the streaming computing framework can collect database archive logs from each physical node in real time, improving the real-time performance of database archive log merging. Since the merged database archive logs contain physical database operation data, they need to be converted to logical database data. Each physical database operation data point in the merged database archive logs is partitioned. During partitioning, the preceding data for the same shard key is stored in a cache queue. When the following data corresponding to the preceding data is obtained, the preceding data is retrieved from the cache queue and merged with the corresponding following data. This process converts the physical database operation data of the shard key into logical database operation data, which is then saved to a message queue. The final merged database archive logs are output via the message queue, ensuring the order of logical database operation data in the final merged database archive logs and thus improving the accuracy of database archive log merging. In addition, on the one hand, the order of operation data in each physical library is ensured by using a cache queue, which further improves accuracy; on the other hand, by adding an expiration tag, the operation data of physical libraries with expired tags are no longer processed, which further improves real-time performance; and on the other hand, by setting an expiration time threshold for the cache queue, it will not occupy storage for a long time without being released, which reduces storage costs.

[0077] Based on the above embodiments, the case where the operation data of any physical library is not shard key data is described in detail below:

[0078] S301: If it is determined that any physical database operation data is not shard key data, then add an expiration tag to any physical database operation data according to the timestamp of any physical database operation data.

[0079] Specifically, if the timestamp of any physical library operation data is later than any physical library operation data in the deferred queue, then an expired tag is added to any physical library operation data.

[0080] S302: Map any physical library operation data with an expired tag to logical library operation data, and deliver the logical library operation data to the message queue.

[0081] Optionally, the FlatMap operator can be used to map any physical library operation data with an expired tag to logical library operation data. The Sink operator then delivers the logical library operation data to a message queue.

[0082] In summary, mapping physical database operation data that is not shard key data to logical database operation data and delivering it to the message queue ensures that all physical database operation data can be converted into logical database operation data, further improving the accuracy of database archive log merging.

[0083] Figure 4 A schematic diagram of the structure of the database archive log merging device provided in the embodiments of this application is shown below. Figure 4 As shown, the database archive log merging device provided in this embodiment includes: a collection module 10, a merging module 20, and a partitioning module 30. The partitioning module 30 includes: a judgment unit 301, a determination unit 302, a confirmation unit 303, a conversion unit 304, and an output unit 305.

[0084] The acquisition module 10 is used to collect the database archive logs of each physical node through the log acquisition component of the streaming computing framework to obtain multiple database archive logs; each database archive log includes multiple physical database operation data.

[0085] The merging module 20 is used to merge physical database operation data in multiple database archive logs to obtain merged database archive logs.

[0086] Partitioning module 30 is used to partition the operation data of each physical database in the merged database archive log; during the partitioning process, the following operations are performed on any physical database operation data:

[0087] Specifically, partitioning module 30 includes:

[0088] The judgment unit 301 is used to determine whether any physical database operation data is shard key data; wherein the shard key data includes pre-data and post-data, and the primary key of the database table of the pre-data and the primary key of the database table of the same shard key data are the same.

[0089] The determination unit 302 is used to determine whether there is a primary key of a database table in the cache queue that is the same as any physical database operation data if any physical database operation data is determined to be shard key data.

[0090] The determining unit 303 is used to determine any physical library operation data as subsequent data if it is determined that there is a library table primary key in the cache queue that is the same as any physical library operation data; and to determine the physical library operation data corresponding to the library table primary key in the cache queue that is the same as any physical library operation data as preceding data.

[0091] The conversion unit 304 is used to retrieve the preceding data from the cache queue, merge the preceding data and the following data, and convert the physical library operation data of the shard key data into logical library operation data.

[0092] Output unit 305 is used to deliver logical library operation data to the message queue to output the final merged database archive log.

[0093] In one possible implementation, the determination unit 301 is specifically used to: determine whether any physical database operation data is a sharding key data by determining whether the foreign key of any physical database operation data is consistent with the primary key of the database table of any physical database operation data; correspondingly, if any physical database operation data is determined to be sharding key data, it includes: if the foreign key of any physical database operation data is determined to be inconsistent with the primary key of the database table of any physical database operation data, then the physical database operation data is determined to be sharding key data.

[0094] In one possible implementation, the database archive log merging device further includes: an expiration tag adding module for obtaining the timestamp of logical database operation data; and adding expiration tags to logical database operation data cached in the cache queue earlier than the timestamp.

[0095] In one possible implementation, the determining unit 303 is further configured to determine any logical data as preceding data if it is determined that there is no database table primary key in the cache queue that is the same as any logical data; and cache the preceding data into the cache queue according to the database table primary key of the preceding data.

[0096] In one possible implementation, the database archive log merging device further includes: a delivery module, used to add an expiration tag to the preceding data if the subsequent data corresponding to the preceding data is not obtained within a preset expiration time threshold; and to remove the preceding data from the cache queue and deliver the preceding data to the message queue.

[0097] In one possible implementation, the delivery module is further configured to, if it is determined that any physical library operation data is not shard key data, add an expiration tag to any physical library operation data according to the timestamp of any physical library operation data; map any physical library operation data with the added expiration tag to logical library operation data; and deliver the logical library operation data to the message queue.

[0098] The database archive log merging device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0099] Figure 5 This is a schematic diagram of the hardware structure of the server provided in an embodiment of this application. Figure 5As shown, the server provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0100] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0101] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0102] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0103] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0104] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0105] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0106] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0107] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0108] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0109] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0111] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0112] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0114] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for merging database archive logs, characterized in that, Applied to servers, including: The log collection component of the streaming computing framework collects database archive logs from each physical node to obtain multiple database archive logs; each database archive log contains operation data from multiple physical databases. The physical database operation data in the multiple database archive logs are merged to obtain the merged database archive logs; Partition each physical database operation data in the merged database archive log; during the partitioning process, perform the following operations on any physical database operation data: Determine whether any physical library operation data is shard key data; wherein the shard key data includes pre-data and post-data, and the primary key of the library table for the pre-data and the primary key of the library table for the same shard key data are the same; If any of the physical library operation data is determined to be shard key data, then determine whether there is a library table primary key in the cache queue that is the same as any of the physical library operation data; If it is determined that there is a database table primary key in the cache queue that is the same as any of the physical database operation data, then the physical database operation data is determined as the subsequent data; the physical database operation data corresponding to the database table primary key in the cache queue that is the same as any of the physical database operation data is determined as the preceding data. The preceding data is retrieved from the cache queue, and the preceding data and the following data are merged to convert the physical library operation data of the shard key data into logical library operation data. The logical library operation data is delivered to the message queue to output the final merged database archive log.

2. The method according to claim 1, characterized in that, The determination of whether any physical library operation data is shard key data includes: By determining whether the foreign key of any physical database operation data is consistent with the primary key of the database table of any physical database operation data, it can be determined whether any physical database operation data is a sharding key data. Accordingly, determining that any physical library operation data is shard key data includes: If it is determined that the foreign key of any physical database operation data is inconsistent with the primary key of the database table of any physical database operation data, then the physical database operation data is determined to be shard key data.

3. The method according to claim 1, characterized in that, After delivering the logic library operation data to the message queue, the process further includes: The timestamp of the logic library operation data is obtained; For logical library operation data cached in the cache queue earlier than the timestamp, add an expiration tag.

4. The method according to claim 1, characterized in that, After determining whether a primary key of a database table exists in the cache queue that is identical to any of the logical data, the method further includes: If it is determined that there is no primary key in the database table in the cache queue that is the same as any of the logical data, then any of the logical data is determined as the preceding data; The preceding data is cached in the cache queue according to the primary key of the database table.

5. The method according to claim 4, characterized in that, After caching the preceding data into the cache queue, the method further includes: If no subsequent data corresponding to the preceding data is obtained within the preset expiration time threshold, then an expiration tag is added to the preceding data. The preceding data is then removed from the cache queue and delivered to the message queue.

6. The method according to any one of claims 1-5, characterized in that, After determining whether any physical library operation data is shard key data, the method further includes: If it is determined that any physical library operation data is not shard key data, then an expiration tag is added to any physical library operation data according to the timestamp of the physical library operation data. Map any physical library operation data with an expired tag to logical library operation data, and deliver the logical library operation data to the message queue.

7. A device for merging database archive logs, characterized in that, Applied to servers, including: The acquisition module is used to collect database archive logs from each physical node through the log acquisition component of the streaming computing framework to obtain multiple database archive logs; each database archive log includes multiple physical database operation data. The merging module is used to merge the physical database operation data in the multiple database archive logs to obtain the merged database archive logs. The partitioning module is used to partition the operation data of each physical database in the merged database archive log; during the partitioning process, the following operations are performed on any physical database operation data: Specifically, the partitioning module includes: The judgment unit is used to determine whether any physical library operation data is shard key data; wherein the shard key data includes pre-data and post-data, and the primary key of the library table of the pre-data and the primary key of the library table of the same shard key data are the same; The determination unit is used to determine whether there is a primary key of a database table in the cache queue that is the same as the data operated by any physical database if the data is determined to be shard key data. The determining unit is configured to, if it is determined that there is a library table primary key in the cache queue that is the same as any of the physical library operation data, determine any of the physical library operation data as the subsequent data; and determine the physical library operation data in the cache queue corresponding to the library table primary key that is the same as any of the physical library operation data as the preceding data. A conversion unit is used to retrieve the preceding data from the cache queue, merge the preceding data and the following data, and convert the physical library operation data of the shard key data into logical library operation data. The output unit is used to deliver the logical library operation data to the message queue to output the final merged database archive log.

8. A server, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images