Monitoring method for GoldenGate data synchronization

Abstract

The invention provides a monitoring method for GoldenGate data synchronization, and relates to the technical field of GoldenGate data synchroniss.The method comprises the steps that the sending time point T1 of a preset field in a preset monitoring table is obtained every first preset duration delta T1; acquiring a time point T2 when the preset field is received; according to T1 and T2, determining time consumption delta T = T2-T1 for transmitting the preset monitoring table from the source end to the target end; if the delta T is greater than the preset data transmission time consumption threshold, determining that the current data synchronization is delayed; otherwise, determining that the current data synchronization is not delayed; according to the method and the device, whether the synchronization process has the problem of false death or relatively high time delay or not can be judged by simply adding the monitoring table, so that successful data synchronization is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of GoldenGate data synchronization, and in particular to a monitoring method for GoldenGate data synchronization. Background Art

[0002] In the field of data synchronization, there are many data synchronization software, such as Oracle GoldenGate data synchronization software; however, this type of data synchronization software only synchronizes data from the source database to the target database, is only responsible for data synchronization, and does not monitor whether the data synchronization process is normal; when the table data in the source database does not change, the files transmitted to the target end will not change. At this time, the data synchronization software cannot determine whether the synchronization process has a pseudo-death or high latency problem, resulting in data synchronization failure. Summary of the invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is: According to a first aspect of the present application, a GoldenGate data synchronization monitoring method is provided, the method comprising the following steps: S100, obtaining a sending time point T1 of a preset field in a preset monitoring table at intervals of a first preset time length ΔT1; wherein the preset monitoring table is set in a source-end database, and the preset field is added to the preset monitoring table at intervals of a second preset time length ΔT2; S200, obtaining the time point T2 at which the preset field is received; S300, based on T1 and T2, determine the time ΔT=T2-T1 required for the preset monitoring table to be transmitted from the source end to the target end; S400, if ΔT>ΔT', it is determined that the current data synchronization is delayed; otherwise, it is determined that the current data synchronization is not delayed; wherein ΔT' is a preset data transmission time threshold; ΔT' is determined by the current amount of data transmitted, the network parameters between the source and the target, and the hardware parameters of the source and the target.

[0004] Further, ΔT' is determined by the following steps: S410, obtaining the current synchronized data volume Q now , the network bandwidth D and network delay Y between the source and the target, the hardware performance level S1 of the source and the hardware performance level S2 of the target; wherein S1 is determined according to the hardware parameters of the source, and S2 is determined according to the hardware parameters of the target; S420, according to Q now , D, Y, S1 and S2, determine the current data transmission parameter vector X = (Q now , D, Y, S1, S2); S430, obtaining a standard data transmission parameter vector corresponding to each preset data transmission time threshold, so as to obtain a standard data transmission parameter vector list A=(A1, A2, ..., A i , …, A n ), i=1, 2,...,n; among them, A i A is the standard data transmission parameter vector corresponding to the i-th preset data transmission time threshold; i =(A i，Q , A i，D , A i，Y , A i，S1 , A i，S2 ); A i，Q To determine the amount of data corresponding to the i-th preset data transmission time threshold, A i，D To determine the network bandwidth corresponding to the i-th preset data transmission time threshold, A i，Y To determine the network delay corresponding to the i-th preset data transmission time threshold, A i，S1 To determine the hardware performance level of the source end corresponding to the i-th preset data transmission time threshold, A i，S2 To determine the hardware performance level of the target end corresponding to the i-th preset data transmission time threshold; S440, obtaining the similarity between X and each standard data transmission parameter vector in A, so as to obtain a similarity list η=(η1, η2, ..., η i ,…,η n ); where η i For X and A i The similarity between S450, according to η, determining the target similarity η'=MAX(η); wherein MAX() is a preset maximum value function; S460: Determine the preset data transmission time threshold corresponding to η' as ΔT'.

[0005] Further, S1 is determined by the following steps: S411, obtaining hardware parameters E1 of the source end; wherein the hardware parameters include CPU parameters, memory parameters and hard disk parameters; S412, determining S1 according to E1 and a preset source-end hardware parameter and hardware level mapping table QT; wherein QT includes a plurality of rows, each row corresponding to a source-end hardware parameter and a corresponding hardware level.

[0006] Further, S2 is determined by the following steps: S413, obtaining hardware parameters E2 of the target end; wherein the hardware parameters include CPU parameters, memory parameters and hard disk parameters; S414, determining S2 according to E2 and a preset target-side hardware parameter and hardware level mapping table QT; wherein QT includes a plurality of rows, each row corresponding to a target-side hardware parameter and a corresponding hardware level.

[0007] Furthermore, after step S400, the method further includes the following steps: S500: If it is determined that the current data synchronization is delayed, ΔT1 is adjusted to ΔT1'; wherein ΔT1'=γ×ΔT1; γ is an adjustment coefficient; γ=ΔT / ΔT'.

[0008] Furthermore, the preset monitoring table includes: ogg_checkpoint table corresponding to Goldengate software.

[0009] The present invention has at least the following beneficial effects: The GoldenGate data synchronization monitoring method of the present invention obtains the sending time point T1 of the preset field in the preset monitoring table at every first preset time interval ΔT1; obtains the time point T2 of receiving the preset field; according to T1 and T2, determines the time ΔT=T2-T1 for the preset monitoring table to be transmitted from the source end to the target end; if ΔT is greater than the preset data transmission time threshold, it is determined that the current data synchronization is delayed; otherwise, it is determined that the current data synchronization is not delayed; thereby judging whether the synchronization process has a pseudo-death or high delay problem by simply adding a monitoring table, ensuring successful data synchronization.

[0010] Furthermore, since the preset data transmission time threshold is determined by the current amount of data transmitted, the network parameters between the source and the target, and the hardware parameters of the source and the target, different transmission environments correspond to different data transmission time thresholds, thereby avoiding the occurrence of erroneous result judgments due to setting a fixed data transmission time threshold, and further improving the success rate of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A flowchart of a GoldenGate data synchronization monitoring method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by technicians in the relevant technical field without creative work are within the scope of protection of the present application.

[0014] It is noteworthy that in the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0015] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or their collections. It should also be understood that when the term "and / or" is used in the present specification and the appended claims, it refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0016] In the description of the present specification and the appended claims, if the term "if" is used, it can be interpreted as "in the case of" or "when" or "uponce" or "in response to determining" or "in response to detecting" according to the context. Similarly, the phrase "if it is determined" or "if [the described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [the described condition or event] is detected" or "in response to detecting [the described condition or event]" according to the context.

[0017] In the description of this application specification and the appended claims, the terms "first", "second", "third", etc., are only used to distinguish the description and are not to be understood as indicating or implying relative importance, nor are they used to describe a specific order or sequence.

[0018] The following will refer to Figure 1 The flowchart of the monitoring method of GoldenGate data synchronization shown in FIG. 1 introduces a monitoring method of GoldenGate data synchronization.

[0019] The GoldenGate data synchronization monitoring method may include the following steps: S100, obtaining a sending time point T1 of a preset field in a preset monitoring table at intervals of a first preset time length ΔT1; wherein the preset monitoring table is set in a source database, and the preset field is added to the preset monitoring table at intervals of a second preset time length ΔT2.

[0020] In this embodiment, Oracle GoldenGate synchronizes data from the source to the target. A preset monitoring table can be added to the source database, and a scheduled task can be added. The scheduled task is responsible for adding a preset field to the monitoring table every second preset time interval ΔT2, for example: ΔT2=1min; specifically, the preset monitoring table can be the ogg_checkpoint table, and the preset field can be last_update_ts. The preset monitoring table in the source database will be synchronized together with the data to be synchronized.

[0021] S200, obtaining the time point T2 when the preset field is received.

[0022] In this embodiment, the target end may obtain the time point T2 when the preset field is received.

[0023] S300, determining the time consumed for transmitting the preset monitoring table from the source end to the target end according to T1 and T2, ΔT=T2-T1.

[0024] In this embodiment, after determining the time point T1 for sending the preset field and the time point T2 for receiving the preset field, the time ΔT taken for the preset field to be sent from the source end to the target end can be determined.

[0025] S400, if ΔT>ΔT', it is determined that the current data synchronization is delayed; otherwise, it is determined that the current data synchronization is not delayed; wherein ΔT' is a preset data transmission time threshold; ΔT' is determined by the current amount of data transmitted, the network parameters between the source and the target, and the hardware parameters of the source and the target.

[0026] Further, ΔT' can be determined by the following steps: S410, obtaining the current synchronized data volume Q now , the network bandwidth D and network delay Y between the source and the target, the hardware performance level S1 of the source and the hardware performance level S2 of the target; wherein S1 is determined according to the hardware parameters of the source, and S2 is determined according to the hardware parameters of the target.

[0027] In this embodiment, the current synchronized data volume Q now , the network bandwidth D and network delay Y between the source and the target, when the hardware performance level S1 of the source and the hardware performance level S2 of the target are different, the corresponding data synchronization delay is also different, so the current synchronized data volume Q can be obtained now, the network bandwidth D and network delay Y between the source and the target, the hardware performance level S1 of the source and the hardware performance level S2 of the target to further determine the corresponding ΔT'.

[0028] Further, S1 can be determined by the following steps: S411, obtaining hardware parameters E1 of the source end; wherein the hardware parameters include CPU parameters, memory parameters and hard disk parameters.

[0029] In this embodiment, the CPU parameter may be a specific model of the CPU, the memory parameter may be the frequency and capacity of the memory, and the hard disk parameter may be the read and write speed of the hard disk.

[0030] S412, determining S1 according to E1 and a preset source-end hardware parameter and hardware level mapping table QT; wherein QT includes a plurality of rows, each row corresponding to a source-end hardware parameter and a corresponding hardware level.

[0031] In this embodiment, a source-end hardware parameter and hardware level mapping table QT is preset, and QT exhaustively lists different source-end hardware parameter combinations and corresponding hardware levels.

[0032] Further, S2 can be determined by the following steps: S413, obtaining hardware parameters E2 of the target end; wherein the hardware parameters include CPU parameters, memory parameters and hard disk parameters.

[0033] S414, determining S2 according to E2 and a preset target-side hardware parameter and hardware level mapping table QT; wherein QT includes a plurality of rows, each row corresponding to a target-side hardware parameter and a corresponding hardware level.

[0034] In this embodiment, the method for determining S2 is the same as the method for determining S1 described above, and will not be described in detail herein.

[0035] S420, according to Q now , D, Y, S1 and S2, determine the current data transmission parameter vector X = (Q now , D, Y, S1, S2).

[0036] S430, obtaining a standard data transmission parameter vector corresponding to each preset data transmission time threshold, so as to obtain a standard data transmission parameter vector list A=(A1, A2, ..., A i , …, A n ), i=1, 2,...,n; among them, A i A is the standard data transmission parameter vector corresponding to the i-th preset data transmission time threshold; i =(A i，Q , A i，D , A i，Y , Ai，S1 , A i，S2 ); A i，Q To determine the amount of data corresponding to the i-th preset data transmission time threshold, A i，D To determine the network bandwidth corresponding to the i-th preset data transmission time threshold, A i，Y To determine the network delay corresponding to the i-th preset data transmission time threshold, A i，S1 To determine the hardware performance level of the source end corresponding to the i-th preset data transmission time threshold, A i，S2 To determine the hardware performance level of the target end corresponding to the i-th preset data transmission time threshold.

[0037] In this embodiment, different data quantities Q can be obtained through a large number of experiments. now , the network bandwidth D and network delay Y between the source and the target, the hardware performance level S1 of the source and the hardware performance level S2 of the target, and the corresponding data synchronization delays, so as to obtain A.

[0038] S440, obtaining the similarity between X and each standard data transmission parameter vector in A, so as to obtain a similarity list η=(η1, η2, ..., η i ,…,η n ); where η i For X and A i The similarity between .

[0039] It should be noted that those skilled in the art can use existing vector similarity determination methods according to actual needs to obtain the similarity between X and each standard data transmission parameter vector in A, which will not be elaborated here.

[0040] S450, according to η, determine the target similarity η'=MAX(η); wherein MAX() is a preset maximum value function.

[0041] In this embodiment, n i The larger the value, the better the current data transmission environment is. i The corresponding data transmission environment is closer.

[0042] S460: Determine the preset data transmission time threshold corresponding to η' as ΔT'.

[0043] In this embodiment, the data transmission environment corresponding to η' is most similar to the current data transmission environment. Therefore, the preset data transmission time threshold corresponding to η' is determined to be ΔT', so that the determined ΔT' is more consistent with the current data transmission environment, improving the accuracy of judgment.

[0044] Furthermore, after step S400, the method further includes the following steps: S500: If it is determined that the current data synchronization is delayed, ΔT1 is adjusted to ΔT1'; wherein ΔT1'=γ×ΔT1; γ is an adjustment coefficient; γ=ΔT / ΔT'.

[0045] In this embodiment, after a delay occurs in data synchronization, the delay duration will gradually increase. Therefore, when it is determined that the current data synchronization is delayed, ΔT1 is lengthened to reduce the calculation frequency, reduce the computing power usage of the device, and improve the operating efficiency of the device.

[0046] The GoldenGate data synchronization monitoring method of this embodiment obtains the sending time point T1 of the preset field in the preset monitoring table at every first preset time interval ΔT1; obtains the time point T2 of receiving the preset field; according to T1 and T2, determines the time ΔT=T2-T1 for the preset monitoring table to be transmitted from the source end to the target end; if ΔT is greater than the preset data transmission time threshold, it is determined that the current data synchronization is delayed; otherwise, it is determined that the current data synchronization is not delayed; thereby judging whether the synchronization process has a pseudo-death or high delay problem by simply adding a monitoring table, ensuring successful data synchronization.

[0047] Furthermore, since the preset data transmission time threshold is determined by the current amount of data transmitted, the network parameters between the source and the target, and the hardware parameters of the source and the target, different transmission environments correspond to different data transmission time thresholds, thereby avoiding the occurrence of erroneous result judgments due to setting a fixed data transmission time threshold, and further improving the success rate of data synchronization.

[0048] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0049] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0050] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0051] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0052] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0053] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0054] An embodiment of the present invention further provides an electronic device, comprising a processor and the aforementioned non-transitory computer-readable storage medium.

[0055] The electronic device is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0056] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processor mentioned above, the at least one memory mentioned above, and a bus connecting different system components (including the memory and the processor).

[0057] The memory stores program codes, which can be executed by the processor, so that the processor executes the steps in various embodiments described in this specification.

[0058] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read only memory (ROM).

[0059] The memory may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0060] The bus may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0061] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device (e.g., routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0062] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0063] An embodiment of the present invention further provides a computer program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0064] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A monitoring method for Goldengate data synchronization, characterized in that: The method comprises the following steps: S100, obtaining a sending time point T1 of a preset field in a preset monitoring table at intervals of a first preset time length ΔT1; wherein the preset monitoring table is set in a source-end database, and the preset field is added to the preset monitoring table at intervals of a second preset time length ΔT2; S200, obtaining the time point T2 at which the preset field is received; S300, based on T1 and T2, determine the time ΔT=T2-T1 required for the preset monitoring table to be transmitted from the source end to the target end; S400, if ΔT>ΔT', it is determined that the current data synchronization is delayed; otherwise, it is determined that the current data synchronization is not delayed; wherein ΔT' is a preset data transmission time threshold; ΔT' is determined by the current amount of data transmitted, the network parameters between the source and the target, and the hardware parameters of the source and the target.

2. The Goldengate data synchronization monitoring method according to claim 1, characterized in that: ΔT' is determined by the following steps: S410, obtaining the current synchronized data volume Q now , the network bandwidth D and network delay Y between the source and the target, the hardware performance level S1 of the source and the hardware performance level S2 of the target; Among them, S1 is determined according to the hardware parameters of the source end, and S2 is determined according to the hardware parameters of the target end; S420, according to Q now , D, Y, S1 and S2, determine the current data transmission parameter vector X = (Q now , D, Y, S1, S2); S430, obtaining a standard data transmission parameter vector corresponding to each preset data transmission time threshold, so as to obtain a standard data transmission parameter vector list A=(A1, A2, ..., A i , …, A n ), i=1, 2,...,n; among them, A i A is the standard data transmission parameter vector corresponding to the i-th preset data transmission time threshold; i =(A i，Q , A i，D , A i，Y , A i，S1 , A i，S2 ); A i，Q To determine the amount of data corresponding to the i-th preset data transmission time threshold, A i，D To determine the network bandwidth corresponding to the i-th preset data transmission time threshold, A i，Y To determine the network delay corresponding to the i-th preset data transmission time threshold, A i，S1 To determine the hardware performance level of the source end corresponding to the i-th preset data transmission time threshold, A i，S2 To determine the hardware performance level of the target end corresponding to the i-th preset data transmission time threshold; S440, obtaining the similarity between X and each standard data transmission parameter vector in A, so as to obtain a similarity list η=(η1, η2, ..., η i ,…,η n ); where η i For X and A i The similarity between S450, according to η, determining the target similarity η'=MAX(η); wherein MAX() is a preset maximum value function; S460: Determine the preset data transmission time threshold corresponding to η' as ΔT'.

3. The Goldengate data synchronization monitoring method according to claim 2, characterized in that: S1 is determined by the following steps: S411, obtaining hardware parameters E1 of the source end; wherein the hardware parameters include CPU parameters, memory parameters and hard disk parameters; S412, determining S1 according to E1 and a preset source-end hardware parameter and hardware level mapping table QT; wherein QT includes a plurality of rows, each row corresponding to a source-end hardware parameter and a corresponding hardware level.

4. The Goldengate data synchronization monitoring method according to claim 2, characterized in that: S2 is determined by the following steps: S413, obtaining hardware parameters E2 of the target end; wherein the hardware parameters include CPU parameters, memory parameters and hard disk parameters; S414, determining S2 according to E2 and a preset target-side hardware parameter and hardware level mapping table QT; wherein QT includes a plurality of rows, each row corresponding to a target-side hardware parameter and a corresponding hardware level.

5. The Goldengate data synchronization monitoring method according to claim 1, characterized in that: After step S400, the method further comprises the following steps: S500: If it is determined that the current data synchronization is delayed, ΔT1 is adjusted to ΔT1'; wherein ΔT1'=γ×ΔT1; γ is an adjustment coefficient; γ=ΔT / ΔT'.

6. The Goldengate data synchronization monitoring method according to claim 1, characterized in that: The preset monitoring tables include: ogg_checkpoint table corresponding to the Goldengate software.

Images