Distributed database data synchronization monitoring method and related equipment thereof

Abstract

The invention relates to a distributed database data synchronization monitoring method and related equipment thereof, and the method comprises the steps: in a distributed database data synchronization process, obtaining a first SQL statement and preset alarm information; analyzing the first SQL statement to obtain first statement information corresponding to the first SQL statement; according to the first statement information and preset alarm information, determining whether statement alarm needs to be performed on the first SQL statement; determining a target alarm strategy corresponding to the first SQL statement according to the first statement information when determining that statement alarm needs to be carried out on the first SQL statement; and according to the target alarm strategy, executing a target alarm operation corresponding to the target alarm strategy on the first SQL statement. According to the method, the influence of execution failure of the first SQL statement on the whole data synchronization process is reduced, and the data synchronization efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a monitoring method for distributed database data synchronization and related equipment. Background Art

[0002] In the financial field, there is a widespread demand for distributed database data synchronization. However, the requirements for distributed database data synchronization are also more stringent, requiring real-time synchronization capabilities. For this real-time synchronization scenario, reasonable and necessary monitoring is a must-have capability. The most important thing about data synchronization is data. If data anomalies during data synchronization are not discovered in time, it will be very difficult to troubleshoot later. The synchronization of incremental data is a long process. How to monitor data in a timely and accurate manner, ensure data consistency, and discover data anomalies in a timely manner is very important. Summary of the invention

[0003] In view of this, in order to solve the above-mentioned technical problems or part of the technical problems, an embodiment of the present application provides a monitoring method for data synchronization of a distributed database and related equipment.

[0004] In a first aspect, the present application provides a method for monitoring data synchronization of a distributed database, comprising:

[0005] During the distributed database data synchronization process, the first SQL statement and preset alarm information are obtained;

[0006] Parsing the first SQL statement to obtain first statement information corresponding to the first SQL statement;

[0007] Determining whether a statement alarm is required for the first SQL statement according to the first statement information and the preset alarm information;

[0008] When it is determined that a statement alarm needs to be issued for the first SQL statement, determining a target alarm strategy corresponding to the first SQL statement according to the first statement information;

[0009] According to the target alarm strategy, a target alarm operation corresponding to the target alarm strategy is performed on the first SQL statement.

[0010] In an optional implementation, when determining that a statement alarm needs to be issued for the first SQL statement, determining a target alarm strategy corresponding to the first SQL statement according to the first statement information includes:

[0011] When it is determined that a statement alarm needs to be issued for the first SQL statement, obtaining a preset alarm filtering condition corresponding to the first SQL statement;

[0012] Determine whether the first SQL statement meets the preset alarm filtering condition according to the first statement information;

[0013] When it is determined that the first SQL statement satisfies the preset alarm filtering condition, determining that the target alarm strategy corresponding to the first SQL statement is the first alarm strategy;

[0014] When it is determined that the first SQL statement does not satisfy the preset alarm filtering condition, determining that the target alarm strategy corresponding to the first SQL statement is a second alarm strategy;

[0015] The performing, according to the target alarm strategy, a target alarm operation corresponding to the target alarm strategy on the first SQL statement includes:

[0016] When the target alarm strategy is the first alarm strategy, no alarm prompt is given for the first SQL statement;

[0017] When the target alarm strategy is the second alarm strategy, an alarm prompt is issued for the first SQL statement.

[0018] In an optional implementation, when the first SQL statement is a DDL statement, the first statement information includes a first shard ID corresponding to the DDL statement, a first library name corresponding to the DDL statement, and a first table name corresponding to the DDL statement;

[0019] The preset alarm filtering condition includes that the first shard ID is consistent with the shard ID corresponding to the second SQL statement, the first library name is consistent with the library name corresponding to the second SQL statement, and the first table name is consistent with the table name corresponding to the second SQL statement, and the second SQL statement is used to indicate the SQL statement for which an alarm prompt has been issued; or,

[0020] The preset alarm filtering condition includes that the first library name belongs to a preset library name set, and / or the first table name belongs to a preset table name set, the preset library names in the preset library name set are used to indicate library names for which no alarm is required, and the preset table names in the preset table name set are used to indicate table names for which no alarm is required.

[0021] In an optional implementation, when the first SQL statement is a DML statement, the first statement information includes a second table name corresponding to the DML statement;

[0022] When it is determined that a statement alarm needs to be issued for the first SQL statement, obtaining a preset alarm filtering condition corresponding to the first SQL statement includes:

[0023] When it is determined that a statement alarm needs to be issued for the first SQL statement, a preset alarm threshold corresponding to the statement alarm is obtained from the preset alarm information;

[0024] Determine a target number of abnormal data appearing in a target table corresponding to the second table name;

[0025] When the target quantity is greater than the preset alarm threshold, a preset alarm filtering condition corresponding to the first SQL statement is obtained.

[0026] In an optional implementation, when the first SQL statement is a DML statement, the first statement information further includes a second shard ID corresponding to the DML statement and a second library name corresponding to the DML statement;

[0027] The preset alarm filtering condition includes that the second shard ID is consistent with the shard ID corresponding to the second SQL statement, the second library name is consistent with the library name corresponding to the second SQL statement, and the second table name is consistent with the table name corresponding to the second SQL statement, and the second SQL statement is used to indicate the SQL statement for which an alarm prompt has been issued; or,

[0028] The preset alarm filtering condition includes that the second library name belongs to a preset library name set, and / or the second table name belongs to a preset table name set, the preset library names in the preset library name set are used to indicate library names for which no alarm is required, and the preset table names in the preset table name set are used to indicate table names for which no alarm is required.

[0029] In an optional implementation, when the target alarm strategy is the second alarm strategy, issuing an alarm prompt for the first SQL statement includes:

[0030] When the target alarm strategy is the second alarm strategy, according to the target statement type corresponding to the first SQL statement, a target alarm terminal corresponding to the target statement type is determined, the target statement type includes a DDL statement and a DML statement, and the target alarm terminal is used to indicate a terminal for issuing an alarm prompt for the first SQL statement;

[0031] Determine target alarm information corresponding to the first SQL statement;

[0032] The target alarm information is pushed to the target alarm terminal to issue an alarm prompt for the first SQL statement.

[0033] In an optional implementation, after executing the step of pushing the target alarm information to the target alarm terminal to issue an alarm prompt for the first SQL statement, the method further includes:

[0034] Receiving target feedback information corresponding to the preset alarm information sent by the target alarm terminal;

[0035] When the target feedback information is used to modify the preset alarm information, modify the preset alarm information according to the target feedback information to complete the update of the preset alarm information;

[0036] After completing the updating of the preset alarm information, the updated preset alarm information is used to execute the step of obtaining the first SQL statement and the preset alarm information during the distributed database data synchronization process.

[0037] In a second aspect, the present application provides a monitoring device for distributed database data synchronization, including:

[0038] An acquisition module, used to acquire the first SQL statement and preset alarm information during the distributed database data synchronization process;

[0039] A parsing module, used for parsing the first SQL statement to obtain first statement information corresponding to the first SQL statement;

[0040] A determination module, configured to determine whether a statement warning is required for the first SQL statement according to the first statement information and the preset warning information;

[0041] The determination module is configured to determine a target alarm strategy corresponding to the first SQL statement according to the first statement information when it is determined that a statement alarm needs to be issued for the first SQL statement;

[0042] An alarm module is used to perform a target alarm operation corresponding to the target alarm strategy on the first SQL statement according to the target alarm strategy.

[0043] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory, wherein the processor is used to execute a distributed database data synchronization monitoring program stored in the memory to implement the distributed database data synchronization monitoring method as described above.

[0044] In a fourth aspect, the present application further provides a storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the distributed database data synchronization monitoring method as described above.

[0045] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art. The method provided by the embodiment of the present application includes: obtaining a first SQL statement and preset alarm information during the data synchronization process of a distributed database; parsing the first SQL statement to obtain the first statement information corresponding to the first SQL statement; determining whether a statement alarm is required for the first SQL statement based on the first statement information and the preset alarm information; when it is determined that a statement alarm is required for the first SQL statement, determining the target alarm strategy corresponding to the first SQL statement based on the first statement information; and executing the target alarm operation corresponding to the target alarm strategy for the first SQL statement based on the target alarm strategy. In the above manner, the embodiment of the present application monitors the first SQL statement during the data synchronization process of a distributed database, so that when it is determined that an alarm is required for the first SQL statement, the target alarm strategy corresponding to the first SQL statement is determined, thereby executing the target alarm operation corresponding to the target alarm strategy to realize the alarm of the first SQL statement, thereby reducing the impact of the failure of the execution of the first SQL statement on the entire data synchronization process and improving the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0049] Figure 1 A schematic diagram of a flow chart of a distributed database data synchronization monitoring method provided in an embodiment of the present application;

[0050] Figure 2 A flowchart of another distributed database data synchronization monitoring method provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of distributed database data synchronization provided in an embodiment of the present application;

[0052] Figure 4A schematic diagram of the structure of a monitoring device for distributed database data synchronization provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0054] In the above attached figure:

[0055] 10. Acquisition module; 20. Analysis module; 30. Determination module; 40. Alarm module;

[0056] 500, electronic device; 501, processor; 502, memory; 5021, operating system; 5022, application; 503, user interface; 504, network interface; 505, bus system. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0058] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0059] refer to Figure 1 , Figure 1 A flow chart of a distributed database data synchronization monitoring method provided in an embodiment of the present application. A distributed database data synchronization monitoring method provided in the present application comprises the following steps:

[0060] S101: During the distributed database data synchronization process, a first SQL statement and preset alarm information are obtained.

[0061] In this embodiment, the scenario involved in this embodiment is a scenario of data synchronization from a source database to a target database, and the source database is a distributed database. Figure 3The schematic diagram of distributed database data synchronization shown in the figure includes an acquisition module, a middleware, an alarm module and a playback module. The acquisition module is used to incrementally obtain the first SQL statement of the distributed database in real time based on accessing the source database; the middleware is used for storage, and message queues are commonly used as middleware, such as Kafka; the alarm module is used to issue statement alarms for the first SQL statement obtained; the playback module is used to consume the middleware and import it into the target database after processing.

[0062] Among them, the preset alarm information can be obtained through a distributed broadcast system, such as Zookeeper, to obtain dynamic configuration items, and the configuration information in the dynamic configuration items is the preset alarm information. The preset alarm information may include the alarm IP and the alarm source, etc. After obtaining the preset alarm information, the preset alarm information can be used to determine whether a statement alarm is required for the first SQL statement, so that when it is determined that a statement alarm is required for the first SQL statement, the target alarm strategy corresponding to the first SQL statement can be determined, and then the target alarm operation corresponding to the target alarm strategy is performed on the first SQL statement.

[0063] S102: Parse the first SQL statement to obtain first statement information corresponding to the first SQL statement.

[0064] In this embodiment, the first SQL statement can be parsed using an existing parsing tool to obtain relevant information (i.e., the first statement) of the first SQL statement operating the distributed database, for example, the information may include a shard ID, a database name, a table name, and relevant execution information of the first SQL statement, etc. After the first SQL statement is parsed to obtain the first statement information corresponding to the first SQL statement, the first statement information and the preset alarm information may be used to determine whether a statement alarm needs to be issued for the first SQL statement.

[0065] S103: Determine whether a statement warning is required for the first SQL statement according to the first statement information and preset warning information.

[0066] In this embodiment, after obtaining the first statement information and preset alarm information corresponding to the first SQL statement, it can be determined whether the first statement information meets the alarm condition included in the preset alarm information, so that when it is determined that the first statement information meets the alarm condition included in the preset alarm information, it is determined that a statement alarm needs to be performed on the first SQL statement.

[0067] Among them, when it is determined that a statement alarm is required for the first SQL statement, if an alarm prompt is directly issued for the first SQL statement, the user experience may be affected due to frequent alarms and the absence of need for alarms. Therefore, when it is determined that a statement alarm is required for the first SQL statement, the target alarm strategy corresponding to the first SQL statement is determined according to the first statement information corresponding to the first SQL statement, so as to use the determined target alarm strategy to execute the corresponding target alarm strategy to avoid the above situation, thereby improving the user experience.

[0068] S104: When it is determined that a statement alarm needs to be issued for the first SQL statement, a target alarm strategy corresponding to the first SQL statement is determined according to the first statement information.

[0069] In this embodiment, the target alarm strategy includes a first alarm strategy and a second alarm strategy. The first alarm strategy is not to issue an alarm prompt for the first SQL statement, and the second alarm strategy is to issue an alarm prompt for the first SQL statement.

[0070] S105: According to the target alarm strategy, execute the target alarm operation corresponding to the target alarm strategy on the first SQL statement.

[0071] In this embodiment, when the target alarm strategy is the first alarm strategy, the target alarm operation corresponding to the first alarm strategy can be executed on the first SQL statement; when the target alarm strategy is the second alarm strategy, the target alarm strategy corresponding to the first alarm strategy can be executed on the first SQL statement. In this way, it is possible to realize monitoring during the data synchronization process of the distributed database, and avoid frequent alarms or unnecessary alarms, thereby improving the user experience.

[0072] The present embodiment provides a distributed database data synchronization monitoring method. During the distributed database data synchronization process, a first SQL statement is monitored to determine a target alarm strategy corresponding to the first SQL statement when it is determined that an alarm needs to be issued for the first SQL statement, thereby executing a target alarm operation corresponding to the target alarm strategy to implement an alarm for the first SQL statement, thereby reducing the impact of the failure to execute the first SQL statement on the entire data synchronization process and improving the efficiency of data synchronization.

[0073] refer to Figure 2 , Figure 2 A flow chart of another distributed database data synchronization monitoring method provided in an embodiment of the present application. A distributed database data synchronization monitoring method provided in an embodiment of the present application comprises the following steps:

[0074] S201: During the distributed database data synchronization process, a first SQL statement and preset alarm information are obtained.

[0075] S202: Parse the first SQL statement to obtain first statement information corresponding to the first SQL statement.

[0076] S203: Determine whether a statement warning is required for the first SQL statement according to the first statement information and preset warning information.

[0077] Regarding the above-mentioned steps S201 to S203, step S201 is consistent with the above-mentioned step S101, step S202 is consistent with the above-mentioned step S102, and step S203 is consistent with the above-mentioned step S103. For details, please refer to the above-mentioned steps S101 to S103, which will not be repeated in this embodiment.

[0078] S204: When it is determined that a statement alarm needs to be issued for the first SQL statement, a preset alarm filtering condition corresponding to the first SQL statement is obtained.

[0079] In this embodiment, the first SQL statement has a target statement type, and the target statement type includes a DDL statement and a DDM statement. The preset alarm filtering condition includes a preset alarm filtering condition corresponding to the DDL statement and a preset alarm filtering condition corresponding to the DDM statement. The preset alarm filtering condition is used to determine whether to issue an alarm prompt for the first SQL statement when it is determined that a statement alarm is required for the first SQL statement, so as to avoid frequent alarms or situations where no alarm is required.

[0080] Among them, when the target statement type corresponding to the first SQL statement is a DDL statement, the first statement information corresponding to the DDL statement includes the first shard ID corresponding to the DDL statement, the first library name corresponding to the DDL statement, and the first table name corresponding to the DDL statement. The preset alarm filtering conditions corresponding to the DDL statement include that the first shard ID is consistent with the shard ID corresponding to the second SQL statement, the first library name is consistent with the library name corresponding to the second SQL statement, and the first table name is consistent with the table name corresponding to the second SQL statement, and the second SQL statement is used to indicate the SQL statement for which an alarm has been issued. The preset alarm filtering conditions corresponding to the DDL statement also include that the first library name belongs to a preset library name set, and / or the first table name belongs to a preset table name set, the preset library name in the preset library name set is used to indicate the library name that does not need to be alarmed, and the preset table name in the preset table name set is used to indicate the table name that does not need to be alarmed.

[0081] Specifically, when it is determined that a statement alarm is required for a DDL statement, the preset alarm filter condition corresponding to the pre-set DDL statement can be obtained to determine whether to issue an alarm prompt for the DDL statement through the preset alarm filter condition. For DDL statements, one is alarm merging. In order to suppress frequent alarms, the alarm keywords are used for merging. The shard ID + library name + table name are selected as the merge keyword, so that the merge keyword can be used to implement alarm filtering; the other is alarm shielding. For alarms of unnecessary libraries and tables, they can be shielded by shielding keywords to avoid misunderstandings.

[0082] More specifically, when the target statement type corresponding to the first SQL statement is a DML statement, the first statement information corresponding to the DML statement includes the second shard ID corresponding to the DML statement, the second library name corresponding to the DML statement, and the second table name corresponding to the DML statement. In the above, when it is determined that a statement alarm is required for the first SQL statement in step S204, a preset alarm filter condition corresponding to the first SQL statement is obtained, including:

[0083] When it is determined that a statement alarm needs to be issued for the first SQL statement, a preset alarm threshold corresponding to the statement alarm is obtained from the preset alarm information;

[0084] Determine a target number of abnormal data appearing in a target table corresponding to the second table name;

[0085] When the target quantity is greater than the preset alarm threshold, a preset alarm filtering condition corresponding to the first SQL statement is obtained.

[0086] In this embodiment, since the amount of data operated by the DML statement is large, in order to avoid frequent alarms, when the target statement type corresponding to the first SQL statement is a DML statement, the preset alarm threshold corresponding to the statement alarm is obtained from the preset alarm information, and the target number of abnormal data appearing in the target table corresponding to the second table name is determined. When the target number is less than or equal to the preset alarm threshold, the DML statement can be directly alarmed. When the target number is greater than the preset alarm threshold, the preset alarm filter condition corresponding to the first SQL statement is obtained to determine whether the DML statement needs to be alarmed through the preset alarm filter condition. Among them, the specific value of the preset alarm threshold can be set according to actual needs, and the specific value of the preset alarm threshold is not limited in this embodiment.

[0087] Specifically, when the target statement type corresponding to the first SQL statement is a DML statement, the preset alarm filtering condition corresponding to the DML statement includes that the second shard ID is consistent with the shard ID corresponding to the second SQL, the second library name is consistent with the library name corresponding to the second SQL statement, and the second table name is consistent with the table name corresponding to the second SQL statement, and the second SQL statement is used to indicate the SQL statement for which an alarm has been issued. The preset alarm filtering condition corresponding to the DML statement also includes that the second library name belongs to a preset library name set, and / or the second table name belongs to a preset table name set, the preset library name in the preset library name set is used to indicate the library name for which no alarm is required, and the preset table name in the preset table name set is used to indicate the table name for which no alarm is required.

[0088] More specifically, when the preset alarm filter condition corresponding to the preset DML statement is obtained, the preset alarm filter condition is used to determine whether to issue an alarm prompt for the DDL statement. For DML statements, one is alarm merging. In order to suppress frequent alarms, the alarm keywords are used for merging. The shard ID + library name + table name are selected as the merge keyword, so that the merge keyword can be used to implement alarm filtering; the other is alarm shielding. For alarms of unnecessary libraries and tables, the shielding keywords can be used to shield them to avoid misunderstandings.

[0089] S205: Determine whether the first SQL statement meets a preset alarm filtering condition according to the first statement information.

[0090] S206: When it is determined that the first SQL statement meets the preset alarm filtering condition, determine that the target alarm policy corresponding to the first SQL statement is the first alarm policy.

[0091] S207: When the target alarm strategy is the first alarm strategy, no alarm prompt is issued for the first SQL statement.

[0092] S208: When it is determined that the first SQL statement does not satisfy the preset alarm filtering condition, determine that the target alarm policy corresponding to the first SQL statement is a second alarm policy.

[0093] S209: When the target alarm strategy is the second alarm strategy, an alarm prompt is issued for the first SQL statement.

[0094] With respect to the above-mentioned steps S205 to S209, after obtaining the preset alarm filtering conditions corresponding to the DDL statement, the first statement information corresponding to the DDL statement can be matched with the preset alarm filtering conditions. When the first statement information corresponding to the DDL statement successfully matches the preset alarm filtering conditions, it indicates that there is no need to issue an alarm prompt for the DDL statement. At this time, the target alarm strategy corresponding to the DDL statement is determined as the first alarm strategy; when the first statement information corresponding to the DDL statement fails to match the preset alarm filtering conditions, it indicates that there is a need to issue an alarm prompt for the DDL statement. At this time, the target alarm strategy corresponding to the DDL statement is determined as the second alarm strategy. Similarly, after obtaining the preset alarm filtering conditions corresponding to the DML statement, the first statement information corresponding to the DML statement can be matched with the preset alarm filtering conditions. When the first statement information corresponding to the DML statement successfully matches the preset alarm filtering conditions, it indicates that there is no need to issue an alarm prompt for the DML statement. At this time, the target alarm strategy corresponding to the DML statement is determined as the first alarm strategy; when the first statement information corresponding to the DML statement fails to match the preset alarm filtering conditions, it indicates that there is a need to issue an alarm prompt for the DML statement. At this time, the target alarm strategy corresponding to the DML statement is determined as the second alarm strategy.

[0095] In this embodiment, in step S209, when the target alarm strategy is the second alarm strategy, an alarm prompt is issued for the first SQL statement, specifically including:

[0096] When the target alarm strategy is the second alarm strategy, according to the target statement type corresponding to the first SQL statement, a target alarm terminal corresponding to the target statement type is determined, the target statement type includes a DDL statement and a DML statement, and the target alarm terminal is a terminal for alarming the first SQL statement;

[0097] Determine the target alarm information corresponding to the first SQL statement;

[0098] The target alarm information is pushed to the target alarm terminal to issue an alarm prompt for the first SQL statement.

[0099] Among them, in order to process alarms of different target statement types in a targeted manner, the corresponding relationship between different target statement types and alarm terminals is set in advance, so that when the first SQL statement that needs to be alarmed is determined, the target alarm information corresponding to the first SQL statement is generated, and the target alarm terminal corresponding to the target statement type is determined, so as to push the target alarm information to the target alarm terminal, so as to realize the targeted alarm prompt of the first SQL statement and the subsequent targeted alarm processing.

[0100] In this embodiment, after executing the step of pushing the target alarm information to the target alarm terminal to issue an alarm prompt for the first SQL statement, the distributed database data synchronization monitoring method provided by this embodiment further includes the following steps:

[0101] Receive target feedback information corresponding to the preset alarm information sent by the target alarm terminal;

[0102] When the target feedback information is used to correct the preset alarm information, the preset alarm information is modified according to the target feedback information to complete the update of the preset alarm information;

[0103] After the preset alarm information is updated, the updated preset alarm information is used to execute the step of obtaining the preset alarm information of the first SQL statement set during the distributed database data synchronization process.

[0104] Among them, in order to avoid the inaccurate setting of the preset alarm information and affect the subsequent data synchronization alarm monitoring, in this embodiment, after the target alarm information is pushed to the corresponding target alarm terminal, the preset alarm information can be displayed on the target alarm terminal, so that the relevant personnel can modify the preset alarm information when determining that the preset alarm information is inaccurate. After the target alarm information and the preset alarm information are displayed on the target alarm terminal, the target feedback information corresponding to the preset alarm information sent by the target alarm terminal is received. The target feedback information can be used to indicate that the preset alarm information does not need to be corrected, and can also indicate that the preset alarm information needs to be corrected. When the target feedback information indicates that the preset alarm information does not need to be corrected, it is directly returned to execute step S201. When the target feedback information indicates that the preset alarm information needs to be corrected, the preset alarm information is corrected according to the target feedback information to complete the update of the preset alarm information, and then the updated preset alarm information is used to return to execute step S201.

[0105] In this embodiment, it should be noted that the relevant information of the entire monitoring process can be stored through the middleware Kalfa for consumption by the web platform. Optionally, it can be injected into the bean management through JXM and connected to a special monitoring framework (such as prometheus); optionally, it can be stored in the persistence layer database and provide a Rest interface to facilitate viewing historical alarms or analyzing alarm information.

[0106] The present embodiment provides a distributed database data synchronization monitoring method. During the distributed database data synchronization process, a first SQL statement is monitored to determine a target alarm strategy corresponding to the first SQL statement when it is determined that an alarm needs to be issued for the first SQL statement, thereby executing a target alarm operation corresponding to the target alarm strategy to implement an alarm for the first SQL statement, thereby reducing the impact of the failure to execute the first SQL statement on the entire data synchronization process and improving the efficiency of data synchronization.

[0107] refer to Figure 4 , Figure 4 A schematic diagram of the structure of a monitoring device for data synchronization of a distributed database provided in an embodiment of the present application. A monitoring device for data synchronization of a distributed database provided in an embodiment of the present application comprises: an acquisition module 10, a parsing module 20, a determination module 30 and an alarm module 40. The acquisition module 10 is used to acquire a first SQL statement and preset alarm information during the distributed database data synchronization process; the parsing module 20 is used to parse the first SQL statement to obtain the first statement information corresponding to the first SQL statement; the determination module 30 is used to determine whether a statement alarm is required for the first SQL statement based on the first statement information and the preset alarm information; the determination module 30 is used to determine the target alarm strategy corresponding to the first SQL statement based on the first statement information when it is determined that a statement alarm is required for the first SQL statement; the alarm module 40 is used to perform a target alarm operation corresponding to the target alarm strategy on the first SQL statement based on the target alarm strategy.

[0108] In this embodiment, the determination module 30 is further configured to:

[0109] When it is determined that a statement alarm needs to be issued for the first SQL statement, obtaining a preset alarm filtering condition corresponding to the first SQL statement;

[0110] Determine whether the first SQL statement meets the preset alarm filtering condition according to the first statement information;

[0111] When it is determined that the first SQL statement satisfies the preset alarm filtering condition, determining that the target alarm strategy corresponding to the first SQL statement is the first alarm strategy;

[0112] When it is determined that the first SQL statement does not satisfy the preset alarm filtering condition, it is determined that the target alarm policy corresponding to the first SQL statement is a second alarm policy.

[0113] In this embodiment, the alarm module 40 is further used for:

[0114] When the target alarm strategy is the first alarm strategy, no alarm prompt is given for the first SQL statement;

[0115] When the target alarm strategy is the second alarm strategy, an alarm prompt is issued for the first SQL statement.

[0116] In this embodiment, when the first SQL statement is a DDL statement, the first statement information includes a first shard ID corresponding to the DDL statement, a first library name corresponding to the DDL statement, and a first table name corresponding to the DDL statement;

[0117] The preset alarm filtering condition includes that the first shard ID is consistent with the shard ID corresponding to the second SQL statement, the first library name is consistent with the library name corresponding to the second SQL statement, and the first table name is consistent with the table name corresponding to the second SQL statement, and the second SQL statement is used to indicate the SQL statement for which an alarm prompt has been issued; or,

[0118] The preset alarm filtering condition includes that the first library name belongs to a preset library name set, and / or the first table name belongs to a preset table name set, the preset library names in the preset library name set are used to indicate library names for which no alarm is required, and the preset table names in the preset table name set are used to indicate table names for which no alarm is required.

[0119] In this embodiment, when the first SQL statement is a DML statement, the first statement information includes a second table name corresponding to the DML statement.

[0120] In this embodiment, the determination module 30 is further configured to:

[0121] When it is determined that a statement alarm needs to be issued for the first SQL statement, a preset alarm threshold corresponding to the statement alarm is obtained from the preset alarm information;

[0122] Determine a target number of abnormal data appearing in a target table corresponding to the second table name;

[0123] When the target quantity is greater than the preset alarm threshold, a preset alarm filtering condition corresponding to the first SQL statement is obtained.

[0124] In this embodiment, when the first SQL statement is a DML statement, the first statement information further includes a second shard ID corresponding to the DML statement and a second library name corresponding to the DML statement;

[0125] The preset alarm filtering condition includes that the second shard ID is consistent with the shard ID corresponding to the second SQL statement, the second library name is consistent with the library name corresponding to the second SQL statement, and the second table name is consistent with the table name corresponding to the second SQL statement, and the second SQL statement is used to indicate the SQL statement for which an alarm prompt has been issued; or,

[0126] The preset alarm filtering condition includes that the second library name belongs to a preset library name set, and / or the second table name belongs to a preset table name set, the preset library names in the preset library name set are used to indicate library names for which no alarm is required, and the preset table names in the preset table name set are used to indicate table names for which no alarm is required.

[0127] In this embodiment, the alarm module 40 is further used for:

[0128] When the target alarm strategy is the second alarm strategy, according to the target statement type corresponding to the first SQL statement, a target alarm terminal corresponding to the target statement type is determined, the target statement type includes a DDL statement and a DML statement, and the target alarm terminal is used to indicate a terminal for issuing an alarm prompt for the first SQL statement;

[0129] Determine target alarm information corresponding to the first SQL statement;

[0130] The target alarm information is pushed to the target alarm terminal to issue an alarm prompt for the first SQL statement.

[0131] In this embodiment, the acquisition module 10 is further used for:

[0132] Receiving target feedback information corresponding to the preset alarm information sent by the target alarm terminal;

[0133] When the target feedback information is used to modify the preset alarm information, modify the preset alarm information according to the target feedback information to complete the update of the preset alarm information;

[0134] After the updating of the preset alarm information is completed, the updated preset alarm information is used to obtain the first SQL statement and the preset alarm information during the distributed database data synchronization process.

[0135] The present embodiment provides a monitoring device for distributed database data synchronization. During the distributed database data synchronization process, the first SQL statement is monitored to determine the target alarm strategy corresponding to the first SQL statement when it is determined that an alarm needs to be issued for the first SQL statement, thereby executing the target alarm operation corresponding to the target alarm strategy to implement the alarm of the first SQL statement, thereby reducing the impact of the failure of the execution of the first SQL statement on the entire data synchronization process and improving the efficiency of data synchronization.

[0136] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5The electronic device 500 shown includes: at least one processor 501, a memory 502, at least one network interface 504 and other user interfaces 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 505 is not shown in FIG. Figure 5 Various buses are labeled as bus system 505 .

[0137] The user interface 503 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0138] It can be understood that the memory 502 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0139] In some implementations, the memory 502 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 5021 and an application program 5022 .

[0140] Among them, the operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 5022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present application can be included in the application 5022.

[0141] In the embodiment of the present application, by calling the program or instruction stored in the memory 502, specifically, the program or instruction stored in the application 5022, the processor 501 is used to execute the method steps provided by each method embodiment.

[0142] The method disclosed in the above embodiment of the present application can be applied to the processor 501, or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 501. The above processor 501 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and completes the steps of the above method in combination with its hardware.

[0143] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0144] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0145] The electronic device provided in this embodiment may be Figure 5 The electronic device shown in FIG. 1 may perform the following steps: Figure 1 and Figure 2 All steps of the monitoring method for distributed database data synchronization in Figure 1 and Figure 2 For details, please refer to the technical effect of the distributed database data synchronization monitoring method shown in Figure 1 and Figure 2 For the sake of brevity, the relevant description is not repeated here.

[0146] The embodiment of the present application also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0147] When one or more programs in the storage medium can be executed by one or more processors, the distributed database data synchronization monitoring method executed on the distributed database data synchronization monitoring device side is implemented.

[0148] The processor is used to execute the distributed database data synchronization monitoring program stored in the memory to implement the following steps of the distributed database data synchronization monitoring method executed on the distributed database data synchronization monitoring device side.

[0149] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0150] It should be noted that the phrases "one implementation", "an embodiment", "an exemplary embodiment", "some embodiments", etc. mentioned in the specification indicate that the described embodiments may include certain features, structures or characteristics, but not every embodiment may include the certain features, structures or characteristics. In addition, such phrases do not necessarily refer to the same embodiment. In addition, when describing certain features, structures or characteristics in conjunction with an embodiment, it is within the knowledge of those skilled in the art to implement such features, structures or characteristics in conjunction with other embodiments, whether explicitly or not explicitly described.

[0151] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed database data synchronization monitoring method, characterized in that: include: During the distributed database data synchronization process, the first SQL statement and preset alarm information are obtained; Parsing the first SQL statement to obtain first statement information corresponding to the first SQL statement; Determining whether a statement alarm is required for the first SQL statement according to the first statement information and the preset alarm information; When it is determined that a statement alarm needs to be issued for the first SQL statement, determining a target alarm strategy corresponding to the first SQL statement according to the first statement information; According to the target alarm strategy, a target alarm operation corresponding to the target alarm strategy is performed on the first SQL statement.

2. The method according to claim 1, characterized in that When it is determined that a statement alarm needs to be performed on the first SQL statement, determining a target alarm strategy corresponding to the first SQL statement according to the first statement information includes: When it is determined that a statement alarm needs to be issued for the first SQL statement, obtaining a preset alarm filtering condition corresponding to the first SQL statement; Determine whether the first SQL statement meets the preset alarm filtering condition according to the first statement information; When it is determined that the first SQL statement satisfies the preset alarm filtering condition, determining that the target alarm strategy corresponding to the first SQL statement is the first alarm strategy; When it is determined that the first SQL statement does not satisfy the preset alarm filtering condition, determining that the target alarm strategy corresponding to the first SQL statement is a second alarm strategy; The performing, according to the target alarm strategy, a target alarm operation corresponding to the target alarm strategy on the first SQL statement includes: When the target alarm strategy is the first alarm strategy, no alarm prompt is given for the first SQL statement; When the target alarm strategy is the second alarm strategy, an alarm prompt is issued for the first SQL statement.

3. The method according to claim 2, characterized in that When the first SQL statement is a DDL statement, the first statement information includes a first shard ID corresponding to the DDL statement, a first library name corresponding to the DDL statement, and a first table name corresponding to the DDL statement; The preset alarm filtering condition includes that the first shard ID is consistent with the shard ID corresponding to the second SQL statement, the first library name is consistent with the library name corresponding to the second SQL statement, and the first table name is consistent with the table name corresponding to the second SQL statement, and the second SQL statement is used to indicate the SQL statement for which an alarm prompt has been issued; or, The preset alarm filtering condition includes that the first library name belongs to a preset library name set, and / or the first table name belongs to a preset table name set, the preset library names in the preset library name set are used to indicate library names for which no alarm is required, and the preset table names in the preset table name set are used to indicate table names for which no alarm is required.

4. The method according to claim 2, characterized in that: When the first SQL statement is a DML statement, the first statement information includes a second table name corresponding to the DML statement; When it is determined that a statement alarm needs to be issued for the first SQL statement, obtaining a preset alarm filtering condition corresponding to the first SQL statement includes: When it is determined that a statement alarm needs to be issued for the first SQL statement, a preset alarm threshold corresponding to the statement alarm is obtained from the preset alarm information; Determine a target number of abnormal data appearing in a target table corresponding to the second table name; When the target quantity is greater than the preset alarm threshold, a preset alarm filtering condition corresponding to the first SQL statement is obtained.

5. The method according to claim 4, characterized in that When the first SQL statement is a DML statement, the first statement information further includes a second shard ID corresponding to the DML statement and a second library name corresponding to the DML statement; The preset alarm filtering condition includes that the second shard ID is consistent with the shard ID corresponding to the second SQL statement, the second library name is consistent with the library name corresponding to the second SQL statement, and the second table name is consistent with the table name corresponding to the second SQL statement, and the second SQL statement is used to indicate the SQL statement for which an alarm prompt has been issued; or, The preset alarm filtering condition includes that the second library name belongs to a preset library name set, and / or the second table name belongs to a preset table name set, the preset library names in the preset library name set are used to indicate library names for which no alarm is required, and the preset table names in the preset table name set are used to indicate table names for which no alarm is required.

6. The method according to claim 2, characterized in that When the target alarm strategy is the second alarm strategy, issuing an alarm prompt for the first SQL statement includes: When the target alarm strategy is the second alarm strategy, according to the target statement type corresponding to the first SQL statement, a target alarm terminal corresponding to the target statement type is determined, the target statement type includes a DDL statement and a DML statement, and the target alarm terminal is used to indicate a terminal for issuing an alarm prompt for the first SQL statement; Determine target alarm information corresponding to the first SQL statement; The target alarm information is pushed to the target alarm terminal to issue an alarm prompt for the first SQL statement.

7. The method according to claim 6, characterized in that After executing the step of pushing the target alarm information to the target alarm terminal to issue an alarm prompt for the first SQL statement, the method further includes: Receiving target feedback information corresponding to the preset alarm information sent by the target alarm terminal; When the target feedback information is used to modify the preset alarm information, modify the preset alarm information according to the target feedback information to complete the update of the preset alarm information; After completing the updating of the preset alarm information, the updated preset alarm information is used to execute the step of obtaining the first SQL statement and the preset alarm information during the distributed database data synchronization process.

8. A monitoring device for distributed database data synchronization, characterized in that: include: An acquisition module, used to acquire the first SQL statement and preset alarm information during the distributed database data synchronization process; A parsing module, used for parsing the first SQL statement to obtain first statement information corresponding to the first SQL statement; A determination module, configured to determine whether a statement warning is required for the first SQL statement according to the first statement information and the preset warning information; The determination module is configured to determine a target alarm strategy corresponding to the first SQL statement according to the first statement information when it is determined that a statement alarm needs to be issued for the first SQL statement; An alarm module is used to perform a target alarm operation corresponding to the target alarm strategy on the first SQL statement according to the target alarm strategy.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a distributed database data synchronization monitoring program stored in the memory to implement the distributed database data synchronization monitoring method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the distributed database data synchronization monitoring method described in any one of claims 1 to 7.

Images