A monitoring method, device, and storage medium

Abstract

This application discloses a monitoring method, which includes: determining a rule to be executed for anomaly monitoring of a target product; parsing the rule to be executed to obtain a database identifier to be accessed and a data table identifier to be accessed; determining the type of the rule to be executed based on the database identifier to be accessed and the data table identifier to be accessed; if the type is a target type, obtaining database attribute information of at least one corresponding target database based on the database identifier to be accessed and the data table identifier to be accessed; generating a target engine based on the database attribute information; and executing the rule to be executed on at least one target database through the target engine to achieve monitoring of the target product. This application also discloses a monitoring device and a storage medium.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a monitoring method, device, and storage medium. Background Technology

[0002] With the rapid development of computer technology, more and more technologies are being applied in the financial field, and the traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on technology. In the field of data processing, the reliability of a product's entire application system is usually determined by checking for anomalies in the data of databases corresponding to different application systems. Currently, different product application systems may correspond to the same database or different databases under the same server in the same region, or different databases under different servers in different regions. Therefore, anomaly data monitoring rules are often divided into the following categories: a. rules within a single table; b. rules across tables within the same database; c. rules across databases and tables within the same server in the same region; d. rules across regions, servers, and databases / tables. When monitoring rules of types a and b, Structured Query Language (SQL) rules can be executed directly. When monitoring rules of type c, multiple databases can be accessed simultaneously by requesting database user permissions, and the execution is essentially the same as monitoring rules of types a and b. When monitoring rules for class d, since the databases are deployed on different server clusters, direct table joins are not possible. The current production technology involves extracting data from the databases that need to be accessed on different server clusters into a data warehouse tool (Hive), and then using Hibernate Query Language (HQL) to perform table joins for monitoring.

[0003] However, when monitoring for type d rules, data needs to be extracted from multiple different databases into Hive for data analysis, resulting in poor real-time performance of the data analysis. Furthermore, the data extraction process consumes a lot of computing resources, making the monitoring and analysis process inefficient.

[0004] Application content

[0005] To address the aforementioned technical problems, this application aims to provide a monitoring method, device, and storage medium that solves the current problem of low monitoring and analysis efficiency for cross-server and cross-database monitoring processes. It implements a real-time monitoring method for cross-server and cross-database processes, eliminating the need to extract data from different databases into Hive, reducing the consumption of computing resources, and improving the efficiency of the monitoring and analysis process.

[0006] The technical solution of this application is implemented as follows:

[0007] Firstly, a monitoring method, the method comprising:

[0008] Identify the rules to be run for anomaly monitoring of the target product;

[0009] Parse the rules to be executed to obtain the database identifier and the data table identifier to be accessed.

[0010] Based on the identifier of the database to be accessed and the identifier of the data table to be accessed, determine the type of the rule to be executed;

[0011] If the type is a target type, based on the database identifier to be accessed and the data table identifier to be accessed, obtain the database attribute information of at least one corresponding target database;

[0012] Based on the database attribute information, a target engine is generated;

[0013] The target engine executes the rules to be run against at least one of the target databases to monitor the target products.

[0014] Secondly, a monitoring device, the device comprising: a memory, a processor, and a communication bus; wherein:

[0015] The memory is used to store executable instructions;

[0016] The communication bus is used to realize the communication connection between the processor and the memory;

[0017] The processor is configured to execute the monitoring program stored in the memory to implement the steps of the monitoring method as described in any of the preceding claims.

[0018] Thirdly, a storage medium storing a monitoring program, which, when executed by a processor, implements the steps of the monitoring method as described in any of the preceding claims.

[0019] In this embodiment, after determining the rules to be run for anomaly monitoring of the target product, the rules are parsed to obtain the identifiers of the databases to be accessed and the tables to be accessed. Then, based on these identifiers, the type of the rule is determined. If the type is a target type, the database attribute information of at least one target database is obtained. Based on this database attribute information, a target engine is generated, and the rules are executed against at least one target database to achieve monitoring of the target product. This approach, by parsing the rules and determining their type, and generating a target engine based on the database attribute information of at least one target database when the type is a target type, enables the execution of the rules and achieves monitoring. This solves the problem of low monitoring and analysis efficiency in current cross-server and cross-database monitoring processes, providing a real-time cross-server and cross-database monitoring method that eliminates the need to extract data from databases to Hive, reducing computational resource consumption and improving the efficiency of the monitoring and analysis process. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a monitoring method provided in an embodiment of this application;

[0021] Figure 2 A flowchart illustrating another monitoring method provided in an embodiment of this application;

[0022] Figure 3 A flowchart illustrating yet another monitoring method provided in an embodiment of this application;

[0023] Figure 4 A flowchart illustrating a monitoring method provided in another embodiment of this application;

[0024] Figure 5 A flowchart illustrating another monitoring method provided in another embodiment of this application;

[0025] Figure 6 A flowchart illustrating yet another monitoring method provided in another embodiment of this application;

[0026] Figure 7 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the structure of a monitoring device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0029] The embodiments of this application provide a monitoring method, referring to... Figure 1 As shown, the method is applied to monitoring equipment and includes the following steps:

[0030] Step 101: Determine the rules to be run for monitoring anomalies in the target product.

[0031] In this embodiment, the monitoring device can be a server device or a service cluster. The target product can be a business product, such as various business products in the financial field; for example, it can be a loan business product in the financial field. The rules to be run are existing rules used to monitor the target product, which in some application scenarios can be referred to as stock rules.

[0032] Step 102: Parse the rules to be run to obtain the identifier of the database to be accessed and the identifier of the data table to be accessed.

[0033] In this embodiment of the application, the rules to be run are parsed and analyzed to determine the databases that need to be accessed in the rules to be run, thereby obtaining the identifier of the database to be accessed and the specific data tables involved in the rules to be run, thus obtaining the identifier of the data tables to be accessed.

[0034] Step 103: Determine the type of the rule to be run based on the identifier of the database to be accessed and the identifier of the data table to be accessed.

[0035] In this embodiment, the identifier of the database to be accessed and the identifier of the data table to be accessed are analyzed to determine the type of the rule to be executed. The rule to be executed can be divided according to whether the database it accesses crosses service regions, specifically into two categories: one is a non-cross-service region type, and the other is a cross-service region type.

[0036] Step 104: If the type is the target type, obtain the database attribute information of at least one target database based on the database identifier to be accessed and the data table identifier to be accessed.

[0037] In this embodiment, the target type is a cross-service region type. When the type of the rule to be run is a cross-service region type, at least one target database is determined based on the identifier of the database to be accessed and the identifier of the data table to be accessed, and the database attribute information of the corresponding at least one target database is obtained. The database attribute information of the target database may include information required for accessing the database, as well as environmental information to which the target database belongs, etc.

[0038] Step 105: Generate the target engine based on the database attribute information.

[0039] In this embodiment of the application, a target engine is generated based on the database attribute information of at least one target database corresponding to the rule to be run, so that the rule to be run can be executed through the target engine to achieve real-time monitoring.

[0040] Step 106: Execute the rules to be run against at least one target database using the target engine.

[0041] Specifically, the target engine executes rules to be run against at least one target database to monitor the target product.

[0042] In this embodiment of the application, the generated target engine converts the rules to be run into runnable rules and executes the runnable rules to achieve the execution of the rules to be run in order to monitor the target product.

[0043] In this embodiment, after determining the rules to be executed for anomaly monitoring of the target product, the rules are parsed to obtain the identifiers of the databases to be accessed and the tables to be accessed. Then, based on these identifiers, the type of the rules is determined. If the type is a target type, the database attribute information of at least one target database is obtained based on the identifiers. A target engine is generated based on this database attribute information, and the rules are executed against at least one target database using the target engine to monitor the target product. This approach, by parsing the rules to be executed and determining their type (when the type is a target type), and generating a target engine based on the database attribute information of at least one target database involved in the rules, enables the execution of the rules and achieves monitoring. This solves the problem of low monitoring and analysis efficiency in current cross-server and cross-database monitoring processes, providing a real-time monitoring method for cross-server and cross-database scenarios. It eliminates the need to extract data from different databases into Hive, reducing computational resource consumption and improving the efficiency of the monitoring and analysis process.

[0044] Based on the foregoing embodiments, embodiments of this application provide a monitoring method, referring to... Figure 2 As shown, this method is applied to monitoring equipment and includes the following steps:

[0045] Step 201: Determine the rules to be run for anomaly monitoring of the target product.

[0046] In this embodiment of the application, the monitoring device obtains existing rules for anomaly monitoring that are currently pending for the target product.

[0047] Step 202: Parse the rules to be run to obtain the identifier of the database to be accessed and the identifier of the data table to be accessed.

[0048] In this embodiment, the monitoring device parses the rules to be run to determine the database identifier and the table identifier to be accessed included in the rules. The database identifier is used to uniquely identify the corresponding database, such as the database name or database number; the table identifier is used to uniquely identify the corresponding table, such as the table name or table number.

[0049] Step 203: Determine the type of the rule to be run based on the identifier of the database to be accessed and the identifier of the data table to be accessed.

[0050] In this embodiment, the identifiers of the database to be accessed and the table to be accessed are analyzed to determine the service regions to which the corresponding databases and tables belong. If the database corresponding to the identifier of the database to be accessed and the table corresponding to the identifier of the table to be accessed are both in the same service region, the type of the rule to be run is determined to be a non-cross-service region type; if the database corresponding to the identifier of the database to be accessed and the table corresponding to the identifier of the table to be accessed are in multiple service regions, the type of the rule to be run is determined to be a cross-service region type.

[0051] Step 204: If the type is a target type, determine at least one target database to be accessed based on the database identifier and the table identifier to be accessed.

[0052] In this embodiment, when the type is not a cross-service region type, the rules to be run can be executed directly. When the type is a target type, i.e., a cross-service region type, the database to which the data table identifier to be accessed belongs and the database corresponding to the database identifier to be accessed are determined. Then, the determined databases are deduplicated to obtain at least one target database to be accessed.

[0053] Step 205: Traverse the test environment corresponding to at least one target database to obtain database attribute information.

[0054] In this embodiment of the application, the test environment corresponding to at least one target database is traversed and analyzed to obtain the database attribute information of each target database. The database attribute information of each target database may include, for example, the test environment corresponding to each target database, the data center node (DCN) region, the Internet Protocol (IP), the port, the username, the password, etc.

[0055] Step 206: Generate the target engine based on the database attribute information.

[0056] In this embodiment of the application, a target engine is generated based on the database attribute information of at least one target database. The target engine may be a federated engine.

[0057] Step 207: Execute the rules to be run against at least one target database through the target engine to achieve monitoring of the target products.

[0058] In this embodiment of the application, the target engine converts the rules to be run into executable rule code for execution, thereby realizing the monitoring of the target product.

[0059] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 3 As shown, after executing step 206, the monitoring device also executes step 208:

[0060] Step 208: Based on the database attribute information, generate an engine data table corresponding to the target engine.

[0061] The engine data table is used to record the characteristic information of the data table corresponding to the rule to be run.

[0062] In this embodiment of the application, an engine data table corresponding to the target engine is generated based on the database attribute information of at least one target database. This table is used to record the feature information of all data corresponding to the rules to be run, ensuring that the target engine can access the source data.

[0063] Correspondingly, step 207 is implemented by step 207a:

[0064] Step 207a: Execute the rules to be run through the target engine and based on the engine data table.

[0065] In this embodiment of the application, the target engine accesses the access data source recorded in the engine data table, executes the rules to be run, and realizes the monitoring operation.

[0066] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 4 As shown, after executing step 207, the monitoring device also executes steps 209 to 210:

[0067] Step 209: If an update to the rules to be run is detected at a preset time interval, the target engine updates the engine data table based on the updated rules to be run.

[0068] In this embodiment, the monitoring device checks the rules to be run at preset time intervals to detect whether there are updates. Updates include rule additions, rule deletions, and / or rule modifications. The process by which the target engine updates its data table based on the updated rules to be run can specifically be as follows: the target engine parses the updated rules to be run, determines at least one reference database corresponding to the updated rules, and determines the database attribute information of the at least one reference database to update the engine data table.

[0069] Step 210: Update the engine data table by recording the update time of the target engine.

[0070] In this embodiment of the application, the update time of the engine data table is updated by the target engine record. In this way, the update time of the engine data table can be used to detect whether the rules to be run have been updated.

[0071] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 5 As shown, after the monitoring device performs capture 209, it is also used to perform step 211:

[0072] Step 211: Execute the updated rules to be run through the target engine, based on the updated engine data table.

[0073] In the embodiments of this application, the implementation process of step 211 can refer to the specific implementation process of step 207a, which will not be described in detail here.

[0074] Based on the foregoing embodiments, in other embodiments of this application, step 207a can be implemented by steps a11 to a13:

[0075] Step a11: Generate the running strategy based on the rules to be run and the engine data table through the target engine.

[0076] In this embodiment, the target engine analyzes the rules to be run and the engine data table according to the requirements of the running strategy generation, and then generates the running strategy.

[0077] Step a12: Generate the target execution statement based on the running strategy through the target engine.

[0078] In this embodiment, the target engine generates corresponding target execution statements based on the generated execution strategy. It should be noted that the execution strategy is generated for different service regions; therefore, the generated target execution statements include execution statements specific to different service regions.

[0079] Step a13: Execute the target execution statement through the target engine to execute the rules to be run.

[0080] In this embodiment of the application, when the target engine executes the target execution statement, it distributes the execution statements of different service areas to the nodes corresponding to the different service areas, and executes the corresponding execution statements through the corresponding nodes to realize the execution of the rules to be run.

[0081] Based on the foregoing embodiments, in other embodiments of this application, step a11 can be implemented by steps a111 to a116:

[0082] Step a111: From the engine data table, determine at least one first data table of at least one database corresponding to the same virtual region under the same service area.

[0083] In this embodiment of the application, the data tables recorded in the engine data table are grouped. One grouping method is to determine at least one first data table in at least one database corresponding to the same virtual region under the same service area.

[0084] Step a112: Based on the rules to be run, combine at least one first data table to obtain the first strategy.

[0085] In this embodiment of the application, at least one first data table is combined according to the relationship in at least one first data table in the rules to be run to obtain a first strategy.

[0086] Step a113: From the engine data table, determine at least one second data table of at least one database corresponding to different virtual regions under different service areas.

[0087] In this embodiment of the application, the data tables recorded in the engine data table are grouped. Another grouping method is to determine at least one second data table in at least one database corresponding to different virtual regions under different service areas.

[0088] Step a114: Based on the rules to be run, combine at least one second data table to obtain a second strategy.

[0089] In this embodiment of the application, at least one second data table is combined according to the relationship in at least one second data table in the rule to be run to obtain a second strategy.

[0090] Step a115: From the engine data table, determine the third data table where there is only one service area corresponding to the virtual area.

[0091] In this embodiment of the application, a third data table is determined under different virtual regions that belong to only one service area.

[0092] Step a116: Based on the rules to be run, determine the third data table as the third strategy.

[0093] The operating strategies include the first strategy, the second strategy, and the third strategy.

[0094] In this embodiment of the application, the third data table is directly determined as the third strategy, without the need for combination processing.

[0095] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 6 As shown, after executing step 207, the monitoring device is also used to execute steps 212 to 214:

[0096] Step 212: Output the execution results through the target engine.

[0097] The execution result is obtained by the target engine executing the rules to be run.

[0098] In this embodiment of the application, after the target engine executes the target execution statement, it obtains the corresponding execution result. The target engine outputs the execution result, which can be displayed.

[0099] Step 213: If the target product is determined to be abnormal based on the execution results, at least one abnormal data rule is determined through the target engine.

[0100] In this embodiment of the application, the execution result is analyzed to determine whether the execution result indicates that the target product is abnormal or the execution result indicates that the target product is abnormal. The abnormal rule in the rules to be run is determined by the target engine to obtain at least one abnormal data rule.

[0101] Step 214: Execute the anomaly repair operation through the target engine based on at least one abnormal data rule and a preset anomaly repair method.

[0102] In this embodiment, the preset anomaly repair method is a pre-defined repair method for data. For example, it could be a data correction method, a data cleanup method, or both. The data correction method is a way to modify the data, and the data cleanup method is a way to clean up the abnormal data. Thus, after the target engine determines at least one abnormal data rule, it uses the preset anomaly repair method to process the data corresponding to the at least one abnormal data rule, thereby achieving the anomaly repair operation.

[0103] Based on the foregoing embodiments, in other embodiments of this application, step 214 can be implemented by steps 214a to 214d:

[0104] Step 214a: Determine at least one list object involved in each anomalous data rule through the target engine.

[0105] In this embodiment of the application, each abnormal data rule is analyzed to determine at least one list object involved in each abnormal data rule. For example, the list object may be the list identifier information of a list.

[0106] Step 214b: Determine the association rules for each list object from at least one target database using the target engine, thus obtaining the target association rules.

[0107] In this embodiment of the application, at least one target database is analyzed to determine the associated objects that are related to each list object. The corresponding association rules are determined based on the associated objects. In this way, all association rules corresponding to at least one abnormal data rule can be determined, which are denoted as target association rules.

[0108] Step 214c: Obtain the rule to be repaired by the target engine based on at least one abnormal data rule and the target association rule.

[0109] In this embodiment of the application, at least one abnormal data rule and a target association rule are analyzed and processed to determine the rule to be repaired.

[0110] Step 214d: The target engine performs an anomaly repair operation based on the rules to be repaired and the preset anomaly repair method.

[0111] In this embodiment, the target engine processes the data corresponding to the rule to be repaired using a preset anomaly repair method to achieve the anomaly repair operation.

[0112] It should be noted that in some application scenarios, before executing step 214d, the data corresponding to the rule to be repaired can be backed up so that a roll-up operation can be performed in case of subsequent failures, or a fault analysis and repair process can be performed for the faults that occur during the abnormal repair operation.

[0113] Based on the foregoing embodiments, in other embodiments of this application, step 214c can be implemented by the following steps: grouping at least one abnormal data rule and the target association rule according to the relationship of the target engine being grouped according to the rule belonging to the same service area, to obtain the rule to be repaired.

[0114] In this embodiment, the target engine groups at least one abnormal data rule and a target association rule according to a grouping relationship that group rules belonging to the same service area, to obtain the rule to be repaired. Where at least one abnormal data rule and a target association rule have duplicate rules, deduplication is required.

[0115] Based on the foregoing embodiments, in other embodiments of this application, step 214d can be implemented by steps b11-b12, or steps b13-b14, or steps b15-b16:

[0116] Step b11: If the preset anomaly repair method is the number repair method, obtain the preset repair rule corresponding to the rule to be repaired through the target engine.

[0117] In this embodiment of the application, when the preset anomaly repair method is the data repair method, the target engine obtains the preset repair rule corresponding to the rule to be repaired. The preset repair rule is a rule that is pre-set for each rule in the rule to be repaired to update the corresponding data.

[0118] Step b12: Execute the preset repair rules through the target engine to perform the anomaly repair operation.

[0119] In this embodiment, the target engine executes a preset repair rule corresponding to the rule to be repaired, thereby performing corresponding repair processing on the data corresponding to the rule to be repaired. For example, it replaces the abnormal data corresponding to the rule to be repaired with the data specified in the preset repair rule.

[0120] Step b13: If the preset anomaly repair method is the cleanup method, perform data cleanup operations on the corresponding rules to be repaired through the target engine.

[0121] Anomaly repair operations include data cleaning operations.

[0122] In this embodiment of the application, when the preset anomaly repair method is the cleanup method, the target engine performs a data cleanup operation on the data corresponding to the rule to be repaired, such as deleting the data corresponding to the rule to be repaired.

[0123] Step b14: If the preset anomaly repair methods include number repair and cleanup, obtain the preset repair rules corresponding to the rules to be repaired through the target engine.

[0124] Step b15: After executing the preset repair rules through the target engine, perform the data cleaning operation.

[0125] In this embodiment, when the preset anomaly repair methods include data manipulation and cleanup, these two repair methods typically have a processing priority order to ensure the accuracy of the final processed data. Generally, the processing priority of the data manipulation method is set higher than that of the cleanup method. That is, during execution, the repair processing corresponding to the data manipulation method must be performed for the rule to be repaired, and then the repair processing corresponding to the cleanup method is performed. However, in some application scenarios, the priority of the cleanup method can also be set higher than that of the data manipulation method, which can be determined according to the actual application scenario.

[0126] Based on the foregoing embodiments, in other embodiments of this application, after the monitoring device executes step 214d, it is also used to execute step 214e:

[0127] Step 214e: If an anomaly is detected in the target product, repeat the step "If the target product is determined to be abnormal based on the execution result, determine at least one abnormal data rule through the target engine" through the target engine until the target product is detected to be normal, and end the anomaly repair operation.

[0128] In this embodiment of the application, if the monitoring device detects that the target product is normal after executing step 214d, that is, there is no abnormality in the monitoring of the target product, it indicates that the abnormality has been eliminated, and the abnormality repair operation ends. Otherwise, if the target product is still detected to be abnormal, the corresponding content of steps 213 to 214 needs to be repeated until the target product is detected to be normal before the abnormality repair operation ends.

[0129] Based on the foregoing embodiments, in other embodiments of this application, this application provides a monitoring method, including:

[0130] Step c11: Determine if there is a cross-service region issue in the existing rules. If there is, proceed to step c12; otherwise, proceed to step c14.

[0131] One application scenario diagram that spans multiple service areas can be found by referring to... Figure 7 As shown, the corresponding stock rules can be denoted as Rac = Ta1 join Tc1; Rbc = Tb1 join Tc1, where Rac is obtained by joining Ta1 in service region 1 and Tc1 in service region 2, realizing cross-region access between service region 1 and service region 2. Similarly, Rbc is obtained by joining Tb1 in service region 1 and Tc1 in service region 2. When determining that a stock rule is a cross-service region rule, it can be marked with preset identification information for subsequent analysis.

[0132] Step c12, Engine initialization.

[0133] After identifying the databases corresponding to the existing rules, the test environment for each database is traversed to obtain database (DB) information. DB information includes at least the test environment, DCN region, IP address, port number, user name, and password. Typically, this information can be obtained from the configuration center of the Configuration Management Database (CMDB). If it is also stored locally, it can be retrieved directly from the local machine.

[0134] For example, when cross-service region stock rules involve one Application Data Management (ADM) region and two DCN regions, and are deployed on different service regions DBSET1 and DBSET2, the corresponding DB information can be recorded as follows: DBSET1: Source environment K, region ADM, database glpdb, IP: 10.1.1.1, PORT: 3301, USER: user1, PASSWD: xxx, table T1; DBSET2: Source environment K, region DCN, database cpsdb, IP: 10.1.1.2, PORT: 3302, USER: user2, PASSWD: xxx, table T2.

[0135] Step c13: Execute the existing rules through the engine.

[0136] First, based on environmental and DCN region information, a federated engine is created. This engine includes a master node for decision-making and slave nodes corresponding to multiple service regions. Simultaneously, a federated table is created corresponding to the engine, with a mapping rule for the federated table: source database name_source table name_region name. During this process, a mapping relationship is established between the federated engine and the databases corresponding to the existing rules.

[0137] Secondly, the federated engine generates different running strategies based on existing rules and the federated table, following these steps:

[0138] 1) Parse existing rules to generate a list of rule regions, resulting in the federated table. For example, some regions in the federated table related to existing rules are shown in the table below; where, table T1: indicates that there is only one logical region in virtual region A0; table T2: indicates that there are two logical regions in virtual region B0; tables T3 and T4: indicate that there are two different service databases in the same virtual region C0.

[0139] Logical area virtual area Library surface A1 A0 db1 T1 B1 B0 db2 T2 B2 B0 db2 T2 C1 C0 db3 T3 C2 C0 db3 T3 C1 C0 db4 T4 C2 C0 db4 T4

[0140] 2) Generate execution strategies based on the federated table. These strategies include: Strategy 1, Strategy 2, and Strategy 3. Strategy 1 combines the same logical regions within the same virtual region but different service regions. For example, when joining tables T3 and T4, the generated combinations are (C1, C1) and (C2, C2). Strategy 2 requires separate combinations of logical regions within different virtual regions and different service regions. For example, when joining tables T2 and T3, the generated combinations are (B1, C1), (B1, C2), (B2, C1), and (B2, C2). Strategy 3 requires only one combination if the corresponding virtual region of the table has only one logical region. For example, when querying table T1 alone, the generated combination is (A1).

[0141] 3) Based on the running strategy, iteratively replace existing rules to generate executable statements for the engine.

[0142] In the process of iteratively replacing existing rules to generate engine-executable statements, the iterative replacement method remains consistent with the principles used to create the federated table. For example, the existing rules before replacement are shown below:

[0143] select count(1)from

[0144] (select a,b from db1.T1)t1

[0145] join

[0146] (select a,b from db2.T2)t2

[0147] on t1.a=t2.a

[0148] where t1.c = 'xx';

[0149] Correspondingly, after the replacement, two combinations (A1, B1) and (A1, B2) are generated according to the strategy combination. The engine's executable statements for the two rules are as follows:

[0150] select count(1)from

[0151] (select a,b from fedlink_db1_A1.T1)t1

[0152] join

[0153] (select a,b from fedlink_db2_B1.T2)t2

[0154] on t1.a=t2.a

[0155] where t1.c = 'xx';

[0156] select count(1)from

[0157] (select a,b from fedlink_db1_A1.T1)t1

[0158] join

[0159] (select a,b from fedlink_db2_B2.T2)t2

[0160] on t1.a=t2.a

[0161] where t1.c = 'xx';

[0162] It should be noted that when generating executable statements for the engine, the database tables of each virtual region involved in the existing rules need to be traversed until executable statements for the engine corresponding to all strategy combinations are generated.

[0163] Finally, the federated engine runs the engine's executable statements in parallel to implement the execution process of existing rules.

[0164] Step c14: Execute the existing rules.

[0165] Based on the foregoing embodiments, in other embodiments of this application, after the federated engine executes step c13, it is further configured to execute the following steps:

[0166] Step c15: Check whether the existing rules have been updated at preset time intervals. If the existing rules have been updated, the federated engine and federated table are also updated synchronously, and then the updated existing rules are executed.

[0167] Based on the foregoing embodiments, in other embodiments of this application, after the federated engine executes step c13 or step c15, it is further configured to execute the following steps:

[0168] Step c16: Output the results of the rule execution.

[0169] The federated engine combines and displays the generated strategies across different regions. If any combination fails to meet the exception verification conditions, the rule execution result is determined to be a rule check failure. For example, if the two rule combinations (A1,B1) and (A1,B2) are split above, and (A1,B1) fails the check while (A1,B2) succeeds, the existing rule is still considered to have failed the check.

[0170] Step c17: If the rule execution result is a verification failure, identify at least one rule with abnormal data.

[0171] First, the key corresponding to the rule is found by iteratively reversing the table fields involving abnormal rules in the existing rules.

[0172] Secondly, perform multi-key extraction within a single database and search for similar association rules within that database. For example, assuming the cross-service region storage rule is R1, the corresponding steps are: 1) Extract the table list T_num involved in R1; 2) If the number of T_num is 1, assuming the table list T_num is T1, find the associated primary key and index of table T1, and then reverse-search the set of all associated keys, associated tables, and abnormal field value lists under the database to which table list T1 belongs to obtain the association rules within the single database; if the number of T_num is greater than 1, then split rule R1 into multiple subqueries. For example, if R1 involves tables T1, T2, and T3, it can be split into three subquery rules for tables T1, T2, and T3. The process of obtaining association rules for each subquery is the same as when the number of T_num is 1.

[0173] For example, the specific process of finding the primary key and index of table T1, and then reverse-searching the set of all keywords, related tables, and abnormal field value lists associated with table T1 in the database to obtain the association rules in a single database can be represented as follows: 1) Retrieve the keyword set t: [k1, k2, k3] from table T1; 2) Based on the keyword set k and the original field set sc [c1, c2, c3] and abnormal field set c: [c1, c2] in R1, find the set of related tables involved in similar association rules in the rule base: [t1, t2, t3]. The method for determining similar association rule tables is as follows: ① First, generate a linked list of tables, with T1 of the current rule R1 as the vertex; ② Find the list of first-level association rule nodes containing table T1.

[0174] Let L1r[R2,R3,…] and its association rule table be the intersection set L1t[t1,t2,t3]. It should be noted that, for…

[0175] To avoid scanning the entire rule base, the number of nodes to be traversed in the linked list of association rules is equal to the initial T_num. This process is repeated until all nodes in the generated linked list have been traversed. Once the association rule search is complete, the association rule corresponding to the final single database T1 can be obtained as: {k: [k1, k2, k3], t: [t1, t2, t3], c: [c1, c2]}.

[0176] Finally, the union of the association rules of multiple databases corresponding to the existing rules is taken to obtain at least one abnormal data rule.

[0177] Step c18: Determine the anomaly repair method, including the data repair method and the cleanup method.

[0178] Step c19: Determine at least one preset repair rule corresponding to an abnormal data rule.

[0179] The search rule base includes a set of preset repair rules Rk matching the value of k and a set of preset repair rules Rt matching the value of t. Then, based on the set of exception fields c{c1,c2}, the preset repair rules for table T1 are filtered from Rk and Rt. For example, rule R1 is denoted as: SELECT COUNT(1) FROM cpsdb.T1 WHERE c1 IS NULL and c2=0 AND c3='A'; The corresponding preset repair rule for rule R1 is: UPDATE cpsdb.T1 set c1=0,c2=1 where c1IS NULL and c2=0 AND c3='A'; The preset repair rule for rule R1 indicates that when c1 is empty, c2 is 0, and c3 is A, c1 will be updated to 0 and c2 to 1.

[0180] Step c20: Before executing the preset repair rules, in order to preserve the scene or trace the abnormal scenario, back up the abnormal data corresponding to at least one abnormal data rule.

[0181] Step c21: Execute the preset repair rule corresponding to at least one abnormal data rule.

[0182] When executing at least one preset repair rule corresponding to an anomaly data rule, it is necessary to determine whether it affects other data tables. If an anomaly is found, continue repeating step c17. Prioritize executing the preset repair rules determined in step c19 until all the final preset repair rules have been executed, resulting in a successful detection result.

[0183] After step c22, the data repair method is completed, the cleanup method will continue to be executed automatically until the cleanup method is completed and the monitoring is successful. If the abnormal data is successfully repaired, the abnormal data repair will be completed. Otherwise, repeat steps c17 to c22.

[0184] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0185] In this embodiment, after determining the rules to be executed for anomaly monitoring of the target product, the rules are parsed to obtain the identifiers of the databases to be accessed and the tables to be accessed. Then, based on these identifiers, the type of the rules is determined. If the type is a target type, the database attribute information of at least one target database is obtained based on the identifiers. A target engine is generated based on this database attribute information, and the rules are executed against at least one target database using the target engine to monitor the target product. This approach, by parsing the rules to be executed and determining their type (when the type is a target type), and generating a target engine based on the database attribute information of at least one target database involved in the rules, enables the execution of the rules and achieves monitoring. This solves the problem of low monitoring and analysis efficiency in current cross-server and cross-database monitoring processes, providing a real-time monitoring method for cross-server and cross-database scenarios. It eliminates the need to extract data from different databases into Hive, reducing computational resource consumption and improving the efficiency of the monitoring and analysis process.

[0186] Based on the foregoing embodiments, embodiments of this application provide a monitoring device that can be applied to... Figures 1-6 In the monitoring method provided in the corresponding embodiment, refer to Figure 8 As shown, the monitoring device 3 may include: a processor 31, a memory 32, and a communication bus 33, wherein:

[0187] Memory 32 is used to store executable instructions;

[0188] Communication bus 33 is used to realize the communication connection between processor 31 and memory 32;

[0189] Processor 31 is used to execute the monitoring program stored in memory 32 to implement the reference Figures 1-6 The implementation process of the corresponding embodiments is not described in detail here.

[0190] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium, simply referred to as a storage medium, which stores one or more programs that can be executed by one or more processors to achieve, as follows: Figures 1-6 The implementation process of the monitoring method provided in the corresponding embodiments will not be described in detail here.

[0191] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0192] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0193] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0194] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0195] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A monitoring method characterized by, The method comprises: determining a to-be-run rule for monitoring an abnormality of a target product; parsing the to-be-run rule to obtain a to-be-accessed database identifier and a to-be-accessed data table identifier; determining a type to which the to-be-run rule belongs based on the to-be-accessed database identifier and the to-be-accessed data table identifier; if the type to which the to-be-run rule belongs is a target type, obtaining database attribute information of at least one target database corresponding to the to-be-accessed database identifier and the to-be-accessed data table identifier; the target type is a cross-service area type; generating a target engine based on the database attribute information; generating an engine data table corresponding to the target engine based on the database attribute information; the engine data table is used to record feature information of a data table corresponding to the to-be-run rule; generating a running strategy by the target engine in the following manner: determining at least one first data table of at least one database corresponding to a same virtual area under a same service area from the engine data table; combining the at least one first data table based on the to-be-run rule to obtain a first strategy; determining at least one second data table of at least one database corresponding to different virtual areas under different service areas from the engine data table; combining the at least one second data table based on the to-be-run rule to obtain a second strategy; determining a third data table of a virtual area corresponding to a service area with only one from the engine data table; determining the third data table as a third strategy based on the to-be-run rule; the running strategy comprises the first strategy, the second strategy, and the third strategy; generating a target execution statement by the target engine based on the running strategy; executing the target execution statement by the target engine to implement execution of the to-be-run rule and monitoring of the target product.

2. The method of claim 1, wherein, If the type to which the to-be-run rule belongs is the target type, the method comprises: determining at least one target database to be accessed based on the to-be-accessed database identifier and the to-be-accessed data table identifier; traversing a test environment corresponding to the at least one target database to obtain the database attribute information.

3. The method of claim 1, wherein, After the target engine executes the to-be-run rule for the at least one target database, the method further comprises: if it is detected that the to-be-run rule is updated according to a preset time interval, updating the engine data table based on the to-be-run rule after the update by the target engine; recording an update time of the engine data table by the target engine.

4. The method of claim 3, wherein, After the engine data table is updated based on the to-be-run rule after the update if it is detected that the to-be-run rule is updated according to a preset time interval, the method further comprises: executing the to-be-run rule after the update based on the engine data table after the update by the target engine.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises the following steps after the target engine executes the to-be-run rules on the at least one target database: The target engine outputs an execution result, wherein the execution result is obtained by the target engine executing the to-be-run rules; If it is determined based on the execution result that the target product has an exception, the target engine determines at least one exception data rule; The target engine executes an exception repair operation based on the at least one exception data rule and a preset exception repair mode.

6. The method of claim 5, wherein, The target engine determines at least one list object involved in each exception data rule; The target engine determines a target association rule from at least one target database in which each list object has an association relationship; The target engine obtains a to-be-repaired rule based on the at least one exception data rule and the target association rule; The target engine executes an exception repair operation based on the to-be-repaired rule and the preset exception repair mode. The target engine groups the at least one exception data rule and the target association rule according to a rule that belongs to the same service area, and obtains the to-be-repaired rule.

7. The method of claim 6, wherein, If the preset exception repair mode is a repair mode, the target engine obtains a preset repair rule corresponding to the to-be-repaired rule; The target engine executes the preset repair rule to implement the exception repair operation; 8. The method according to claim 6 or 7, characterized in that, If the preset exception repair mode is a cleaning mode, the target engine executes a data cleaning operation corresponding to the to-be-repaired rule; wherein the exception repair operation includes the data cleaning operation; If the preset exception repair mode includes the repair mode and the cleaning mode, the target engine obtains the preset repair rule corresponding to the to-be-repaired rule; The target engine executes the preset repair rule and then executes the data cleaning operation. The method further comprises the following steps after the target engine executes the exception repair operation based on the at least one exception data rule and the preset exception repair mode: If the target product is monitored to have an exception, the target engine repeatedly executes the step of "if it is determined based on the execution result that the target product has an exception, the target engine determines at least one exception data rule" until the target product is monitored to be normal, and the exception repair operation is ended. The device comprises a memory, a processor and a communication bus; wherein:

9. The method of claim 5, wherein, The memory is used to store executable instructions; The communication bus is used to realize the communication connection between the processor and the memory; 10. A monitoring device, characterized by ​ ​ ​ The processor is configured to execute a monitoring program stored in the memory to implement the steps of the monitoring method of any one of claims 1 to 9.

11. A storage medium, characterized by The storage medium stores a monitoring program, and the monitoring program, when executed by a processor, implements the steps of the monitoring method of any one of claims 1 to 9.

Images