Data monitoring method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a data monitoring method and device, electronic equipment and a storage medium, wherein the data monitoring method comprises: in the case of detecting target data table update, traversing the monitoring rule expression associated with the target data table to determine a plurality of first monitoring objects corresponding to the target data table in the monitoring rule expression; obtaining the first object parameter of each first monitoring object according to the latest data of the target data table and storing it in a preset storage area; for each first monitoring object, if it is determined that the monitoring rule expression includes the first monitoring object and at least one second monitoring object and the second object parameter of each second monitoring object is stored in the preset storage area, then generating monitoring information according to the monitoring rule expression, the first object parameter and each second object parameter; and determining whether to trigger an alarm according to the monitoring information, thereby being able to reuse the object parameters stored in the preset storage area and reducing the computational workload.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data monitoring method, apparatus, electronic device and storage medium. Background Technology

[0002] With the development of electronic technology, the application of big data is becoming increasingly widespread. In big data platforms, data is often stored in units of data tables, and these tables are updated regularly. To ensure data integrity, monitoring rules are often configured for one or more data tables. The monitoring system responsible for data quality monitoring typically executes the monitoring tasks corresponding to each pre-configured monitoring rule.

[0003] In practical applications, when monitoring rules are complex, executing monitoring tasks often requires a lot of repetitive calculations, resulting in low efficiency. Summary of the Invention

[0004] This application provides a data monitoring method, apparatus, electronic device, and storage medium to reduce repetitive calculations in monitoring tasks.

[0005] In a first aspect, embodiments of this application provide a data monitoring method, including:

[0006] Upon detecting an update to the target data table, the monitoring rule expressions associated with the target data table are traversed to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expressions; the monitoring rule expressions are used to generate monitoring information based on the latest data of at least one data table; the at least one data table includes the target data table;

[0007] Based on the latest data in the target data table, obtain the first object parameter for each of the first monitored objects and store it in a preset storage area;

[0008] For each of the first monitored objects, if it is determined that the monitoring rule expression includes the first monitored object and at least one second monitored object, and the preset storage area stores the second object parameters of each second monitored object, then monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameters, and each second object parameter; the second monitored object corresponds to other data tables besides the target data table.

[0009] Based on the monitoring information, determine whether to trigger an alarm.

[0010] Secondly, embodiments of this application provide a data monitoring device, including:

[0011] The traversal unit is used to traverse the monitoring rule expression associated with the target data table when an update to the target data table is detected, and to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used to generate monitoring information based on the latest data of at least one data table; the at least one data table includes the target data table.

[0012] The acquisition unit is used to acquire the first object parameter of each of the first monitored objects based on the latest data of the target data table and store it in a preset storage area;

[0013] The generation unit is configured to, for each of the first monitored objects, if it is determined that the monitoring rule expression includes the first monitored object and at least one second monitored object, and the preset storage area stores the second object parameters of each second monitored object, generate monitoring information corresponding to the monitoring rule expression based on the monitoring rule expression, the first object parameters, and each second object parameter; the second monitored object corresponds to other data tables besides the target data table;

[0014] The first determining unit is used to determine whether to trigger an alarm based on the monitoring information.

[0015] Thirdly, embodiments of this application provide an electronic device, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the data monitoring method described in the first aspect.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the data monitoring method as described in the first aspect.

[0017] As can be seen, in this embodiment of the application, when an update to the target data table is detected, by determining multiple first monitoring objects corresponding to the target data table, and by obtaining the first object parameters of each first monitoring object based on the latest data of the target data table and storing them in a preset storage area, the preset storage area may store the object parameters of each monitoring object corresponding to each data table. These object parameters are generated based on the latest data after each data table is updated. Furthermore, by generating monitoring information based on the monitoring rule expression and the object parameters read from the preset storage area when it is determined that all object parameters required for the monitoring rule expression associated with the target data table are stored in the preset storage area, and determining whether to issue an alarm based on the monitoring information, the object parameters required for generating monitoring information based on each monitoring rule expression are obtained from the preset storage area or newly generated. The newly generated object parameters are also stored in the preset storage area and can be reused by other monitoring rule expressions. Therefore, different monitoring rule expressions can reuse the same object parameter stored in the preset storage area, reducing the amount of repetitive calculation in the monitoring task, instead of independently calculating each object parameter from scratch according to the monitoring rule expression corresponding to each monitoring rule when generating monitoring information corresponding to each monitoring rule expression. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a data monitoring method provided in this application embodiment;

[0020] Figure 2 A flowchart illustrating a configuration method for a monitoring system used to perform a data monitoring method, provided in an embodiment of this application;

[0021] Figure 3 A flowchart illustrating another data monitoring and processing method provided in this application embodiment;

[0022] Figure 4 A schematic diagram of a data monitoring device provided in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this application.

[0025] In practical applications of data monitoring, it may be necessary to configure a large number of monitoring rules with complex logical relationships. Each monitoring rule corresponds to a monitoring task, which is executed separately by the monitoring system responsible for monitoring data quality. During the execution of monitoring tasks, different monitoring rules may be related to the same data table and share the same part of the data, but the data between different monitoring tasks is not interconnected, inevitably requiring a large amount of repetitive calculations. To solve the above problems, this application provides a data monitoring method.

[0026] Figure 1 This is a flowchart illustrating a data monitoring method provided in an embodiment of this application. Figure 1 Data monitoring methods can be performed by electronic devices, which can be terminal devices such as mobile phones, laptops, smart interactive devices, etc.; or, the electronic device can be a server, such as a standalone physical server, a server cluster, or a cloud server capable of cloud computing. (See reference...) Figure 1 The data monitoring method provided in this embodiment specifically includes steps S102 to S108.

[0027] Step S102: If an update to the target data table is detected, traverse the monitoring rule expressions associated with the target data table to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expressions; the monitoring rule expressions are used to generate monitoring information based on the latest data of at least one data table; at least one data table includes the target data table.

[0028] In practice, platforms or systems with data monitoring needs can store data in the form of data tables. A single platform or system can have multiple data tables. Each data table can contain multiple metadata entries. Metadata, also known as intermediary data or relay data, is data about data, primarily describing data properties to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0029] For example, metadata may include: table name, whether it is a partitioned table, the type of each field, etc.

[0030] The update cycles for different data tables can be the same or different. Update cycle types include, but are not limited to, daily, weekly, monthly, and yearly updates. New data is generated for each new update cycle.

[0031] For each data table, its update cycle can be fixed or freely switched. For example, the business data corresponding to the data table is in the peak business season from March to September, and the data is updated frequently. From October to February, it is the off-season, and the data is updated less frequently. Therefore, from March to September, the update cycle type of the data table is daily, and from October to February, the update cycle type of the data table is switched to weekly.

[0032] The target data table can be any one of the multiple data tables in a platform or system that has data monitoring requirements.

[0033] When an update to the target data table is detected, it can be when a new update cycle for the target data table arrives.

[0034] If an update to the target data table is detected, the monitoring rule expressions associated with the target data table are traversed to determine the multiple first monitoring objects corresponding to the target data table in the monitoring rule expressions.

[0035] The target data table can be associated with one or more monitoring rule expressions.

[0036] The monitoring rule expression may include at least one monitoring object. Among the at least one monitoring object, the monitoring object corresponding to the target data table may be identified as the first monitoring object, and the monitoring object corresponding to other data tables besides the target data table may be identified as the second monitoring object.

[0037] Monitoring rule expressions are expressions written by users according to pre-configured monitoring rule syntax. Each monitoring rule expression corresponds to a monitoring rule, which is used to generate monitoring information based on the latest data from one or more data tables. Furthermore, after generating the monitoring information, it can be determined whether to trigger the alarm corresponding to that monitoring rule.

[0038] In the monitoring rule expression associated with the target data table, the monitoring rule corresponding to the monitoring rule expression is used to generate monitoring information based on the latest data in one or more data tables, including the target data table.

[0039] One or more monitoring rule expressions associated with the target data table can be determined as follows: if any monitoring rule expression includes at least one first monitoring object corresponding to the target data table, then the monitoring rule expression is determined as a monitoring rule expression associated with the target data table.

[0040] Traverse the monitoring rule expressions associated with the target data table to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expressions. This can be done by traversing one or more monitoring rule expressions associated with the target data table and, for each monitoring rule expression, determining at least one first monitoring object corresponding to the target data table in that monitoring rule expression.

[0041] The predefined monitoring rule syntax can be configured with monitoring objects, comparison operators, and object parameters of the monitoring objects.

[0042] The monitored object can be a data quality monitoring object, metadata in a data table, or a parameter type determined based on a data table. For example, the monitored object could be the number of rows in a new partition of a data table, the maximum value of a certain field, the average volatility of a certain field over the past N days, etc.

[0043] In a monitoring system that executes data monitoring methods, monitoring objects related to each data table can be pre-configured, and a corresponding object identifier can be configured for each monitoring object.

[0044] The monitoring rule syntax allows configuration of how monitored objects are represented in the monitoring rule expression. Specifically, monitored objects can be represented by object identifiers and a first preset symbol. The first preset symbol is used by electronic devices to identify object identifiers when parsing monitoring rule expressions based on the monitoring rule syntax.

[0045] For example, in a monitoring rule expression, ${id} can be used to represent the monitored object. For instance, if the object identifier for the average volatility of the amount field over the past N days is 100, then ${100} can be used to represent the monitored object.

[0046] Comparison operators include, but are not limited to: greater than, greater than or equal to, less than, less than or equal to, equal to, and not equal to. The syntax for monitoring rules can be configured with the representation of comparison operators in the monitoring rule expressions, as shown in Table 1.

[0047] Greater than > Greater than or equal to >＝ Less than < Less than or equal to <＝ equal = or == Not equal to ! = or <>

[0048] Table 1

[0049] Table 1 shows the representation of each comparison operator in the monitoring rule expression in one embodiment.

[0050] Each monitored object has a corresponding object parameter in each update cycle. For each monitored object, the object parameter can be the monitoring value obtained after the monitoring task is completed in the new update cycle. This monitoring value can be a decimal or an integer, and can be represented numerically.

[0051] For example, if the average volatility of the field 'amount' for monitored object 1 over the past 7 days is calculated to be 5.8%, then the object parameter for monitored object 1 is 0.058.

[0052] A monitoring rule expression can be a simple expression consisting of a monitored object, a comparison operator, and the object parameters of the monitored object. For example, ${100}>=102.8. Here, ${100} represents the monitored object with the object identifier "100", ">=" is the comparison operator "greater than or equal to", and 102.8 is a preset value used to compare with the object parameters of the monitored object.

[0053] In practice, after obtaining the object parameters of the monitored object, these parameters can be substituted into the monitored object to compare with the preset values ​​and obtain the comparison result. This comparison result can then be used to determine the monitoring information corresponding to the simple expression.

[0054] Monitoring rule expressions can also be compound expressions composed of multiple simple expressions. The syntax for monitoring rules can be configured to specify how compound expressions are constructed. These construction methods can connect multiple simple expressions using logical AND or logical OR operations.

[0055] Logical AND can use the && symbol, and logical OR can use the || symbol. For example, ${100}>=102.8&&${200}<500. This compound expression is composed of the simple expression 1: ${100}>=102.8, the logical AND symbol "&&", and the simple expression 2: ${200}<500.

[0056] Monitoring rule expressions can also be nested expressions. Nested expressions can be based on simple or compound expressions. Specifically, parentheses can be used to represent nesting relationships. Based on nesting relationships, the priority order of each calculation step in the monitoring rule expression can be determined.

[0057] For example, ${100}<80&&(${200}>200&&${200}<500). This nested expression is composed of the simple expression 1: ${100}<80, the logical AND symbol "&&", parentheses "()", and the compound expression 1: ${200}>200&&${200}<500.

[0058] In one specific implementation, the number of monitoring rule expressions associated with the target data table is multiple; traversing the monitoring rule expressions associated with the target data table to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expressions includes: for each monitoring rule expression, traversing the monitoring rule expressions and extracting at least one object identifier; among the at least one object identifier, determining the object identifier associated with the target data table as the first object identifier; and determining the monitoring object corresponding to each first object identifier as the first monitoring object.

[0059] Each monitoring rule can have one or more monitored objects. Furthermore, the monitoring rule expression corresponding to each monitoring rule can include object identifiers of one or more monitored objects. By traversing the monitoring rule expression, at least one object identifier can be extracted from the monitoring rule expression.

[0060] At least one object identifier may include only the first object identifier, or it may include both the first object identifier and other object identifiers besides the first object identifier. Based on the target data table, the object identifier associated with the target data table can be determined as the first object identifier among at least one object identifier.

[0061] After identifying at least one first object identifier, the monitoring object corresponding to each first object identifier can be identified as the first monitoring object.

[0062] In one specific implementation, traversing the monitoring rule expression and extracting at least one object identifier includes: traversing the monitoring rule expression and extracting at least one object identifier by running a first subclass file that inherits from the base class file; the base class file is converted from a pre-written expression syntax file; the first subclass file is used to extract the object identifier from the monitoring rule expression according to the syntax rules corresponding to the expression syntax file.

[0063] After the monitoring rule syntax is predetermined, a syntax file for that monitoring rule syntax can be written using tools such as ANLTR, resulting in an expression syntax file. This expression syntax file can be shared by multiple monitoring rules; therefore, for a platform or system with data monitoring needs, only one pre-written expression syntax file is required.

[0064] For example, in the expression syntax file, ${id} can be used to represent the monitored object, where id can be a number used to represent the object identifier.

[0065] For example, in an expression syntax file, COMP can be used to represent comparison operators, which include, but are not limited to, the symbols corresponding to greater than, greater than or equal to, less than, less than or equal to, equal to, and not equal to.

[0066] For example, in the expression syntax file, NUMBER can be used to represent the object parameter of the monitored object. The object parameter can be an integer, defined using INT, or a decimal, defined using FLOAT.

[0067] For example, in an expression syntax file, you can define concatenation symbols for expressions, using logical AND and logical OR to connect them. Logical AND uses the && symbol, and logical OR uses the || symbol.

[0068] For example, in an expression syntax file, a simple expression can be defined as consisting of a monitored object, a comparison operator, and an object parameter of the monitored object. A simple expression can still be defined as a simple expression if it is enclosed in parentheses.

[0069] For example, in an expression syntax file, a compound expression can be defined as consisting of a simple expression and a logical AND / OR concatenation.

[0070] After writing the expression syntax file described above, you can use the ANLTR tool or other tools to automatically generate a base class file based on it. This base class file includes the default parsing code. In the default parsing code, the # comments after simple and compound expression syntaxes are used to indicate the method descriptions in the generated default parsing code.

[0071] Before iterating through the monitoring rule expressions, it is also necessary to write a first subclass file that inherits from the above base class file. This first subclass file can be used to extract object identifiers from the monitoring rule expressions according to the syntax rules corresponding to the expression syntax file.

[0072] For example, the base class file could be the DqcAlarmWhereBaseListener file, and the first subclass file needs to override the enterExprComp method to extract the value of "id" from ${id}.

[0073] The pre-written first subclass file can be used to extend the code of the base class file and implement the extraction of object identifiers.

[0074] By running the first subclass file that inherits from the base class file, the monitoring rule expressions can be traversed and at least one object identifier can be extracted.

[0075] Step S104: Based on the latest data in the target data table, obtain the first object parameters of each first monitored object and store them in a preset storage area.

[0076] The default storage area can be a cache area. For example, the default storage area can be a Redis (RemoteDictionary Server) cache area. The storage format can be ${id}->value.

[0077] The latest data in the target data table can be the data after an update to the target data table is detected.

[0078] The first monitoring object can be metadata from the target data table, or a parameter type determined based on the target data table. For example, the first monitoring object could be the number of rows in a new partition of the data table, the maximum value of a certain field, the average volatility of a certain field over the past N days, etc.

[0079] In the latest data of the target data table, the object parameters of the first monitored object can be obtained. If the first monitored object is metadata in the target data table, the method of obtaining it can be to collect object parameters. If the first monitored object is a parameter type determined based on the target data table, the method of obtaining it can be to calculate object parameters.

[0080] In one specific implementation, the first object parameter of each first monitoring object is obtained based on the latest data of the target data table, including: performing a monitoring value calculation task corresponding to each first monitoring object based on the latest data of the target data table to obtain the first object parameter of each first monitoring object.

[0081] When the first monitored object is a parameter type determined based on the target data table, each first monitored object corresponds to a monitoring value calculation task.

[0082] Based on the latest data in the target data table, the monitoring value calculation task corresponding to each first monitoring object can be executed to obtain the first object parameter of each first monitoring object. For example, if the first monitoring object is the average value of field 1, field 2 and field 3, the field values ​​of field 1, field 2 and field 3 are read from the latest data in the target data table, and the monitoring value calculation task is executed to calculate the average value of the three field values ​​to obtain the first object parameter of the first monitoring object.

[0083] In another implementation, the monitoring value calculation task corresponding to the first monitoring object can be performed based on the latest and historical data of the target data table to obtain the first object parameter of the first monitoring object.

[0084] Step S106: For each first monitoring object, if the monitoring rule expression includes the first monitoring object and at least one second monitoring object, and the second object parameters of each second monitoring object are stored in the preset storage area, then the monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameters and each second object parameter; the second monitoring object corresponds to other data tables outside the target data table.

[0085] For each first monitored object, if it is determined that the monitoring rule expression only includes the first monitored object, then the monitoring information corresponding to the monitoring rule expression is generated based on the monitoring rule expression and the first object parameter.

[0086] For each first monitored object, if it is determined that the monitoring rule expression includes both the first monitored object and one or more second monitored objects, then query whether the second object parameter of each second monitored object is stored in the preset storage area.

[0087] If a monitoring rule expression is determined to include a first monitoring object and at least one second monitoring object, then the monitoring rule expression belongs to both the monitoring rule expression associated with the target data table and the monitoring rule expression of other data tables.

[0088] Other data tables besides the target data table can be any of the "at least one data table" mentioned in step S102 above, except for the target data table. That is, the monitoring rule expression is used to generate monitoring information based on the latest data from at least one data table, which includes both the target data table and other data tables besides the target data table.

[0089] For example, the target data table is associated with two monitoring rule expressions: Monitoring rule expression 1 is used to generate monitoring information based on the latest data of the target data table, data table 1, and data table 2. In this case, the first monitoring object is the monitoring object corresponding to the target data table, and the second monitoring object includes the monitoring object corresponding to data table 1 and the monitoring object corresponding to data table 2. Monitoring rule expression 2 is used to generate monitoring information based on the latest data of the target data table and data table 3. In this case, the second monitoring object is the monitoring object corresponding to data table 3.

[0090] It should be noted that for other data tables, similar to the target data table, the following steps are performed:

[0091] When an update to another data table is detected, the monitoring rule expressions associated with that other data table are traversed to determine multiple second monitoring objects corresponding to that other data table in the monitoring rule expressions. The monitoring rule expressions are used to generate monitoring information based on the latest data of at least one data table. At least one data table includes the other data table. Based on the latest data of the other data table, the second object parameters of each second monitoring object are obtained and stored in a preset storage area.

[0092] Therefore, if an update to the target data table is detected but other data tables have not yet been updated, the preset storage area does not store the second object parameters for each second monitored object.

[0093] Query whether the second object parameter of each second monitoring object is stored in the preset storage area. If the query result is no, continue to wait for the next data table update. When the next data table update is detected, the data table can be identified as the target data table, and return to execute step S102 and subsequent steps.

[0094] If the query result is yes, then monitoring information corresponding to the monitoring rule expression is generated based on the monitoring rule expression, the first object parameter, and each of the second object parameters.

[0095] In one specific implementation, after obtaining the first object parameter of each first monitored object based on the latest data of the target data table and storing it in a preset storage area, and before generating monitoring information corresponding to the monitoring rule expression based on the monitoring rule expression, the first object parameter, and each second object parameter, the data monitoring method further includes: determining an object identifier other than the first object identifier as a second object identifier in at least one object identifier; and querying the preset storage area to see if a corresponding second object parameter exists based on the second object identifier.

[0096] In step S102, among the at least one object identifier extracted, object identifiers other than the first object identifier can be identified as second object identifiers, and each second object identifier corresponds to a second monitored object.

[0097] Based on the second object identifier, it is possible to query in the preset storage area whether the second object parameter of the second monitoring object corresponding to the second object identifier is stored.

[0098] In one specific implementation, monitoring information corresponding to the monitoring rule expression is generated based on the monitoring rule expression, the first object parameter, and each second object parameter. This includes: traversing the monitoring rule expression and parsing the relational symbols; and generating the monitoring information corresponding to the monitoring rule expression based on the first monitoring object, the first object parameter, each second monitoring object, each second object parameter, and the parsing result of the relational symbols.

[0099] By traversing the monitoring rule expression and parsing the relational symbols, the parsing results of the relational symbols can be obtained. Based on the first monitoring object, the first object parameter, each second monitoring object, each second object parameter, and the parsing results of the relational symbols, the monitoring information corresponding to the monitoring rule expression is generated.

[0100] For example, in the monitoring rule expression ${100}>=102.8&&${200}<500, the first object parameter is substituted into the corresponding position of the first monitored object ${100} in the monitoring rule expression, and compared with the preset value 102.8. If the comparison result determines that the first object parameter is greater than or equal to 102.8, then the monitoring sub-information corresponding to the simple expression ${100}>=102.8 is true; otherwise, it is false. Similarly, the second object parameter is substituted into the corresponding position of the second monitored object ${200} in the monitoring rule expression, and compared with the preset value 500. If the comparison result determines that the second object parameter is less than 500, then the monitoring sub-information corresponding to the simple expression ${200}<500 is true; otherwise, it is false. When the monitoring sub-information corresponding to both simple expressions is true, the monitoring information corresponding to the monitoring rule expression is determined to be true; otherwise, the monitoring information corresponding to the monitoring rule expression is false.

[0101] In one specific implementation, traversing the monitoring rule expression and parsing relational symbols includes: traversing the monitoring rule expression, extracting relational symbols, parsing the calculation methods corresponding to the relational symbols, and parsing the monitoring objects corresponding to the relational symbols by running a second subclass file that inherits from the base class file; the base class file is converted from a pre-written expression syntax file; the second subclass file is used to parse relational symbols from the monitoring rule expression according to the syntax rules corresponding to the expression syntax file.

[0102] Before traversing the monitoring rule expressions, a second subclass file that inherits from the above base class file needs to be written. This second subclass file can be used to parse relational symbols from the monitoring rule expressions according to the syntax rules corresponding to the expression syntax file.

[0103] For example, the base class file can be the DqcAlarmWhereBaseListener file, and the second subclass file needs to override the visitWhereBracket, visitExprBracket, visitWhereAnd, visitWhereOr, visitWhereExpr, and visitExprComp methods.

[0104] The pre-written second subclass file can be used to extend the code of the base class file and implement the parsing of relational symbols.

[0105] By running the second subclass file that inherits from the base class file, you can traverse the monitoring rule expressions, extract relational symbols, parse the calculation methods corresponding to the relational symbols, and parse the monitoring objects corresponding to the relational symbols.

[0106] The calculation method corresponding to the relation symbol is analyzed. For example, the relation symbol is ">", and its corresponding calculation method is to compare the values ​​on the left and right sides. If the value on the left is greater than the value on the right, the monitoring sub-information corresponding to the simple expression with relation symbol ">" is true, and otherwise the monitoring sub-information corresponding to the simple expression with relation symbol ">" is false.

[0107] Parsing relational symbols can reveal the monitoring objects corresponding to them. For example, if the relational symbol is ">", the value on the left is the first monitoring object, a1. Additionally, parsing relational symbols can also extract the preset values ​​on the right side of the symbol.

[0108] In this embodiment, different monitoring rules can share the same expression syntax file, base class file, first subclass file, and second subclass file. Therefore, when the monitoring rules are complex, if a user wants to configure a new rule or modify an old rule, they only need to write or modify the monitoring rule expression. They do not need to modify the code or perform cumbersome nested configuration operations, which improves the efficiency of rule configuration and significantly reduces the dependence of rule configuration operations on the professional skills of the configuration personnel in scenarios where monitoring rules need to be adjusted frequently.

[0109] Step S108: Determine whether to trigger an alarm based on the monitoring information.

[0110] The monitoring information may include first monitoring information indicating the existence of data quality anomalies, and second monitoring information indicating the absence of data quality anomalies.

[0111] If the monitoring information is the first monitoring information, then an alarm will be triggered; if the monitoring information is the second monitoring information, then an alarm will not be triggered.

[0112] The following concrete example will help illustrate steps S102-S108:

[0113] For example, when a new update cycle for table A arrives, the first monitoring object corresponding to table A can be retrieved. Calculation tasks can be performed on each first monitoring object to collect its first object parameters, which are then stored in the Redis cache. Because the object parameters of a monitoring object may be used by multiple monitoring rule expressions, storing them in the Redis cache allows other monitoring rule expressions to reuse them, eliminating the need to repeatedly calculate the object parameters of each monitoring object when parsing other monitoring rule expressions.

[0114] After the first object parameter is stored in the Redis cache area, attempt to parse all monitoring rule expressions related to table A, and execute the parsing of each monitoring rule expression one by one.

[0115] When parsing a monitoring rule expression, the system first checks if all required object parameters are available in Redis. If all are available, the rule expression can be parsed. After retrieving all object parameters from Redis, the system automatically parses the monitoring information and determines whether to trigger an alarm. If so, an alarm is issued. If at least one of the required object parameters is not fully computed, the system waits for a new data cycle in another table and repeats the above steps.

[0116] In such Figure 1 In the illustrated embodiment, when an update to the target data table is detected, by identifying multiple first monitoring objects corresponding to the target data table, and by obtaining the first object parameters of each first monitoring object based on the latest data of the target data table and storing them in a preset storage area, the preset storage area may store object parameters of each monitoring object corresponding to each data table. These object parameters are generated based on the latest data after each data table is updated. Furthermore, by generating monitoring information based on the monitoring rule expression associated with the target data table and the object parameters read from the preset storage area when all the object parameters required are stored in the preset storage area, and determining whether to issue an alarm based on the monitoring information, the object parameters required when generating monitoring information based on each monitoring rule expression can be obtained from the preset storage area or newly generated. The newly generated object parameters are also stored in the preset storage area and can be reused by other monitoring rule expressions. Therefore, different monitoring rule expressions can reuse the same object parameter stored in the preset storage area, reducing the amount of redundant calculation in the monitoring task, instead of independently calculating each object parameter from scratch according to the monitoring rule expression corresponding to each monitoring rule when generating monitoring information. Based on the same technical concept as the aforementioned method embodiments, this application also provides an embodiment of a monitoring system. Figure 2This is a flowchart illustrating a configuration method for a monitoring system used to execute a data monitoring method, provided in an embodiment of this application. See also... Figure 2 The configuration method of the monitoring system specifically includes steps S202 to S210.

[0117] The monitoring system is used to monitor the data quality of each data table according to pre-configured monitoring rule expressions.

[0118] Step S202: Determine the monitoring rule syntax.

[0119] The syntax for monitoring rules can be found in [reference]. Figure 1 The corresponding description section is for step S102 in the embodiment.

[0120] Step S204: Design the ANTLR syntax file.

[0121] The ANTLR syntax file can be found here. Figure 1 The corresponding description section of the expression syntax file in the embodiment.

[0122] Step S206: Generate default syntax parsing code.

[0123] The default syntax parsing code can be referenced. Figure 1 The corresponding description section of the base class file in the embodiment.

[0124] Step S208: Inherit and extend the default syntax parsing code.

[0125] The code for inheriting and extending the default syntax parsing can be found in [reference]. Figure 1 The corresponding description of the first subclass file and the second subclass file in the embodiment.

[0126] Step S210: Embed the monitoring system.

[0127] Since the technical concept is the same, the description in this embodiment is relatively simple. For the relevant parts, please refer to the corresponding descriptions of the method embodiments provided above.

[0128] Based on the same technical concept as the aforementioned method embodiments, this application also provides an embodiment of yet another data monitoring method. Figure 3 A flowchart illustrating another data monitoring and processing method provided in this application embodiment. See also... Figure 3 The data monitoring method specifically includes steps S302 to S322.

[0129] Step S302: A new data table cycle is generated.

[0130] For example, a new update cycle for table A is approaching.

[0131] Step S304: Traverse all monitoring rules related to the data table to obtain the list of monitoring objects.

[0132] For example, in monitoring rule 1, it includes the number of rows of table A in the data table ${1} and the volatility of the number of rows of table B in the data table over the past 7 days ${2}. It can be determined that the list of monitoring objects includes the number of rows of table A ${1}.

[0133] Step S306: Execute the monitoring tasks respectively to calculate the object parameters of the monitoring objects.

[0134] Step S308: Store the object parameters in the Redis cache.

[0135] For example, when a new update cycle of table A arrives, execute the monitoring value calculation task for the monitoring object ${1}. Suppose the collected object parameter is 300 million, and store this object parameter in the Redis cache area. The storage format is: 1->300000000.

[0136] Step S310: Traverse all monitoring rules related to the data table and parse each monitoring rule one by one.

[0137] Step S312: Parse the monitoring rule expression.

[0138] Step S314: Obtain each object parameter required by the monitoring rule expression from Redis.

[0139] Step S316: Check whether each object parameter has been calculated.

[0140] If yes, execute step S318; if no, return to step S302.

[0141] For example, try to parse this monitoring rule and find that only the object parameter of ${1} exists in the Redis cache area, and the object parameter of ${2} does not exist. Then do nothing and continue to wait. When a new update cycle of table B arrives, execute the monitoring value calculation task for the monitoring object ${2}. Suppose the collected object parameter is 0.4, and store the object parameter in the Redis cache area. The storage format is: 2->0.4. Try to parse this monitoring rule expression and obtain these two object parameters from the Redis cache area, getting ${1} as 300 million and ${2} as 0.4.

[0142] Step S318: Generate monitoring information.

[0143] For example, parse the monitoring rule: (${1}>200000000||${1}<1000000>0)&&${2}>0.3. The left expression is true, and the right one is also true. Finally, left && right is still true.

[0144] Step S320: Is the monitoring information true?

[0145] If yes, proceed to step S322; otherwise, end the process.

[0146] Step S322: Trigger an alarm.

[0147] Since the technical concept is the same, the description in this embodiment is relatively simple. For the relevant parts, please refer to the corresponding descriptions of the method embodiments provided above.

[0148] In the above embodiments, a data monitoring method is provided. Correspondingly, based on the same technical concept, the embodiments of this application also provide a data monitoring device, which will be described below with reference to the accompanying drawings.

[0149] Figure 4 This is a schematic diagram of a data monitoring device provided in an embodiment of this application.

[0150] This embodiment provides a data monitoring device 400, including:

[0151] The traversal unit 401 is used to traverse the monitoring rule expression associated with the target data table when an update to the target data table is detected, and to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used to generate monitoring information based on the latest data of at least one data table; the at least one data table includes the target data table.

[0152] The acquisition unit 402 is used to acquire the first object parameter of each of the first monitored objects according to the latest data of the target data table and store it in a preset storage area;

[0153] The generation unit 403 is configured to, for each of the first monitored objects, if it is determined that the monitoring rule expression includes the first monitored object and at least one second monitored object, and the preset storage area stores the second object parameters of each second monitored object, generate monitoring information corresponding to the monitoring rule expression based on the monitoring rule expression, the first object parameters, and each second object parameter; the second monitored object corresponds to other data tables besides the target data table;

[0154] The first determining unit 404 is used to determine whether to trigger an alarm based on the monitoring information.

[0155] Optionally, the number of monitoring rule expressions associated with the target data table is multiple; the traversal unit 401 includes:

[0156] Extraction subunits are used to traverse the monitoring rule expressions for each monitoring rule expression and extract at least one object identifier;

[0157] The first determining subunit is used to determine the object identifier associated with the target data table as the first object identifier among the at least one object identifier;

[0158] The second determining subunit is used to determine the monitoring object corresponding to each of the first object identifiers as the first monitoring object.

[0159] Optionally, the data monitoring device also includes:

[0160] The second determining unit is used to determine an object identifier other than the first object identifier as the second object identifier among the at least one object identifier;

[0161] The query unit is used to query whether a corresponding second object parameter exists in the preset storage area based on the second object identifier.

[0162] Optionally, the generating unit 403 includes:

[0163] The parsing subunit is used to traverse the monitoring rule expression and parse the relational symbols;

[0164] A generation subunit is used to generate monitoring information corresponding to the monitoring rule expression based on the first monitoring object, the first object parameter, each second monitoring object, each second object parameter, and the parsing result of the relational symbol.

[0165] Optionally, sub-units are extracted, specifically for:

[0166] By running the first subclass file that inherits from the base class file, the monitoring rule expression is traversed and at least one object identifier is extracted; the base class file is converted from a pre-written expression syntax file; the first subclass file is used to extract the object identifier from the monitoring rule expression according to the syntax rules corresponding to the expression syntax file.

[0167] Optionally, parsing sub-units is specifically used for:

[0168] By running a second subclass file that inherits from the base class file, the monitoring rule expression is traversed, the relational symbols are extracted, the calculation methods corresponding to the relational symbols are parsed, and the monitoring objects corresponding to the relational symbols are parsed; the base class file is converted from a pre-written expression syntax file; the second subclass file is used to parse the relational symbols from the monitoring rule expression according to the syntax rules corresponding to the expression syntax file.

[0169] Optionally, the acquisition unit 402 is specifically used for:

[0170] Based on the latest data in the target data table, execute the monitoring value calculation task corresponding to each of the first monitoring objects to obtain the first object parameters for each of the first monitoring objects.

[0171] The data monitoring device provided in this application includes: a traversal unit, configured to traverse the monitoring rule expression associated with the target data table when an update to the target data table is detected, and determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used to generate monitoring information based on the latest data of at least one data table; the at least one data table includes the target data table; an acquisition unit, configured to acquire a first object parameter of each first monitoring object based on the latest data of the target data table and store it in a preset storage area; and a generation unit, configured to, for each first monitoring object, if it is determined that the monitoring rule expression includes the first monitoring object and at least one second monitoring object, and the preset storage area stores the second object parameter of each second monitoring object. Based on the monitoring rule expression, the first object parameter, and each of the second object parameters, monitoring information corresponding to the monitoring rule expression is generated; the second monitoring object corresponds to other data tables besides the target data table; the first determining unit is used to determine whether to trigger an alarm based on the monitoring information. Thus, in cases where a monitoring rule may involve multiple monitoring objects in multiple data tables, and each monitoring object in each data table may be used by multiple monitoring rules, by determining multiple first monitoring objects when the target data table is updated, and by obtaining the first object parameters of each first monitoring object based on the latest data of the target data table and storing them in a preset storage area, the first object parameters can be reused by different monitoring rules, reducing the amount of repetitive calculation in the monitoring task.

[0172] Corresponding to the data monitoring method described above, based on the same technical concept, this application also provides an electronic device for executing the data monitoring method provided above. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0173] like Figure 5As shown, electronic devices can vary considerably due to differences in configuration or performance. They may include one or more processors 501 and memories 502, with the memory 502 storing one or more application programs or data. The memory 502 can be temporary or persistent storage. The application programs stored in the memory 502 may include one or more modules (not shown), each module including a series of computer-executable instructions within the electronic device. Furthermore, the processor 501 may be configured to communicate with the memory 502, executing the series of computer-executable instructions stored in the memory 502 on the electronic device. The electronic device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input / output interfaces 505, one or more keyboards 506, etc.

[0174] In one specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0175] Upon detecting an update to the target data table, the monitoring rule expressions associated with the target data table are traversed to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expressions; the monitoring rule expressions are used to generate monitoring information based on the latest data of at least one data table; the at least one data table includes the target data table;

[0176] Based on the latest data in the target data table, obtain the first object parameter for each of the first monitored objects and store it in a preset storage area;

[0177] For each of the first monitored objects, if it is determined that the monitoring rule expression includes the first monitored object and at least one second monitored object, and the preset storage area stores the second object parameters of each second monitored object, then monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameters, and each second object parameter; the second monitored object corresponds to other data tables besides the target data table.

[0178] Based on the monitoring information, determine whether to trigger an alarm.

[0179] This specification provides an embodiment of a computer-readable storage medium as follows:

[0180] Corresponding to the data monitoring method described above, based on the same technical concept, this application also provides a computer-readable storage medium.

[0181] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed by a processor, implement the following process:

[0182] Upon detecting an update to the target data table, the monitoring rule expressions associated with the target data table are traversed to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expressions; the monitoring rule expressions are used to generate monitoring information based on the latest data of at least one data table; the at least one data table includes the target data table;

[0183] Based on the latest data in the target data table, obtain the first object parameter for each of the first monitored objects and store it in a preset storage area;

[0184] For each of the first monitored objects, if it is determined that the monitoring rule expression includes the first monitored object and at least one second monitored object, and the preset storage area stores the second object parameters of each second monitored object, then monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameters, and each second object parameter; the second monitored object corresponds to other data tables besides the target data table.

[0185] Based on the monitoring information, determine whether to trigger an alarm.

[0186] It should be noted that the embodiments of computer-readable storage media in this specification and the embodiments of data monitoring methods in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0187] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0188] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, this specification can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0190] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0192] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0193] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0194] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0195] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0196] The embodiments of this application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0197] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0198] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A data monitoring method, characterized in that, include: If an update to the target data table is detected, the monitoring rule expression associated with the target data table is traversed to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expression; The monitoring rule expression is used to generate monitoring information based on the latest data from at least one data table; The at least one data table includes the target data table; Based on the latest data in the target data table, obtain the first object parameter for each of the first monitored objects and store it in a preset storage area; The step of obtaining the first object parameter of each first monitoring object based on the latest data of the target data table includes: executing a monitoring value calculation task corresponding to each first monitoring object based on the latest data of the target data table to obtain the first object parameter of each first monitoring object. For each of the first monitored objects, if it is determined that the monitoring rule expression includes the first monitored object and at least one second monitored object, and the preset storage area stores the second object parameters of each second monitored object, then monitoring information corresponding to the monitoring rule expression is generated according to the monitoring rule expression, the first object parameters, and each second object parameter; the second monitored object corresponds to other data tables besides the target data table. Based on the monitoring information, determine whether to trigger an alarm.

2. The method according to claim 1, characterized in that, The number of monitoring rule expressions associated with the target data table is multiple; the step of traversing the monitoring rule expressions associated with the target data table to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expressions includes: For each of the monitoring rule expressions, traverse the monitoring rule expressions and extract at least one object identifier; In the at least one object identifier, the object identifier associated with the target data table is determined as the first object identifier; Each monitoring object corresponding to the first object identifier is identified as the first monitoring object.

3. The method according to claim 2, characterized in that, After obtaining the first object parameter of each first monitored object based on the latest data of the target data table and storing it in a preset storage area, and before generating the monitoring information corresponding to the monitoring rule expression based on the monitoring rule expression, the first object parameter, and each second object parameter, the method further includes: In the at least one object identifier, an object identifier other than the first object identifier is determined as the second object identifier; Based on the second object identifier, query the preset storage area to see if there is a corresponding second object parameter.

4. The method according to claim 1, characterized in that, The step of generating monitoring information corresponding to the monitoring rule expression based on the monitoring rule expression, the first object parameter, and each of the second object parameters includes: Iterate through the monitoring rule expression and parse the relational symbols; Based on the first monitored object, the first object parameter, each second monitored object, each second object parameter, and the parsing result of the relational symbol, the monitoring information corresponding to the monitoring rule expression is generated.

5. The method according to claim 2, characterized in that, The step of traversing the monitoring rule expression and extracting at least one object identifier includes: By running the first subclass file that inherits from the base class file, the monitoring rule expression is traversed and at least one object identifier is extracted; the base class file is converted from a pre-written expression syntax file; the first subclass file is used to extract the object identifier from the monitoring rule expression according to the syntax rules corresponding to the expression syntax file.

6. The method according to claim 4, characterized in that, The step of traversing the monitoring rule expression and parsing the relational symbols includes: By running a second subclass file that inherits from the base class file, the monitoring rule expression is traversed, the relational symbols are extracted, the calculation methods corresponding to the relational symbols are parsed, and the monitoring objects corresponding to the relational symbols are parsed; the base class file is converted from a pre-written expression syntax file; the second subclass file is used to parse the relational symbols from the monitoring rule expression according to the syntax rules corresponding to the expression syntax file.

7. A data monitoring device, characterized in that, The device includes: The traversal unit is used to traverse the monitoring rule expression associated with the target data table when an update to the target data table is detected, and to determine multiple first monitoring objects corresponding to the target data table in the monitoring rule expression; the monitoring rule expression is used to generate monitoring information based on the latest data of at least one data table; the at least one data table includes the target data table. The acquisition unit is used to acquire the first object parameter of each first monitoring object according to the latest data of the target data table and store it in a preset storage area; the acquisition unit is specifically used to: execute the monitoring value calculation task corresponding to each first monitoring object according to the latest data of the target data table to obtain the first object parameter of each first monitoring object; The generation unit is configured to, for each of the first monitored objects, if it is determined that the monitoring rule expression includes the first monitored object and at least one second monitored object, and the preset storage area stores the second object parameters of each second monitored object, generate monitoring information corresponding to the monitoring rule expression based on the monitoring rule expression, the first object parameters, and each second object parameter; the second monitored object corresponds to other data tables besides the target data table; The first determining unit is used to determine whether to trigger an alarm based on the monitoring information.

8. An electronic device, characterized in that, The device includes: A processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the data monitoring method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store computer-executable instructions that, when executed by a processor, implement the data monitoring method as described in any one of claims 1-6.

Images