Database data alarm method and device and storage medium

Abstract

The invention discloses a data alarm method and device for a database and a storage medium, a system information table is used for storing the number of data contained in each data table in a database instance, and the method comprises the steps that the number of data corresponding to the data tables in the database instance is updated according to data modification operation, and obtaining the data quantity contained in each data table in the system information table corresponding to each database instance, obtaining a data snapshot table corresponding to each database instance, and sending the data snapshot table to an alarm detection end. The alarm detection end detects whether the data quantity of each data table in the data snapshot table meets the data alarm rule or not according to the data alarm rule, and if yes, alarm information is generated. Therefore, during data fluctuation detection, data quantity statistics is realized by querying the system information table to generate the data snapshot table, the problem of resource occupation caused by directly scanning the data table of the database instance is avoided, and the response speed of data alarm detection is improved.

Description

Technical Field

[0001] The present application relates to the technical field of database management, and in particular to a method, device and storage medium for data alarm of a database. Background Art

[0002] With the development of the big data era, all walks of life are generating huge amounts of data fragments every day, and big data needs to be processed by a big data platform with a distributed computing architecture. The big data platform can realize functions such as data integration, data processing, data storage, data analysis, and visualization of massive data.

[0003] How to quickly detect abnormal fluctuations in the data collected and calculated in the big data platform, such as sudden increase, sudden decrease, long-term unchanged data, etc., is a problem that needs to be solved urgently in this field. Summary of the invention

[0004] In order to solve the above technical problems, the present application at least provides a database data alarm method, device and storage medium.

[0005] In a first aspect, the present application provides a data alarm method for a database, which is applied to a big data platform. The big data platform includes multiple database instances, each of which contains multiple data tables. A system information table is provided for each database instance, and the system information table is used to store the amount of data contained in each data table in the database instance. The method includes: detecting whether there is a data modification operation on the database instance, and if so, updating the amount of data corresponding to the data table in the system information table based on the data modification operation; obtaining the amount of data contained in each data table in the system information table corresponding to each database instance, and obtaining a data snapshot table corresponding to each database instance; sending the data snapshot table to an alarm detection end, so that the alarm detection end obtains data alarm rules matching the data snapshot table, the data alarm rules including a data quantity fluctuation threshold and / or a data quantity threshold, detecting whether the amount of data in each data table in the data snapshot table meets the data alarm rule, and if so, generating an alarm message.

[0006] In one embodiment, before obtaining the amount of data contained in each data table in the system information table corresponding to each database instance and obtaining the data snapshot table corresponding to each database instance, it also includes: obtaining an update interval threshold; calculating the time interval between the last update time of the data snapshot table and the current time; if the time interval is greater than the update interval threshold, executing the step of obtaining the amount of data contained in each data table in the system information table corresponding to each database instance and obtaining the data snapshot table corresponding to each database instance.

[0007] In one embodiment, obtaining the update interval threshold includes: counting historical data volume changes of the database instance based on a data snapshot table corresponding to the database instance within a preset historical time period; and setting the update interval threshold based on the historical data volume changes.

[0008] In one embodiment, based on the data snapshot table corresponding to the database instance within a preset historical time period, the historical data volume changes of the database instance are counted, including: obtaining the data snapshot tables obtained by the database instance in multiple basic statistical periods, respectively, where the basic statistical period is composed of multiple periodic time periods; performing statistics on the data snapshot tables obtained in each basic statistical period to obtain the historical data volume changes corresponding to each periodic time period; and / or, based on the current moment, dividing the adjacent time periods, obtaining the data snapshot tables obtained by the database instance in the adjacent time periods; performing statistics on the data snapshot tables obtained in the adjacent time periods to obtain the historical data volume changes corresponding to the adjacent time periods.

[0009] In one embodiment, an update interval threshold is set based on the change in the amount of historical data, including: determining the periodic time period in which the current moment is located, and setting the update interval threshold based on the change in the amount of historical data corresponding to the periodic time period; or, setting the update interval threshold based on the change in the amount of historical data corresponding to an adjacent time period; or, determining a basic interval threshold corresponding to the current moment based on the change in the amount of historical data corresponding to the periodic time period in which the current moment is located, and determining an interval adjustment value corresponding to the current moment based on the change in the amount of historical data corresponding to an adjacent time period; and modifying the basic interval threshold based on the interval adjustment value to obtain the update interval threshold corresponding to the current moment.

[0010] A second aspect of the present application provides a data alarm method for a database, which is applied to an alarm detection end, and the method includes: obtaining a data snapshot table corresponding to each database instance sent by a big data platform; wherein the big data platform includes multiple database instances, and the database instance contains multiple data tables. The database instance stores the amount of data contained in each data table in the database instance through a system information table. The big data platform detects whether there is a data modification operation on the database instance. If there is a data modification operation, the amount of data corresponding to the data table in the system information table is updated based on the data modification operation; the amount of data contained in each data table in the system information table corresponding to each database instance is obtained to obtain a data snapshot table corresponding to each database instance; querying data alarm rules matching the data snapshot table, the data alarm rules including a data quantity fluctuation threshold and / or a data quantity threshold; detecting whether the amount of data in each data table in the data snapshot table meets the data alarm rule, and if so, generating an alarm message.

[0011] In one embodiment, querying data alarm rules that match the data snapshot table includes: based on the data snapshot table corresponding to the database instance within a preset historical time period, counting the historical data volume changes of the database instance; based on the historical data volume changes, setting a data quantity fluctuation threshold and / or a data quantity threshold to generate a data alarm rule that matches the current data snapshot table.

[0012] In one embodiment, querying data alarm rules that match the data snapshot table includes: obtaining a candidate rule set containing multiple candidate rules, each candidate rule corresponding to a limited rule effectiveness period; and obtaining a data statistical period corresponding to the data snapshot table; selecting candidate rules whose rule effectiveness period coincides with the data statistical period from the candidate rule set, to obtain data alarm rules that match the data snapshot table.

[0013] According to a third aspect of the present application, there is provided a data alarm device for a database, which is deployed on a big data platform. The device includes: a quantity update module, which is used to detect whether there is a data modification operation on a database instance. If there is a data modification operation, the data quantity corresponding to the data table in the system information table is updated based on the data modification operation; a data snapshot module, which is used to obtain the data quantity contained in each data table in the system information table corresponding to each database instance, and obtain a data snapshot table corresponding to each database instance; wherein the system information table is used to store the data quantity contained in each data table in the database instance; a data sending module, which is used to send the data snapshot table to an alarm detection end, so that the alarm detection end obtains a data alarm rule matching the data snapshot table, the data alarm rule including a data quantity fluctuation threshold and / or a data quantity threshold, and detects whether the data quantity of each data table in the data snapshot table satisfies the data alarm rule. If so, an alarm message is generated.

[0014] The fourth aspect of the present application provides a data alarm device for a database, which is deployed at an alarm detection end, and the device includes: a data acquisition module, which is used to obtain a data snapshot table corresponding to each database instance sent by a big data platform; wherein the big data platform includes multiple database instances, and the database instance contains multiple data tables. The database instance stores the amount of data contained in each data table in the database instance through a system information table. The big data platform detects whether there is a data modification operation on the database instance. If there is a data modification operation, the amount of data corresponding to the data table in the system information table is updated based on the data modification operation; the amount of data contained in each data table in the system information table corresponding to each database instance is obtained, and a data snapshot table corresponding to each database instance is obtained; a rule query module, which is used to query data alarm rules matching the data snapshot table, the data alarm rules including a data quantity fluctuation threshold and / or a data quantity threshold; an alarm judgment module, which is used to detect whether the amount of data in each data table in the data snapshot table meets the data alarm rule, and if so, an alarm information is generated.

[0015] A fifth aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the data alarm method of the above-mentioned database.

[0016] In a sixth aspect, the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the data alarm method of the above-mentioned database is implemented.

[0017] The above scheme utilizes a system information table to store the amount of data contained in each data table in a database instance. The system information table can automatically update the amount of data corresponding to the data table in the database instance according to the data modification operation, and then directly obtains the amount of data contained in each data table in the system information table corresponding to each database instance, obtains a data snapshot table corresponding to each database instance, and sends the data snapshot table to the alarm detection end, so that the alarm detection end obtains data alarm rules matching the data snapshot table. The data alarm rules include a data quantity fluctuation threshold and / or a data quantity threshold, and detects whether the data quantity of each data table in the data snapshot table meets the data alarm rules. If so, an alarm message is generated. Therefore, when detecting data fluctuations, data quantity statistics are realized by querying the system information table to generate a data snapshot table, thereby avoiding the problem of resource occupation caused by directly scanning the data table of the database instance and improving the response speed of data alarm detection.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0021] Figure 2 is a flow chart of a data alarm method for a database shown in an exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram of obtaining a data snapshot table according to an exemplary embodiment of the present application;

[0023] Figure 4 is a flow chart of a data alarm method for a database shown in another exemplary embodiment of the present application;

[0024] Figure 5 is a schematic diagram of data alarm detection shown in an exemplary embodiment of the present application;

[0025] Figure 6 is a block diagram of a data alarm device for a database shown in an exemplary embodiment of the present application;

[0026] Figure 7 is a block diagram of a data alarm device for a database shown in another exemplary embodiment of the present application;

[0027] Figure 8 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0028] Fig. 9 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0029] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0030] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0031] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0032] The applicant has discovered that in the prior art, when database data fluctuation detection is required, a full table scan is performed on the disk interval occupied by the target actual table on the disk. When the amount of data in the target actual table is too large, the operating system needs to perform memory scheduling or central processing unit (CPU) scheduling, which seriously affects the response speed of data fluctuation detection.

[0033] Based on this, the present application provides a database data alarm method, device and storage medium.

[0034] The following is a description of the database data alarm method provided in the embodiment of the present application.

[0035] Please refer to Figure 1, which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include a big data platform 110 and an alarm detection terminal 120, and the big data platform 110 and the alarm detection terminal 120 are connected to each other in communication.

[0036] The big data platform 110 includes multiple database instances.

[0037] The alarm detection end 120 may be a server or a terminal.

[0038] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto.

[0039] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0040] It should be noted that the big data platform 110 and the alarm detection terminal 120 can be set up independently, or the big data platform 110 and the alarm detection terminal 120 can be set up centrally, and this application does not limit this.

[0041] In one example, the big data platform 110 obtains the amount of data contained in each data table in the system information table corresponding to each database instance, and obtains a data snapshot table corresponding to each database instance; the data snapshot table is sent to the alarm detection terminal 120, so that the alarm detection terminal 120 obtains data alarm rules that match the data snapshot table. The data alarm rules include data quantity fluctuation thresholds and / or data quantity thresholds, and detects whether the data quantity of each data table in the data snapshot table meets the data alarm rules. If so, an alarm message is generated.

[0042] In the big data platform, the amount of data contained in each data table in the database instance is stored based on the system information table. When a data modification operation is detected, the system information table can promptly update the corresponding data quantity in the data table. Therefore, when detecting data fluctuations, data fluctuation detection can be achieved by querying the data snapshot table instead of directly scanning the system information table of the database instance, thereby reducing the use of CPU, memory, disk and other resources.

[0043] For example, a database instance has 10 billion data and 100 tables. The existing method of scanning and counting all the data in each table in the database instance will basically occupy all the resources, and the response time is expected to exceed 10 seconds or even more than 1 minute. However, for the same hardware resources and data volume, by querying the system information table of each database instance, the data volume of each data table can be obtained, which can be responded within 1 second and occupies very little hardware resources.

[0044] See also Figure 2 , Figure 2 is a flowchart of a database data alarm method shown in an exemplary embodiment of the present application. The database data alarm method can be applied to Figure 1 The implementation environment shown is specifically implemented by the big data platform in the implementation environment.

[0045] like Figure 2 As shown, the database data alarm method at least includes steps S210 to S230, which are described in detail as follows:

[0046] Step S210: Detect whether there is a data modification operation on the database instance. If there is a data modification operation, update the data quantity corresponding to the data table in the system information table based on the data modification operation.

[0047] The big data platform includes multiple database instances, each of which contains multiple data tables. A system information table is provided for each database instance, and the system information table is used to store the amount of data contained in each data table in the database instance.

[0048] For example, if the database instance is implemented based on the clickhouse database, the system table systm.tables in the clickhouse database can be used to store the amount of data contained in each data table. Specifically, a data quantity field corresponding to each data table is added to systm.tables to store the data quantity of each data table according to the data quantity field.

[0049] After detecting that a data modification operation has occurred on the database instance, the data quantity corresponding to the data table in the system information table is updated according to the data modification operation.

[0050] Specifically, the data modification operation includes operations such as data storage, data modification, data deletion, etc. After detecting that the database instance has operations such as data storage, data modification, data deletion, etc., the system information table corresponding to the database instance is updated.

[0051] For example, the data table corresponding to the data modification operation and the change in the data quantity are determined, and then the data quantity of the data table in the system information table is modified to achieve real-time synchronization of the system information table.

[0052] Step S220: Obtain the amount of data contained in each data table in the system information table corresponding to each database instance, and obtain a data snapshot table corresponding to each database instance; wherein a system information table is set for each database instance, and the system information table is used to store the amount of data contained in each data table in the database instance.

[0053] The amount of data contained in each data table in the system information table corresponding to each database instance is obtained, and a data snapshot table corresponding to each database instance is obtained.

[0054] For example, see Figure 3 , Figure 3 This is a schematic diagram of obtaining a data snapshot table according to an exemplary embodiment of the present application, as shown in FIG. Figure 3 As shown, the big data platform includes n database instances, each database instance stores multiple data tables and system information tables. The big data platform queries the system information table of the database instance through a query module to obtain the number of data contained in each data table in the database instance, and updates the query results to the data snapshot table corresponding to the database instance to obtain snapshots of the number of data contained in each data table of the database instance at different times.

[0055] For example, the system information table systm.tables in the clickhouse database instance to be queried is queried through the structured query language (SQL) (for example, SQL = select database, name, total_rows from system.tables), and the real-time data quantity of all databases and tables of each clickhouse instance is recorded, where the data quantity can be the number of data rows corresponding to each clickhouse instance and / or data table. The query results are updated to the data snapshot table corresponding to the database instance.

[0056] Optionally, the data snapshot table may be stored in an independent cluster using a storage tool such as MySQL, for example, in a designated snapshot data storage medium.

[0057] Exemplarily, the storage content of the data snapshot table includes, but is not limited to, information such as the database instance address, the data table engine, the number of data rows contained in the data table, etc., which is not limited in this application.

[0058] Step S230: Send the data snapshot table to the alarm detection end so that the alarm detection end obtains data alarm rules that match the data snapshot table. The data alarm rules include data quantity fluctuation thresholds and / or data quantity thresholds. Check whether the data quantity of each data table in the data snapshot table meets the data alarm rules. If so, generate alarm information.

[0059] The big data platform may periodically send a data snapshot table to the alarm detection end; or the alarm detection end may send a snapshot table acquisition request to the big data platform, and the big data platform responds to the snapshot table acquisition request and sends the latest data snapshot table of each current database instance to the alarm detection end.

[0060] The alarm detection end detects whether there is abnormal data fluctuation in each database instance based on the data snapshot table. The specific detection steps can be found in the subsequent related step description with the alarm detection end as the execution body, which will not be repeated here.

[0061] Next, some embodiments of the present application are described in detail:

[0062] In some embodiments, before obtaining the amount of data contained in each data table in the system information table corresponding to each database instance and obtaining the data snapshot table corresponding to each database instance, it also includes: obtaining an update interval threshold; calculating the time interval between the last update time of the data snapshot table and the current time; if the time interval is greater than the update interval threshold, executing the step of obtaining the amount of data contained in each data table in the system information table corresponding to each database instance and obtaining the data snapshot table corresponding to each database instance.

[0063] The update interval threshold may be preset or flexibly determined according to actual conditions.

[0064] Exemplarily, based on the data snapshot table corresponding to the database instance within a preset historical time period, the historical data volume changes of the database instance are counted; based on the historical data volume changes, an update interval threshold is set.

[0065] Example 1: Get the data snapshot tables obtained by the database instance in multiple basic statistical periods. The basic statistical period consists of multiple periodic time periods. Perform statistics on the data snapshot tables obtained in each basic statistical period to obtain the historical data volume changes corresponding to each periodic time period. Then, determine the periodic time period at the current moment, and set the update interval threshold based on the historical data volume changes corresponding to the periodic time period.

[0066] For example, taking days as the basic statistical period, the basic statistical period is divided into multiple periodic time periods (such as 24 periodic time periods, each periodic time period corresponding to 1 hour). For any periodic time period, statistics are collected on the data snapshot table corresponding to the database instance in the same periodic time period of m days to obtain the change in the amount of historical data corresponding to the periodic time period, so as to set the update interval threshold at the current moment according to the change in the amount of historical data in the periodic time period at the current moment.

[0067] For example, statistics are collected on the data snapshot table of database instance ck1 on January 1, January 2, and January 3 to obtain the historical data volume changes in each periodic time period (such as the periodic time period includes 0:00 to 1:00, 1:00 to 2:00, ..., 23:00 to 24:00, a total of 24 periodic time periods). According to the historical data volume changes in each periodic time period of the above three basic statistical periods, the update interval threshold for each periodic time period on January 4 is set, and the system information table of database instance ck1 is queried according to the update interval threshold. If the current time is between 23:00 and 24:00, the update interval threshold obtained by setting the historical data volume changes from 23:00 to 24:00 obtained by the above statistics is used as the update interval threshold corresponding to the current time.

[0068] Example 2: Based on the current time, the adjacent time period is divided, and the data snapshot table obtained by the database instance in the adjacent time period is obtained; the data snapshot table obtained in the adjacent time period is counted to obtain the historical data volume change corresponding to the adjacent time period. Then, based on the historical data volume change corresponding to the adjacent time period, the update interval threshold is set.

[0069] For example, for the current time t1, the data snapshot table corresponding to the database instance between the adjacent time periods t1-a to t1 is counted to obtain the historical data volume change corresponding to the adjacent time period, so as to set the update interval threshold of the current time t1 according to the historical data volume change. Where a is the preset statistical time length.

[0070] For example, the current time is the time when the system information table of the database instance ck1 was most recently queried. If the current time is 9:30 and the value of a is 10 minutes, the data snapshot table corresponding to the database instance ck1 between the adjacent time periods of 9:20 and 9:30 is counted to obtain the historical data volume changes in the adjacent time periods. The update interval threshold corresponding to the current time of 9:30 is set based on the historical data volume changes, and the system information table of the database instance ck1 is queried next time based on the update interval threshold.

[0071] Example 3, based on the data snapshot table corresponding to the database instance in the preset historical time period, the historical data volume change of the database instance is counted, including: obtaining the data snapshots obtained by the database instance in multiple basic statistical periods respectively; performing statistics on the data snapshot tables obtained in multiple basic statistical periods to obtain a first data volume change; and, based on the current moment, obtaining adjacent time periods, the data snapshot table obtained by the database instance in the adjacent time period; performing statistics on the data snapshot table obtained in the adjacent time period to obtain a second data volume change. Further, based on the first data volume change, the basic interval threshold corresponding to the periodic time period at the current moment is determined; and, based on the second data volume change, the interval adjustment value corresponding to the current moment is determined; based on the interval adjustment value, the basic interval threshold is modified to obtain an update interval threshold corresponding to the current moment.

[0072] That is, the method of Example 1 is used to perform statistics on multiple basic statistical periods to obtain the changes in the amount of historical data in each periodic time period. According to the changes in the amount of historical data in each periodic time period, the basic interval threshold of each periodic time period is set respectively. Then, the method of Example 2 is used to determine the interval adjustment value after each system information query is executed. The basic interval threshold is modified in combination with the interval adjustment value to obtain the update interval threshold. The system information table of the database instance is queried next time according to the update interval threshold.

[0073] Optionally, the specific statistical method for the change in the amount of historical data can be flexibly determined according to the amount of data in the database instance, the amount of remaining available resources of the execution subject, and the like.

[0074] Furthermore, historical changes in data volume may include data volume change trends, such as an increase in data volume or a decrease in data volume. For example, if the data volume increases, the update interval threshold is shortened; if the data volume decreases, the update interval threshold is extended. Historical changes in data volume may also include the size of data volume fluctuations. For example, if the data volume fluctuations are large, the update interval threshold is shortened; if the data volume fluctuations are small, the update interval threshold is extended.

[0075] Through the above embodiments, the overhead of computing resources and storage resources can be reduced while ensuring the accuracy of the data snapshot table.

[0076] See also Figure 4 , Figure 4 is a flowchart of a data alarm method for a database shown in another exemplary embodiment of the present application. The data alarm method for the database can be applied to Figure 1 The implementation environment shown is specifically executed by the alarm detection terminal in the implementation environment.

[0077] like Figure 4 As shown, the database data alarm method at least includes steps S410 to S430, which are described in detail as follows:

[0078] Step S410: Obtain the data snapshot table corresponding to each database instance sent by the big data platform; wherein the big data platform detects whether there is a data modification operation on the database instance, and if so, updates the number of data corresponding to the data table in the system information table based on the data modification operation; obtains the number of data contained in each data table in the system information table corresponding to each database instance, and obtains the data snapshot table corresponding to each database instance.

[0079] The big data platform includes multiple database instances, each of which contains multiple data tables. The database instance stores the amount of data contained in each data table in the database instance through a system information table.

[0080] Step S420: querying the data alarm rule that matches the data snapshot table, where the data alarm rule includes a data quantity fluctuation threshold and / or a data quantity threshold.

[0081] For example, the candidate rule set contains multiple candidate rules, different candidate rules are applicable to different database instances, and the candidate rules matching the database instance are selected from the candidate rule set and used as the data alarm rules matching the data snapshot table.

[0082] For another example, data alarm rules are generated in real time, such as by statistically analyzing the historical data volume changes of the database instance based on the data snapshot table recorded by the database instance during the historical time period, and generating data quantity fluctuation thresholds and / or data quantity thresholds based on the historical data volume changes to obtain data alarm rules.

[0083] Step S430: Check whether the data quantity of each data table in the data snapshot table meets the data alarm rule. If yes, generate alarm information.

[0084] Specifically, according to the data quantity fluctuation threshold and / or data quantity threshold corresponding to the data alarm rule, the data quantity fluctuation and / or data quantity corresponding to the acquired data snapshot table is detected to determine whether the rule judgment conditions corresponding to the data quantity fluctuation threshold and / or data quantity threshold in the data alarm rule are met. If so, an alarm message is generated.

[0085] For example, it is determined whether the data quantity corresponding to the data snapshot table is greater than a data quantity threshold, or it is determined whether the data quantity fluctuation corresponding to the data snapshot table is less than a data quantity fluctuation threshold, and the like.

[0086] Next, some embodiments of the present application are described in detail:

[0087] In some embodiments, querying data alarm rules that match the data snapshot table includes: based on the data snapshot table corresponding to the database instance within a preset historical time period, counting the historical data volume changes of the database instance; based on the historical data volume changes, setting a data quantity fluctuation threshold and / or a data quantity threshold to generate a data alarm rule that matches the current data snapshot table.

[0088] The statistical method of the change in the amount of historical data can be found in Examples 1, 2 and 3 above, and will not be elaborated here.

[0089] According to the changes in historical data volume, set the data volume fluctuation threshold and / or data volume threshold to generate data alarm rules that match the current data snapshot table.

[0090] Take the historical data volume changes including data volume fluctuations as an example:

[0091] For example, the weighted value of the data quantity fluctuation values ​​corresponding to the same period time period of multiple basic statistical periods is used as the data quantity fluctuation threshold, wherein the weight used in the weighted calculation can be pre-set, such as the weight value of the period time period corresponding to each basic statistical period is the same; the weight used in the weighted calculation can also be flexibly determined according to the time interval between the basic statistical period and the current moment, such as the shorter the time interval, the greater the weight, and the longer the time interval, the smaller the weight.

[0092] For another example, based on the data volume fluctuation value corresponding to the adjacent time period, the data volume fluctuation threshold corresponding to the current moment is calculated. For example, a preset fluctuation value mapping formula is used to map the data volume fluctuation value corresponding to the adjacent time period to the data volume fluctuation threshold corresponding to the current moment.

[0093] For another example, data snapshots obtained by the database instance in multiple basic statistical periods are obtained; the data snapshot tables obtained in multiple basic statistical periods are counted to obtain a first data volume change; and, based on the current moment, adjacent time periods are obtained, and the data snapshot tables obtained by the database instance in adjacent time periods are counted; the data snapshot tables obtained in adjacent time periods are counted to obtain a second data volume change. Further, based on the first data volume change, a basic quantity fluctuation threshold corresponding to the periodic time period at the current moment is determined; and, based on the second data volume change, a quantity fluctuation adjustment value corresponding to the current moment is determined; the basic quantity fluctuation threshold is modified based on the quantity fluctuation adjustment value to obtain a data quantity fluctuation threshold corresponding to the current moment.

[0094] Take the historical data volume changes including data volume values ​​as an example:

[0095] For example, the historical data volume change may also include the maximum data volume and / or minimum data volume in a preset historical time period, such as taking the weighted value of the maximum data volume corresponding to the same period time period of each basic statistical period as the data volume threshold. Alternatively, taking the maximum data volume and / or minimum data volume corresponding to the adjacent time period corresponding to the current moment as the data volume threshold.

[0096] Exemplarily, based on the data snapshot table corresponding to the database instance in the preset historical time period, the historical data volume change of the database instance is counted, including: obtaining the data snapshots obtained by the database instance in multiple basic statistical periods respectively; performing statistics on the data snapshot tables obtained in multiple basic statistical periods to obtain a first data volume change; and, based on the current moment, obtaining adjacent time periods, the data snapshot table obtained by the database instance in the adjacent time period; performing statistics on the data snapshot table obtained in the adjacent time period to obtain a second data volume change. Further, based on the first data volume change, determining the basic quantity threshold corresponding to the time period at the current moment; and, based on the second data volume change, determining the quantity adjustment value corresponding to the current moment; modifying the basic quantity threshold based on the quantity adjustment value to obtain the data quantity threshold corresponding to the current moment.

[0097] Based on the data quantity fluctuation threshold and / or data quantity threshold calculated in the above embodiments, a data alarm rule matching the current data snapshot table is automatically generated.

[0098] Of course, the data alarm rules may also be pre-entered by the management personnel, and this application does not limit this.

[0099] In some implementations, the data alarm rules and the data snapshot table are stored in the same storage device to facilitate associated query.

[0100] In some embodiments, the data alarm rules include but are not limited to the database name of the applicable database instance, the applicable data table name, the data quantity fluctuation threshold and / or the data quantity threshold, the rule effectiveness period, the enabled status and other information.

[0101] In some embodiments, querying data alarm rules that match the data snapshot table includes: obtaining a candidate rule set containing multiple candidate rules, each candidate rule corresponding to a limited rule effectiveness period; and obtaining a data statistical period corresponding to the data snapshot table; selecting candidate rules whose rule effectiveness period coincides with the data statistical period from the candidate rule set, to obtain data alarm rules that match the data snapshot table.

[0102] For example, see Figure 5 , Figure 5 This is a schematic diagram of data alarm detection shown in an exemplary embodiment of the present application, as shown in Figure 5As shown, the snapshot data storage medium stores the data snapshot table obtained by scanning at each moment, and the alarm detection end obtains the data snapshot table in the historical time period from the snapshot data storage medium to perform statistics on the data snapshot table in the historical time period, and generates a data alarm rule according to the statistical result; or, in response to the rule input operation of the management personnel, obtains the data alarm rule; according to the data alarm rule, judges the database and / or data table with abnormalities in the data snapshot table in the preset detection time period, specifically: calculates the data quantity fluctuation and / or data quantity corresponding to the data snapshot table in the detection time period, and according to the data alarm rule matching the data snapshot table, detects the database information and / or data table information whose data quantity fluctuation and / or data quantity meet the alarm condition, so as to generate the corresponding alarm information. Wherein, the historical time period is before the detection time period.

[0103] Furthermore, an alarm tool corresponding to the alarm platform is used to forward or display the alarm information, and the number of the alarm tools may be one or more.

[0104] In some implementations, the data snapshot table of each database instance, the data alarm rule matched by the data snapshot table, and the alarm information are visualized.

[0105] The data alarm method for a database provided by the present application uses a system information table to store the amount of data contained in each data table in a database instance. The system information table can automatically update the amount of data corresponding to the data table in the database instance according to the data modification operation, and then directly obtains the amount of data contained in each data table in the system information table corresponding to each database instance to obtain a data snapshot table corresponding to each database instance, and sends the data snapshot table to an alarm detection terminal so that the alarm detection terminal obtains a data alarm rule matching the data snapshot table. The data alarm rule includes a data quantity fluctuation threshold and / or a data quantity threshold, and detects whether the data quantity of each data table in the data snapshot table satisfies the data alarm rule. If so, an alarm message is generated. Therefore, when data fluctuation is detected, data quantity statistics are realized by querying the system information table to generate a data snapshot table, thereby avoiding the problem of resource occupation caused by directly scanning the data table of the database instance and improving the response speed of data alarm detection.

[0106] Figure 6 FIG. 1 is a block diagram of a data alarm device for a database shown in an exemplary embodiment of the present application. The data alarm device for the database is deployed on a big data platform. Figure 6 As shown, the exemplary database data alarm device 600 includes: a quantity update module 610, a data snapshot module 620 and a data sending module 630. Specifically:

[0107] The quantity updating module 610 is used to detect whether there is a data modification operation in the database instance, and if there is a data modification operation, update the data quantity corresponding to the data table in the system information table based on the data modification operation;

[0108] The data snapshot module 620 is used to obtain the amount of data contained in each data table in the system information table corresponding to each database instance, and obtain the data snapshot table corresponding to each database instance; wherein the system information table is used to store the amount of data contained in each data table in the database instance;

[0109] The data sending module 630 is used to send the data snapshot table to the alarm detection end so that the alarm detection end obtains the data alarm rules that match the data snapshot table. The data alarm rules include data quantity fluctuation thresholds and / or data quantity thresholds. It detects whether the data quantity of each data table in the data snapshot table meets the data alarm rules. If so, an alarm message is generated.

[0110] Figure 7 FIG. 1 is a block diagram of a data alarm device for a database shown in another exemplary embodiment of the present application. The data alarm device for the database is deployed at an alarm detection terminal. Figure 7 As shown, the exemplary database data alarm device 700 includes: a data acquisition module 710, a rule query module 720 and an alarm judgment module 730. Specifically:

[0111] The data acquisition module 710 is used to acquire the data snapshot table corresponding to each database instance sent by the big data platform; wherein the big data platform includes multiple database instances, and the database instance contains multiple data tables. The database instance stores the amount of data contained in each data table in the database instance through the system information table. The big data platform detects whether there is a data modification operation in the database instance. If there is a data modification operation, the amount of data corresponding to the data table in the system information table is updated based on the data modification operation; the amount of data contained in each data table in the system information table corresponding to each database instance is acquired, and the data snapshot table corresponding to each database instance is obtained;

[0112] A rule query module 720 is used to query a data alarm rule that matches the data snapshot table, where the data alarm rule includes a data quantity fluctuation threshold and / or a data quantity threshold;

[0113] The alarm judgment module 730 is used to detect whether the data quantity of each data table in the data snapshot table meets the data alarm rule, and if so, generate alarm information.

[0114] It should be noted that the data alarm device for the database provided in the above embodiment and the data alarm method for the database provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the data alarm device for the database provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0115] See also Figure 8 , Figure 8 800 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 800 includes a memory 801 and a processor 802, and the processor 802 is used to execute program instructions stored in the memory 801 to implement the steps in the data alarm method embodiment of any of the above databases. In a specific implementation scenario, the electronic device 800 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 800 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0116] Specifically, the processor 802 is used to control itself and the memory 801 to implement the steps in the data alarm method embodiment of any of the above-mentioned databases. The processor 802 can also be called a central processing unit (Central Processing Unit, CPU). The processor 802 may be an integrated circuit chip with signal processing capabilities. The processor 802 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 802 can be implemented by an integrated circuit chip.

[0117] See also Fig. 9 , Fig. 9 The computer-readable storage medium 900 stores program instructions 910 that can be executed by a processor, and the program instructions 910 are used to implement the steps in any of the above-mentioned database data alarm method embodiments.

[0118] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0119] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0120] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0121] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

Claims

1. A database data alarm method, applied to a big data platform, wherein the big data platform includes multiple database instances, and the database instance contains multiple data tables, characterized in that: A system information table is provided for each database instance, and the system information table is used to store the amount of data contained in each data table in the database instance. The method includes: Detecting whether a data modification operation exists in the database instance, and if the data modification operation exists, updating the data quantity corresponding to the data table in the system information table based on the data modification operation; Obtain the amount of data contained in each data table in the system information table corresponding to each database instance, and obtain a data snapshot table corresponding to each database instance; The data snapshot table is sent to the alarm detection end so that the alarm detection end obtains data alarm rules that match the data snapshot table. The data alarm rules include data quantity fluctuation thresholds and / or data quantity thresholds. It is detected whether the data quantity of each data table in the data snapshot table meets the data alarm rules. If so, an alarm message is generated.

2. The method according to claim 1, characterized in that Before obtaining the amount of data contained in each data table in the system information table corresponding to each database instance and obtaining the data snapshot table corresponding to each database instance, the method further includes: Get the update interval threshold; Calculate the time interval between the last update time of the data snapshot table and the current time; If the time interval is greater than the update interval threshold, the step of obtaining the amount of data contained in each data table in the system information table corresponding to each database instance to obtain the data snapshot table corresponding to each database instance is performed.

3. The method according to claim 2, characterized in that The obtaining of the update interval threshold comprises: Based on the data snapshot table corresponding to the database instance within a preset historical time period, statistics are collected on the historical data volume changes of the database instance; Based on the change in the amount of historical data, an update interval threshold is set.

4. The method according to claim 3, characterized in that The counting of historical data volume changes of the database instance based on the data snapshot table corresponding to the database instance within a preset historical time period includes: Obtain data snapshot tables obtained by the database instance in multiple basic statistical periods, where the basic statistical period consists of multiple periodic time periods; perform statistics on the data snapshot tables obtained in each basic statistical period to obtain changes in the amount of historical data corresponding to each periodic time period; And / or, obtain adjacent time periods based on the current moment, obtain a data snapshot table obtained by the database instance in the adjacent time period; perform statistics on the data snapshot table obtained in the adjacent time period to obtain changes in the amount of historical data corresponding to the adjacent time period.

5. The method according to claim 4, characterized in that The step of setting an update interval threshold based on the change in the amount of historical data includes: Determine the periodic time period in which the current moment is located, and set an update interval threshold based on the change in the amount of historical data corresponding to the periodic time period; Alternatively, an update interval threshold is set based on the change in the amount of historical data corresponding to the adjacent time periods; Alternatively, based on the change in the amount of historical data corresponding to the periodic time period in which the current moment is located, the basic interval threshold corresponding to the current moment is determined, and based on the change in the amount of historical data corresponding to the adjacent time period, the interval adjustment value corresponding to the current moment is determined; the basic interval threshold is modified based on the interval adjustment value to obtain the update interval threshold corresponding to the current moment.

6. A database data alarm method, applied to an alarm detection terminal, characterized in that: Methods include: Obtain a data snapshot table corresponding to each database instance sent by the big data platform; wherein the big data platform includes multiple database instances, the database instance contains multiple data tables, the database instance stores the amount of data contained in each data table in the database instance through a system information table, the big data platform detects whether there is a data modification operation on the database instance, and if so, updates the amount of data corresponding to the data table in the system information table based on the data modification operation; obtain the amount of data contained in each data table in the system information table corresponding to each database instance, and obtain a data snapshot table corresponding to each database instance; Querying a data alarm rule that matches the data snapshot table, wherein the data alarm rule includes a data quantity fluctuation threshold and / or a data quantity threshold; It is detected whether the data quantity of each data table in the data snapshot table meets the data alarm rule, and if so, an alarm message is generated.

7. The method according to claim 6, characterized in that The querying of the data alarm rule matching the data snapshot table includes: Based on the data snapshot table corresponding to the database instance within a preset historical time period, statistics are collected on the historical data volume changes of the database instance; Based on the historical data volume changes, a data volume fluctuation threshold and / or a data volume threshold is set to generate a data alarm rule that matches the current data snapshot table.

8. The method according to claim 6, characterized in that The querying of the data alarm rule matching the data snapshot table includes: Obtain a candidate rule set containing a plurality of candidate rules, each candidate rule corresponding to a limited rule validity period; and obtain a data statistical period corresponding to the data snapshot table; A candidate rule whose rule effectiveness period coincides with the data statistical period is selected from the candidate rule set to obtain a data alarm rule that matches the data snapshot table.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 8.

Images