Data monitoring method, device, computer equipment and storage medium
By configuring monitoring rules in Oracle database and sending them to Hive database, the degree of difference and the cause of errors are automatically calculated and the repair script is generated, and the cross-border data monitoring efficiency and timeliness are solved, automatic data repair is realized, and processing capabilities are improved .
Patent Information
- Application Number
- CN202010092311.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-02-14
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-02-14
AI Technical Summary
Traditional data monitoring methods have poor processing efficiency and timeliness in cross-store environments (such as Hive database and Oracle database), especially when the database structure is different, it requires a lot of manual participation and cannot handle abnormalities in a timely manner, resulting in serious consequences.
Configure monitoring rules in Oracle database, send them to the Hive database, obtain monitoring results within the specified time range, calculate the degree of difference, if the preset threshold is exceeded, enter the monitoring results processing model to automatically parse the cause of the error and generate a repair script to automatically repair the data.
It improves the processing efficiency and timeliness of cross-border data monitoring between Hive database and Oracle database, realizes automatic repair of error data, reduces manual intervention, and avoids serious consequences.
Smart Images

Figure CN111367754B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of system data monitoring, and in particular to a data monitoring method, device, computer equipment and storage medium. Background Art
[0002] Data monitoring is of great significance to data operations. Data monitoring can promptly detect anomalies in the data calculation process and the results of data calculation, so as to intervene in time and repair the anomalies when data anomalies occur.
[0003] Traditional data monitoring can only monitor data on a single database, or simply compare two databases with similar structures. If operational data is monitored across databases with large structural differences, such as Hive databases and Oracle databases, it is necessary to collect statistics on each database's monitoring data and then export it for processing and comparison, which requires more manual participation and has poor processing efficiency and timeliness. If anomalies between databases cannot be handled in a timely manner, serious adverse consequences will occur. Summary of the invention
[0004] Based on this, it is necessary to provide a data monitoring method, device, computer equipment and storage medium to address the above technical problems, so as to improve the processing efficiency and timeliness of cross-database data monitoring between Hive database and Oracle database.
[0005] A data monitoring method, comprising:
[0006] Configure monitoring rules in the Oracle database, where the monitoring rules are used to monitor specified data indicators;
[0007] Sending the monitoring rules to the Hive database;
[0008] Acquire a first monitoring result within a specified time range in the Oracle database according to the monitoring rule; acquire a second monitoring result within the specified time range in the Hive database according to the monitoring rule;
[0009] Importing the second monitoring result into the Oracle database, and calculating the difference between the first monitoring result and the second monitoring result;
[0010] If the difference is greater than a preset alarm threshold, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model;
[0011] Obtaining the error cause of the output of the preset monitoring result processing model;
[0012] Obtain a data repair script adapted to the cause of the error;
[0013] Repair the data of the Oracle database and / or the Hive database according to the data repair script.
[0014] A data monitoring device, comprising:
[0015] A rule configuration module is used to configure monitoring rules in an Oracle database, wherein the monitoring rules are used to monitor specified data indicators;
[0016] A rule sending module, used for sending the monitoring rules to a Hive database;
[0017] A monitoring result acquisition module is used to acquire a first monitoring result within a specified time range in the Oracle database according to the monitoring rule; and acquire a second monitoring result within the specified time range in the Hive database according to the monitoring rule;
[0018] A difference calculation module, used for importing the second monitoring result into the Oracle database, and calculating the difference between the first monitoring result and the second monitoring result;
[0019] A model analysis module, configured to input the first monitoring result and the second monitoring result into a preset monitoring result processing model if the difference is greater than a preset alarm threshold;
[0020] A model output module, used to obtain the error cause of the output of the preset monitoring result processing model;
[0021] A repair script acquisition module is used to acquire a data repair script adapted to the cause of the error;
[0022] A data repair module is used to repair the data of the Oracle database and / or the Hive database according to the data repair script.
[0023] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above data monitoring method when executing the computer program.
[0024] A computer-readable storage medium stores a computer program, and the computer program implements the above data monitoring method when executed by a processor.
[0025] The above-mentioned data monitoring method, device, computer equipment and storage medium configure monitoring rules in an Oracle database, and the monitoring rules are used to monitor specified data indicators to customize the parameters to be monitored. The monitoring rules are sent to a Hive database to synchronize the monitoring rules. According to the monitoring rules, a first monitoring result within a specified time range is obtained in the Oracle database; according to the monitoring rules, a second monitoring result within the specified time range is obtained in the Hive database to obtain the monitoring results of the two databases. The second monitoring result is imported into the Oracle database, and the difference between the first monitoring result and the second monitoring result is calculated to compare the difference between the two monitoring results. If the difference is greater than a preset alarm threshold, the first monitoring result and the second monitoring result are input into a preset monitoring result processing model to automatically parse the abnormal monitoring result. The error cause output by the preset monitoring result processing model is obtained to determine the cause of the data error in the database. A data repair script adapted to the error cause is obtained to automatically match the repair solution to improve the data repair efficiency. The data of the Oracle database and / or the Hive database is repaired according to the data repair script to achieve automatic repair of the erroneous data. The data monitoring method provided by the present invention improves the processing efficiency and timeliness of cross-database data monitoring between a Hive database and an Oracle database, and can realize automatic repair of erroneous data. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0027] Figure 1 is a schematic diagram of an application environment of a data monitoring method in one embodiment of the present invention;
[0028] Figure 2 is a flow chart of a data monitoring method in one embodiment of the present invention;
[0029] Figure 3 is a flow chart of a data monitoring method in one embodiment of the present invention;
[0030] Figure 4 is a flow chart of a data monitoring method in one embodiment of the present invention;
[0031] Figure 5 is a flow chart of a data monitoring method in one embodiment of the present invention;
[0032] Figure 6 is a flow chart of a data monitoring method in one embodiment of the present invention;
[0033] Figure 7 is a flow chart of a data monitoring method in one embodiment of the present invention;
[0034] Figure 8 is a structural schematic diagram of a data monitoring device in one embodiment of the present invention;
[0035] Fig. 9 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] The data monitoring method provided in this embodiment can be applied in Figure 1 In an application environment, a client communicates with a server through a network. The client includes but is not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.
[0038] In one embodiment, if Figure 2 As shown, a data monitoring method is provided, which is applied in Figure 1 The server in the example is used as an example to illustrate the following steps:
[0039] S10. Configure monitoring rules in the Oracle database, where the monitoring rules are used to monitor specified data indicators;
[0040] S20, sending the monitoring rules to the Hive database;
[0041] S30, acquiring a first monitoring result within a specified time range in the Oracle database according to the monitoring rule; acquiring a second monitoring result within the specified time range in the Hive database according to the monitoring rule;
[0042] S40, importing the second monitoring result into the Oracle database, and calculating the difference between the first monitoring result and the second monitoring result;
[0043] S50: if the difference is greater than a preset alarm threshold, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model;
[0044] S60, obtaining the error cause of the output of the preset monitoring result processing model;
[0045] S70, obtaining a data repair script adapted to the cause of the error;
[0046] S80. Repair the data of the Oracle database and / or the Hive database according to the data repair script.
[0047] The data monitoring method provided in the embodiment of the present invention is used for data monitoring of databases of different structural types. Specifically, it can be used for data monitoring between an Oracle database and a Hive database. Oracle is the abbreviation of Oracle Database, also known as Oracle RDBMS, and is a relational database management system of Oracle Corporation. Hive is a data warehouse tool based on Hadoop, which can map structured data files to a database table, and provides simple SQL query functions, and can convert SQL statements into MapReduce tasks for execution. The data storage structure of Oracle database and Hive database is quite different.
[0048] Monitoring rules are used to determine the data objects that need to be monitored, specify data indicators, and monitoring methods. For example, monitoring rules can set data tables on key processing links as monitored data objects. Data objects can refer to business data to be monitored by Oracle databases and Hive databases. Specified data indicators can refer to parameters of data objects, such as library names, table names, business rule names, dimension names, indicator names, etc. The monitoring method can be monitoring according to a certain time period, such as every day, every three days, etc. Here, on the server running the Oracle database, a corresponding monitoring platform can be set up, and the monitoring platform is used to configure the above monitoring rules. For the same data table, different monitoring rules can be set, or a combination of monitoring rules can be used to monitor the data of the Oracle database.
[0049] After configuring the monitoring rules, you can synchronize the monitoring rules to the server running the Hive database through the above monitoring platform.
[0050] After synchronization is completed, you can set a scheduled task to regularly read monitoring rules, generate corresponding monitoring tasks, execute monitoring tasks, and obtain corresponding monitoring results. For example, you can generate a first monitoring result in an Oracle database, which can be the monitoring data obtained after monitoring multiple specified data indicators on August 7, 2019; you can generate a second monitoring result in a Hive database, which can be the monitoring data obtained after monitoring multiple specified data indicators on August 7, 2019.
[0051] After the Hive database generates the second monitoring result, the second monitoring result can be sent to the server running the Oracle database, and then the second monitoring result can be imported into the Oracle database, and the first monitoring result and the second monitoring result can be compared to calculate the difference. Under normal circumstances, the first monitoring result and the second monitoring result are the same, so the difference is zero. Under abnormal circumstances, there is a difference between the first monitoring result and the second monitoring result, and the difference is not zero.
[0052] The preset alarm threshold can be set according to actual needs, such as the preset alarm threshold can be set to zero. When comparing the first monitoring result and the second monitoring result, the difference between them can be calculated, and then the absolute value is taken as the difference. Therefore, the difference between the first monitoring result and the second monitoring result can be a value not less than zero.
[0053] When the difference is greater than the preset alarm threshold, the first monitoring result and the second monitoring result are input into the preset monitoring result processing model to determine the cause of the error in the monitoring result. The preset monitoring result processing model can be constructed based on the historical monitoring result data, and can accurately locate the cause of the data error and the importance of the specific data. After being processed by the preset monitoring result processing model, the cause of the error and the error level (related to the importance of the data in which the error occurred) of the monitoring result corresponding to this monitoring behavior can be obtained.
[0054] Multiple data repair scripts can be pre-set to adapt to different error causes. When there is a data repair script that is adapted to the error cause output by the preset monitoring result processing model, the adapted data repair script can be used to automatically modify the database data to ensure the correctness of the database data. Here, the erroneous data may be the data of the Oracle database or the Hive database, or both databases may have different erroneous data.
[0055] In one example, the error reason output by the preset monitoring result processing model is: the inconsistency of the settlement amount caused by the exchange rate settlement factor. At this time, the erroneous data is the settlement amount obtained by converting the initial transaction amount. A data repair script adapted to the error reason is stored on the server. When the data in the database is modified using the data repair script, the following steps will be performed: obtaining the settlement time of the initial transaction amount; querying the standard exchange rate of the settlement time, calculating the standard value of the settlement amount matching the initial transaction amount according to the standard exchange rate, and replacing the settlement amount in the database with the standard value of the settlement amount. In this way, the data repair script realizes the automatic correction of the erroneous data in the database.
[0056] When there is no data repair script adapted to the cause of the error output by the preset monitoring result processing model, it can be determined whether to send a reminder message to the designated client based on the error level output by the preset monitoring result processing model. If the error level is low, no reminder may be sent or the reminder may be sent with a delay; if the error level is high, the reminder is sent immediately. Here, the designated client can refer to the computer, mobile phone or other communication device used by the administrator, or it can be the alarm device of the data monitoring center. The reminder information can be a pop-up message, short message, light reminder signal, reminder email, etc.
[0057] In steps S10-S80, a monitoring rule is configured in the Oracle database, and the monitoring rule is used to monitor the specified data index to customize the parameters to be monitored. The monitoring rule is sent to the Hive database to synchronize the monitoring rule. According to the monitoring rule, the first monitoring result within the specified time range is obtained in the Oracle database; according to the monitoring rule, the second monitoring result within the specified time range is obtained in the Hive database to obtain the monitoring results of the two databases. The second monitoring result is imported into the Oracle database, and the difference between the first monitoring result and the second monitoring result is calculated to compare the difference between the two monitoring results. If the difference is greater than the preset alarm threshold, the first monitoring result and the second monitoring result are input into the preset monitoring result processing model to automatically parse the abnormal monitoring result. The error cause output by the preset monitoring result processing model is obtained to determine the cause of the data error in the database. A data repair script adapted to the error cause is obtained to automatically match the repair solution to improve the data repair efficiency. The data of the Oracle database and / or the Hive database is repaired according to the data repair script to achieve automatic repair of the erroneous data.
[0058] Optional, such as Figure 3 As shown, step S10 includes:
[0059] S11, receiving a first input instruction, where the first input instruction is used to select a number of the specified data indicators and set monitoring parameters;
[0060] S12. Receive a second input instruction, where the second input instruction is used to generate the monitoring rule, and the monitoring rule is used to collect monitoring data of the specified data indicator according to the monitoring parameter to generate the first monitoring result or the second monitoring result.
[0061] In this embodiment, the first input instruction may be an input instruction generated by a database administrator selecting a specified data indicator and inputting a monitoring parameter operation on a server running an Oracle database. The administrator may select important data indicators as specified data indicators according to actual needs. In an application example, the specified data indicator may be a library name, table name, business rule name, dimension name, indicator name, etc. to be monitored. Monitoring parameters include, but are not limited to, monitoring intervals, monitoring time ranges, etc. In some cases, only incremental data may be monitored to reduce the number of read and write times of the database data and improve data monitoring efficiency.
[0062] The second input instruction may be an input instruction generated by a database administrator performing a confirmation operation on a server running an Oracle database. In the monitoring platform, after the database administrator selects the specified data indicator and sets the monitoring parameters, a confirmation operation screen or button will appear. The database administrator performs the confirmation operation according to the instructions on the screen, or clicks the confirmation button, and the second input instruction is generated. After receiving the second input instruction, the server can generate corresponding monitoring rules according to the first input instruction provided by the database administrator in the previous step.
[0063] Steps S11-S12, receiving a first input instruction, the first input instruction is used to select a number of the specified data indicators and set monitoring parameters. Since the specified data indicators and monitoring parameters are preset for selection, the efficiency of configuring monitoring parameters is greatly improved. Receive a second input instruction, the second input instruction is used to generate the monitoring rule, the monitoring rule is used to collect the monitoring data of the specified data indicators according to the monitoring parameters to generate the first monitoring result or the second monitoring result. This step can generate monitoring rules with one click, which greatly improves the configuration efficiency.
[0064] Optional, such as Figure 4 As shown, step S30 includes:
[0065] S301, creating a first monitoring result table, and writing the specified data indicator into the first monitoring result table;
[0066] S302: Generate a first query statement according to the monitoring rule, where the first query statement includes the specified time range;
[0067] S303: Execute the first query statement in the Oracle database to obtain first monitoring data corresponding to the specified data indicator within the specified time range;
[0068] S304: Import the first monitoring data into the first monitoring result table to generate a first monitoring result.
[0069] In this embodiment, a first monitoring result table can be created on a server running an Oracle database, and the specified data indicator can be written into the first monitoring result table. When the first monitoring result table is created for the first time, the first monitoring result table can be a blank table. In some cases, a new first monitoring result table can be created each time a monitoring operation is performed. In other cases, if the monitoring rules have not changed, the last first monitoring result table can be used. The first monitoring result table includes multiple information columns, such as a library name column, a table name column, a business rule column, a dimension column, an indicator column, etc. Each information column corresponds to a specified data indicator.
[0070] A query task for an Oracle database can be created according to the monitoring rules to generate a first query statement that can perform a query operation in the Oracle database. The first query statement includes the specified data indicators and the specified time range to be queried. After the Oracle database completes the query operation, the corresponding query result, i.e., the first monitoring data, can be obtained. Then, the first monitoring data is written into the first monitoring result table to obtain the first monitoring result. For example, in an Oracle database monitoring task, the first monitoring result of Table A on August 7, 2019 can be obtained, and the amount of data recorded in the first monitoring result is 1000.
[0071] In steps S301-S304, a first monitoring result table is created, and the specified data indicator is written into the first monitoring result table, so as to write the specified data indicator included in the monitoring rule into the first monitoring result table. A first query statement is generated according to the monitoring rule, and the first query statement includes the specified time range to generate a first query statement that can perform a query operation in an Oracle database. The first query statement is executed in the Oracle database to obtain the first monitoring data corresponding to the specified data indicator within the specified time range to obtain the monitoring data of the Oracle database. The first monitoring data is imported into the first monitoring result table to generate a first monitoring result to obtain the monitoring result of the Oracle database.
[0072] Optional, such as Figure 5 As shown, step S30 also includes:
[0073] S305: Create a second monitoring result table, and write the specified data indicator into the second monitoring result table;
[0074] S306, generating a MapReduce task according to the monitoring rule, wherein the MapReduce task includes the specified time range;
[0075] S307: Execute the MapReduce task in the Hive database to obtain second monitoring data corresponding to the specified data indicator within the specified time range;
[0076] S308: Import the second monitoring data into the second monitoring result table to generate a second monitoring result.
[0077] In this embodiment, a second monitoring result table can be created on a server running a Hive database, and the specified data indicator can be written into the second monitoring result table. When the second monitoring result table is created for the first time, the second monitoring result table can be a blank table. In some cases, a new second monitoring result table can be created each time a monitoring operation is performed. In other cases, if the monitoring rules have not changed, the previous second monitoring result table can be used. The second monitoring result table includes multiple information columns, such as a library name column, a table name column, a business rule column, a dimension column, an indicator column, etc. Each information column corresponds to a specified data indicator.
[0078] A query task of the Hive database can be created according to the monitoring rules to generate a MapReduce task that can perform query operations in the Hive database. The MapReduce task includes the specified data indicators and the specified time range to be queried. After the Hive database completes the query operation, the corresponding query result, that is, the second monitoring data, can be obtained. Then the second monitoring data is written into the second monitoring result table to obtain the second monitoring result. For example, in a monitoring task of the Hive database, the second monitoring result of Table A on August 7, 2019 can be obtained, and the amount of data recorded in the second monitoring result is 1001.
[0079] In steps S305-S308, a second monitoring result table is created, and the specified data indicator is written into the second monitoring result table, so as to write the specified data indicator included in the monitoring rule into the second monitoring result table. A MapReduce task is generated according to the monitoring rule, and the MapReduce task includes the specified time range to generate a MapReduce task that can be executed in the Hive database. The MapReduce task is executed in the Hive database to obtain the corresponding second monitoring data of the specified data indicator within the specified time range to obtain the monitoring data of the Hive database. The second monitoring data is imported into the second monitoring result table to generate a second monitoring result to obtain the monitoring result of the Hive database.
[0080] Optional, such as Figure 6 As shown, step S40 includes:
[0081] S401, obtaining a preset processing script;
[0082] S402: parsing the first monitoring result and the second monitoring result according to the preset processing script, and calculating the indicator difference of each of the specified data indicators;
[0083] S403: Generate a difference between the first monitoring result and the second monitoring result based on the indicator difference of the specified data indicator.
[0084] In this embodiment, the preset processing script can be used to compare the difference between each specified data indicator in the first monitoring result table and the second monitoring result table, that is, the indicator difference. For example, the value of data indicator X in the first monitoring result table is 1002, and the value in the second monitoring result table is 1002, then the indicator difference of data indicator X is 0. The difference of each indicator can be the absolute value of the difference between the values in the monitoring result table. For example, the value of data indicator Y in the first monitoring result table is 1005, and the value in the second monitoring result table is 1006, and the indicator difference of data indicator Y is 1; the value of data indicator Z in the first monitoring result table is 1006, and the value in the second monitoring result table is 1004, then the indicator difference of data indicator Z is 2.
[0085] The difference between the first monitoring result and the second monitoring result may be a weighted sum of the differences of all indicators. If the weight of each designated data indicator is equal, the difference between the first monitoring result and the second monitoring result may be the sum of the differences of all indicators.
[0086] In steps S401-S403, a preset processing script is obtained to obtain a script that can be used to process the monitoring results. The first monitoring result and the second monitoring result are parsed according to the preset processing script, and the indicator difference of each of the specified data indicators is calculated to calculate the indicator difference between the first monitoring result and the second monitoring result on each specified data indicator. The difference between the first monitoring result and the second monitoring result is generated based on the indicator difference of the specified data indicator to obtain the overall difference between the first monitoring result and the second monitoring result.
[0087] Optional, such as Figure 7 As shown, before step S50, it also includes:
[0088] S51, obtaining a preset alarm evaluation rule, wherein the preset alarm evaluation rule includes a warning threshold and importance of each of the specified data indicators;
[0089] S52, calculating a threshold comparison value of the difference degree according to the warning threshold and the importance;
[0090] S53, determining whether the threshold comparison value is greater than the preset alarm threshold;
[0091] S54: If the threshold comparison value is greater than the preset alarm threshold, determine that the difference is greater than the preset alarm threshold.
[0092] In this embodiment, the preset alarm evaluation rule can set the warning threshold and importance of each specified data indicator. Specifically, the warning threshold in the preset alarm evaluation rule can be represented by an array, such as {p1, p2, p3, ... p n} (where p represents the warning threshold and n is the total number of warning thresholds). The importance of the preset alarm evaluation rule can be represented by an array, such as {q1, q2, q3, ...q n}(where q represents the importance and n is the total number of warning thresholds).
[0093] In some cases, the difference between the first monitoring result and the second monitoring result can be represented by an array, such as {c1, c2, c3, ... c n}(where c represents the indicator difference and n is the total number of indicator differences).
[0094] Thus, the threshold comparison value K of the difference can be calculated, and the calculation formula is as follows:
[0095]
[0096] When the threshold comparison value is greater than the preset alarm threshold, it is determined that the difference is greater than the preset alarm threshold. Here, the preset alarm threshold may refer to an overall threshold, and its value may be zero. In some cases, different alarm levels may be set, and the alarm level changes with the difference between the threshold comparison value and the preset alarm threshold.
[0097] In steps S51-S54, the preset alarm judgment rules are obtained, and the preset alarm judgment rules include the warning threshold and importance of each of the specified data indicators, so as to pre-set the alarm judgment rules to facilitate the judgment of the difference. The threshold comparison value of the difference is calculated according to the warning threshold and the importance to convert the difference into an intermediate value that can be compared with the preset alarm threshold. It is determined whether the threshold comparison value is greater than the preset alarm threshold to compare the two. If the threshold comparison value is greater than the preset alarm threshold, it is determined that the difference is greater than the preset alarm threshold to determine the result of the difference comparison.
[0098] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0099] In one embodiment, a data monitoring device is provided, which corresponds one-to-one to the data monitoring method in the above embodiment. Figure 8 As shown, the data monitoring device includes a rule configuration module 10, a rule sending module 20, a monitoring result acquisition module 30, a difference calculation module 40, a model analysis module 50, a model output module 60, an acquisition repair script module 70 and a data repair module 80. The functional modules are described in detail as follows:
[0100] A rule configuration module 10 is used to configure monitoring rules in an Oracle database, wherein the monitoring rules are used to monitor specified data indicators;
[0101] A rule sending module 20, used for sending the monitoring rules to a Hive database;
[0102] Acquiring monitoring result module 30, used for acquiring a first monitoring result within a specified time range in the Oracle database according to the monitoring rule; acquiring a second monitoring result within the specified time range in the Hive database according to the monitoring rule;
[0103] A difference calculation module 40, used for importing the second monitoring result into the Oracle database and calculating the difference between the first monitoring result and the second monitoring result;
[0104] A model analysis module 50, configured to input the first monitoring result and the second monitoring result into a preset monitoring result processing model if the difference is greater than a preset alarm threshold;
[0105] A model output module 60, used to obtain the error cause of the output of the preset monitoring result processing model;
[0106] A repair script acquisition module 70 is used to acquire a data repair script adapted to the cause of the error;
[0107] The data repair module 80 is used to repair the data of the Oracle database and / or the Hive database according to the data repair script.
[0108] Optionally, the rule configuration module 10 includes:
[0109] A first input unit, used to receive a first input instruction, wherein the first input instruction is used to select a number of the specified data indicators and set monitoring parameters;
[0110] The second input unit is used to receive a second input instruction, where the second input instruction is used to generate the monitoring rule, and the monitoring rule is used to collect the monitoring data of the specified data indicator according to the monitoring parameter to generate the first monitoring result or the second monitoring result.
[0111] Optionally, the monitoring result acquisition module 30 includes:
[0112] A first table creation unit, used to create a first monitoring result table and write the specified data indicator into the first monitoring result table;
[0113] A query statement unit, configured to generate a first query statement according to the monitoring rule, wherein the first query statement includes the specified time range;
[0114] An acquisition first monitoring data unit is used to execute the first query statement in the Oracle database to acquire first monitoring data corresponding to the specified data indicator within the specified time range;
[0115] The first monitoring result generating unit is used to import the first monitoring data into the first monitoring result table to generate a first monitoring result.
[0116] Optionally, the monitoring result acquisition module 30 further includes:
[0117] A second table creation unit, used to create a second monitoring result table and write the specified data indicator into the second monitoring result table;
[0118] A task generating unit, configured to generate a MapReduce task according to the monitoring rule, wherein the MapReduce task includes the specified time range;
[0119] Obtain a second monitoring data unit, which is used to execute the MapReduce task in the Hive database to obtain second monitoring data corresponding to the specified data indicator within the specified time range;
[0120] The second monitoring result generating unit is used to import the second monitoring data into the second monitoring result table to generate a second monitoring result.
[0121] Optionally, the difference calculation module 40 includes:
[0122] Get script unit, used to get preset processing script;
[0123] An indicator difference calculation unit, used for parsing the first monitoring result and the second monitoring result according to the preset processing script, and calculating the indicator difference of each of the specified data indicators;
[0124] A difference generating unit is used to generate the difference between the first monitoring result and the second monitoring result based on the indicator difference of the specified data indicator.
[0125] Optionally, the data monitoring device further includes a difference comparison module, and the difference comparison module includes:
[0126] An acquisition rule unit is used to acquire preset alarm evaluation rules, wherein the preset alarm evaluation rules include warning thresholds and importance of each of the specified data indicators;
[0127] A comparison value calculation unit, used for calculating a threshold comparison value of the difference degree according to the warning threshold and the importance;
[0128] A size comparison unit, used to determine whether the threshold comparison value is greater than the preset alarm threshold;
[0129] The alarm determination unit is configured to determine that the difference is greater than the preset alarm threshold if the threshold comparison value is greater than the preset alarm threshold.
[0130] For the specific definition of the data monitoring device, please refer to the definition of the data monitoring method above, which will not be repeated here. Each module in the above-mentioned data monitoring device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0131] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the data monitoring method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data monitoring method is implemented.
[0132] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:
[0133] Configure monitoring rules in the Oracle database, where the monitoring rules are used to monitor specified data indicators;
[0134] Sending the monitoring rules to the Hive database;
[0135] Acquire a first monitoring result within a specified time range in the Oracle database according to the monitoring rule; acquire a second monitoring result within the specified time range in the Hive database according to the monitoring rule;
[0136] Importing the second monitoring result into the Oracle database, and calculating the difference between the first monitoring result and the second monitoring result;
[0137] If the difference is greater than a preset alarm threshold, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model;
[0138] Obtaining the error cause of the output of the preset monitoring result processing model;
[0139] Obtain a data repair script adapted to the cause of the error;
[0140] Repair the data of the Oracle database and / or the Hive database according to the data repair script.
[0141] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0142] Configure monitoring rules in the Oracle database, where the monitoring rules are used to monitor specified data indicators;
[0143] Sending the monitoring rules to the Hive database;
[0144] Acquire a first monitoring result within a specified time range in the Oracle database according to the monitoring rule; acquire a second monitoring result within the specified time range in the Hive database according to the monitoring rule;
[0145] Importing the second monitoring result into the Oracle database, and calculating the difference between the first monitoring result and the second monitoring result;
[0146] If the difference is greater than a preset alarm threshold, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model;
[0147] Obtaining the error cause of the output of the preset monitoring result processing model;
[0148] Obtaining a data repair script adapted to the cause of the error;
[0149] Repair the data of the Oracle database and / or the Hive database according to the data repair script.
[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0151] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0152] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A data monitoring method, characterized in that: include: Configure monitoring rules in the Oracle database, where the monitoring rules are used to monitor specified data indicators; Sending the monitoring rules to the Hive database; Acquiring a first monitoring result within a specified time range in the Oracle database according to the monitoring rule includes: creating a first monitoring result table, and writing the specified data indicator in the first monitoring result table; generating a first query statement according to the monitoring rule, the first query statement including the specified time range; executing the first query statement in the Oracle database to acquire first monitoring data corresponding to the specified data indicator within the specified time range; importing the first monitoring data into the first monitoring result table to generate a first monitoring result; Acquiring the second monitoring result within the specified time range in the Hive database according to the monitoring rule, including: creating a second monitoring result table, and writing the specified data indicator in the second monitoring result table; generating a MapReduce task according to the monitoring rule, the MapReduce task including the specified time range; executing the MapReduce task in the Hive database to acquire the second monitoring data corresponding to the specified data indicator within the specified time range; importing the second monitoring data into the second monitoring result table to generate a second monitoring result; Importing the second monitoring result into the Oracle database, and calculating the difference between the first monitoring result and the second monitoring result; If the difference is greater than a preset alarm threshold, inputting the first monitoring result and the second monitoring result into a preset monitoring result processing model; Obtaining the error cause of the output of the preset monitoring result processing model; Obtain a data repair script adapted to the cause of the error; Repair the data of the Oracle database and / or the Hive database according to the data repair script.
2. The data monitoring method according to claim 1, characterized in that: The monitoring rules configured in the Oracle database are used to monitor the specified data indicators, including: Receiving a first input instruction, wherein the first input instruction is used to select a number of the specified data indicators and set monitoring parameters; A second input instruction is received, where the second input instruction is used to generate the monitoring rule, where the monitoring rule is used to collect monitoring data of the specified data indicator according to the monitoring parameter to generate the first monitoring result or the second monitoring result.
3. The data monitoring method according to claim 1, characterized in that: The step of importing the second monitoring result into the Oracle database and calculating the difference between the first monitoring result and the second monitoring result includes: Get the preset processing script; Analyze the first monitoring result and the second monitoring result according to the preset processing script, and calculate the indicator difference degree of each of the specified data indicators; The difference between the first monitoring result and the second monitoring result is generated based on the indicator difference of the specified data indicator.
4. The data monitoring method according to claim 1, characterized in that: If the difference is greater than a preset alarm threshold, before the first monitoring result and the second monitoring result are input into a preset monitoring result processing model, the method includes: Obtaining preset alarm evaluation rules, wherein the preset alarm evaluation rules include warning thresholds and importance of each of the specified data indicators; Calculate the threshold comparison value of the difference degree according to the warning threshold and the importance; Determine whether the threshold comparison value is greater than the preset alarm threshold; If the threshold comparison value is greater than the preset alarm threshold, it is determined that the difference is greater than the preset alarm threshold.
5. A data monitoring device, characterized in that: include: A rule configuration module is used to configure monitoring rules in an Oracle database, wherein the monitoring rules are used to monitor specified data indicators; A rule sending module, used for sending the monitoring rules to a Hive database; A monitoring result acquisition module is used to acquire a first monitoring result within a specified time range in the Oracle database according to the monitoring rule, create a first monitoring result table, and write the specified data indicator into the first monitoring result table; generate a first query statement according to the monitoring rule, and the first query statement includes the specified time range; Execute the first query statement in the Oracle database to obtain first monitoring data corresponding to the specified data indicator within the specified time range; Importing the first monitoring data into the first monitoring result table to generate a first monitoring result; acquiring the second monitoring result within the specified time range in the Hive database according to the monitoring rule, creating a second monitoring result table, and writing the specified data indicator into the second monitoring result table; generating a MapReduce task according to the monitoring rule, the MapReduce task including the specified time range; executing the MapReduce task in the Hive database to acquire the second monitoring data corresponding to the specified data indicator within the specified time range; Importing the second monitoring data into the second monitoring result table to generate a second monitoring result; A difference calculation module, used for importing the second monitoring result into the Oracle database, and calculating the difference between the first monitoring result and the second monitoring result; A model analysis module, configured to input the first monitoring result and the second monitoring result into a preset monitoring result processing model if the difference is greater than a preset alarm threshold; A model output module, used to obtain the error cause of the output of the preset monitoring result processing model; A repair script acquisition module is used to acquire a data repair script adapted to the cause of the error; A data repair module is used to repair the data of the Oracle database and / or the Hive database according to the data repair script.
6. The data monitoring device according to claim 5, characterized in that: The rule configuration module includes: A first input unit, used to receive a first input instruction, wherein the first input instruction is used to select a number of the specified data indicators and set monitoring parameters; The second input unit is used to receive a second input instruction, where the second input instruction is used to generate the monitoring rule, and the monitoring rule is used to collect the monitoring data of the specified data indicator according to the monitoring parameter to generate the first monitoring result or the second monitoring result.
7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data monitoring method according to any one of claims 1 to 4 is implemented.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data monitoring method according to any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
Monitoring and warning method, monitoring and warning device, monitoring and warning equipment and computer-readable storage medium
CN109688188A
System for data management in a large scale data repository
CN110300963A
Monitoring alarm method, device and system for weblogic server, and computer storage medium
WO2020015061A1
