Database platform index monitoring and early warning assembly and method based on machine learning

Abstract

The invention discloses a database platform index monitoring and early warning assembly and method based on machine learning, belongs to the technical field of database monitoring, and aims to solve the technical problem of how to carry out monitoring and early warning on data in a database. Comprising a machine learning module, an alarm module and a feedback module. The machine learning module interacts with the outside through a machine learning interface and is used for selecting a data column, a model and a preprocessing mode and configuring the model; the alarm module interacts with the outside through an alarm interface, is used for providing a data deviation model and setting alarms, and is used for performing deviation model index calculation on input data based on an index formula in the data deviation model or an input deviation index expression to form an alarm prediction result, and sending the alarm prediction result to the feedback module; and the feedback module is used for modifying the machine learning model based on the feedback prediction result.

Description

Technical Field

[0001] The present invention relates to the technical field of database monitoring, and in particular to a database platform indicator monitoring and early warning component and method based on machine learning. Background Art

[0002] Currently, many indicator data visualization tools (such as Grafana) have the function of displaying data charts and warnings in web browsers. In the era of big data, there are many independent machine learning software that can learn and model data in databases (TensorFlow, Torch, etc.).

[0003] However, there are currently no universal visualization tools that can use big data models to issue alerts for newly entered abnormal data. If third-party software is relied upon, unnecessary efficiency reduction and incompatibility issues will occur.

[0004] How to monitor and warn the data in the database is a technical problem that needs to be solved. Summary of the invention

[0005] The technical task of the present invention is to address the above shortcomings and provide a database platform indicator monitoring and early warning component and method based on machine learning to solve the technical problem of how to monitor and warn data in the database.

[0006] In a first aspect, the present invention provides a database platform indicator monitoring and early warning component based on machine learning, including a machine learning module, an alarm module and a feedback module;

[0007] The machine learning module interacts with the outside world through the machine learning interface, which is used to select data columns, models, and preprocessing methods, and to configure the model;

[0008] The alarm module interacts with the outside world through the alarm interface, is used to provide a data deviation model and set alarms, is used to calculate the indicators of the deviation model for the input data based on the indicator formula in the data deviation model or the input deviation indicator expression, forms an alarm prediction result, and sends the alarm prediction result to the feedback module;

[0009] The feedback module is used to modify the machine learning model based on the feedback prediction results.

[0010] Preferably, the machine learning module is used to perform the following operations:

[0011] Select the data columns to be monitored, and support data columns from tables in different databases;

[0012] Selecting a model, including a mathematical model and a machine learning model, wherein the mathematical model includes fitting calculations; configuring the initial parameters of the model;

[0013] Select available preprocessing methods to precompute selected data columns in the database;

[0014] Choose how often to update data to trigger model training.

[0015] Preferably, the machine learning module supports selecting data columns during model training, inserting new data for model training, and supporting modifying configured parameters and update data that triggers model training frequency during model training.

[0016] Preferably, the alarm module is used to perform the following operations:

[0017] Select the indicator formula of the data deviation model provided by the alarm module or input the deviation indicator expression;

[0018] Configure the indicator threshold for generating an alarm. When the deviation indicator value exceeds the threshold, it is considered that the data point deviates from the model.

[0019] Set the threshold M of the number of times the data continuously deviates from the model required to generate an alarm;

[0020] Select the alert method, including email and SMS;

[0021] The indicator formula of the deviation indicator model or the input deviation indicator expression is used to calculate the indicator of the deviation model, and the obtained deviation indicator value is compared with the threshold to form an alarm prediction result. If N consecutive data deviate from the model, an alarm is required, where N is greater than M.

[0022] Preferably, the feedback module is used as follows:

[0023] If it is inconvenient to maintain the model based on the alarm prediction results, continue to monitor the data;

[0024] If the model is modified, the deviated data points are added to the existing model for learning and training, or historical data with timestamps earlier than the predetermined time are discarded and learning and training is continued based on the remaining data and the newly added data.

[0025] In a second aspect, the present invention provides a database platform indicator monitoring and early warning method based on machine learning, which is used to implement indicator monitoring and early warning of the database platform through the database platform indicator monitoring and early warning system based on machine learning as described in any one of the first aspects, and the method comprises the following steps:

[0026] Data and model selection: select data columns, models, and preprocessing methods, and configure the model;

[0027] Alarm analysis: Based on the indicator formula in the data deviation model or the input deviation indicator expression, the indicator of the deviation model is calculated for the input data, the alarm prediction result is formed, and the alarm prediction result is sent to the feedback module;

[0028] Result feedback: Modify the machine learning model based on the feedback prediction results.

[0029] Preferably, data and model selection includes the following steps:

[0030] Select the data columns to be monitored, and support data columns from tables in different databases;

[0031] Selecting a model, including a mathematical model and a machine learning model, wherein the mathematical model includes fitting calculations; configuring the initial parameters of the model;

[0032] Select available preprocessing methods to precompute selected data columns in the database;

[0033] Choose how often to update data to trigger model training.

[0034] Preferably, the data and model selection also includes the following steps: selecting data columns during model training, inserting new data for model training, and modifying the configured parameters and the update data that triggers the model training frequency during model training.

[0035] Preferably, the alarm analysis includes the following steps:

[0036] Select the indicator formula of the data deviation model provided by the alarm module or input the deviation indicator expression;

[0037] Configure the indicator threshold for generating an alarm. When the deviation indicator value exceeds the threshold, it is considered that the data point deviates from the model.

[0038] Set the threshold M of the number of times the data continuously deviates from the model required to generate an alarm;

[0039] Select the alert method, including email and SMS;

[0040] The indicator formula of the deviation indicator model or the input deviation indicator expression is used to calculate the indicator of the deviation model, and the obtained deviation indicator value is compared with the threshold to form an alarm prediction result. If N consecutive data deviate from the model, an alarm is required, where N is greater than M.

[0041] Preferably, the result feedback includes the following steps:

[0042] If it is inconvenient to maintain the model based on the alarm prediction results, continue to monitor the data;

[0043] If the model is modified, the deviated data points are added to the existing model for learning and training, or historical data with timestamps earlier than the predetermined time are discarded and learning and training is continued based on the remaining data and the newly added data.

[0044] The database platform indicator monitoring and early warning component and method based on machine learning of the present invention have the following advantages:

[0045] 1. Be able to perform machine learning and model training on the newly inserted data in the database and display the results;

[0046] 2. Automatically alert users when certain data deviates from the model;

[0047] 3. The model can be modified based on the alarm feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] The present invention is further described below in conjunction with the accompanying drawings.

[0050] Figure 1 This is a flowchart of a database platform indicator monitoring and early warning method based on machine learning in Example 2. DETAILED DESCRIPTION

[0051] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0052] The embodiments of the present invention provide a database platform indicator monitoring and early warning component and method based on machine learning, which are used to solve the technical problem of how to monitor and warn data in a database.

[0053] Embodiment 1:

[0054] The present invention discloses a database platform indicator monitoring and early warning component based on machine learning, comprising a machine learning module, an alarm module and a feedback module.

[0055] The machine learning module interacts with the outside world through the machine learning interface, which is used to select data columns, models, and preprocessing methods, and to configure the model.

[0056] In this embodiment, the machine learning module is used to perform the following operations:

[0057] (1) Select the data columns to be monitored, and support data columns from tables in different databases;

[0058] (2) Selecting a model, including a mathematical model and a machine learning model, wherein the mathematical model includes fitting calculations; configuring initial parameters of the model;

[0059] (3) selecting an available preprocessing method for precalculating the selected data columns in the database;

[0060] (4) Select the frequency of updating data to trigger model training.

[0061] In this implementation, the model will learn based on historical data and update the model when each new data is inserted.

[0062] The machine learning module supports the selection of data columns during model training, the insertion of new data for model training, and the modification of configuration parameters and update data that triggers model training frequency during model training.

[0063] The alarm module interacts with the outside world through the alarm interface, and is used to provide a data deviation model and set alarms. It is used to calculate the indicators of the deviation model for the input data based on the indicator formula in the data deviation model or the input deviation indicator expression, form alarm prediction results, and send the alarm prediction results to the feedback module.

[0064] As a specific implementation, the alarm module is used to perform the following operations:

[0065] (1) Select the index formula of the data deviation model provided by the alarm module or input the deviation index expression;

[0066] (2) Configure the indicator threshold for generating an alarm. When the deviation indicator value exceeds the threshold, it is considered that the data point deviates from the model.

[0067] (3) Setting a threshold M of the number of times the data continuously deviates from the model required to generate an alarm;

[0068] (4) Select the alert channel, including email and SMS;

[0069] (5) The indicator of the deviation model is calculated based on the indicator formula of the deviation indicator model or the input deviation indicator expression, and the obtained deviation indicator value is compared with the threshold to form an alarm prediction result. If N consecutive data deviate from the model, an alarm is required, where N is greater than M.

[0070] The feedback module is used to modify the machine learning model based on the feedback prediction results.

[0071] As a specific implementation, the feedback module is used as follows:

[0072] (1) If it is inconvenient to maintain the model based on the alarm prediction results, continue to monitor the data;

[0073] (2) If the model is modified, the deviated data points are added to the existing model for learning and training, or historical data with timestamps earlier than the predetermined time are discarded and learning and training is continued based on the remaining data and the newly added data.

[0074] The components of this embodiment have the following functions:

[0075] (1) Model initialization: Users can set which data columns need to be added to machine learning, which model to choose, and the initial parameters. Users can also set whether the selected data columns should be processed in batches (pre-calculated) before adding them to model training;

[0076] (2) Mid-course modification of the model: Users can stop learning the newly added data midway, change the learning frequency, modify the model parameters, or change the pre-calculated algorithm;

[0077] (3) Model display: Users can request to see the results of simple models (e.g., fitting) or make predictions about future data;

[0078] (4) Deviation data monitoring settings: Users can select the indicator formula of the data deviation model provided by the component, or enter the deviation indicator expression by themselves, and provide the indicator threshold for generating an alarm. In addition, users can also set the number of times the indicator of the data deviation model exceeds the threshold required to generate an alarm, as well as the alarm channel (email, SMS, etc.);

[0079] (5) Data deviation alarm. When the alarm requirements are met, the component can send a signal to the user through the path provided by the user, and record the alarm content, such as the specific information of the deviation data, the number of deviations, etc. At this time, the model learning needs to be terminated and wait for the user to return;

[0080] (6) User feedback on alarms. When the user learns that the data deviates from the original model, the user can choose to discard the data (i.e., not add them to the model training) or update the model training (add them to the existing model training; or discard part of the historical data and retrain the remaining and latest data).

[0081] Embodiment 2:

[0082] The present invention provides a database platform indicator monitoring and early warning method based on machine learning. The indicator monitoring and early warning of the database platform are realized through the method disclosed in Example 1. The method includes the following three steps: data and model selection, alarm analysis and result feedback.

[0083] Step S100: Data and model selection: select data columns, models and preprocessing methods, and configure the model.

[0084] In this embodiment, machine learning is performed as follows:

[0085] (1) Select the data columns to be monitored, and support data columns from tables in different databases;

[0086] (2) Selecting a model, including a mathematical model and a machine learning model, wherein the mathematical model includes fitting calculations; configuring initial parameters of the model;

[0087] (3) selecting an available preprocessing method for precalculating the selected data columns in the database;

[0088] (4) Select the frequency of updating data to trigger model training.

[0089] In this implementation, the model will learn based on historical data and update the model when each new data is inserted.

[0090] The machine learning process supports the selection of data columns during model training, the insertion of new data for model training, and the modification of configuration parameters and update data that triggers model training frequency during model training.

[0091] Step S200 alarm analysis: based on the indicator formula in the data deviation model or the input deviation indicator expression, the indicator of the deviation model is calculated for the input data, an alarm prediction result is formed, and the alarm prediction result is sent to the feedback module.

[0092] As a specific implementation of alarm analysis, perform the following operations:

[0093] (1) Select the index formula of the data deviation model provided by the alarm module or input the deviation index expression;

[0094] (2) Configure the indicator threshold for generating an alarm. When the deviation indicator value exceeds the threshold, it is considered that the data point deviates from the model.

[0095] (3) Setting a threshold M of the number of times the data continuously deviates from the model required to generate an alarm;

[0096] (4) Select the alert channel, including email and SMS;

[0097] (5) The indicator of the deviation model is calculated based on the indicator formula of the deviation indicator model or the input deviation indicator expression, and the obtained deviation indicator value is compared with the threshold to form an alarm prediction result. If N consecutive data deviate from the model, an alarm is required, where N is greater than M.

[0098] Step S300: Result feedback: modify the machine learning model based on the feedback prediction results.

[0099] As a specific implementation, the result feedback includes the following steps:

[0100] (1) If it is inconvenient to maintain the model based on the alarm prediction results, continue to monitor the data;

[0101] (2) If the model is modified, the deviated data points are added to the existing model for learning and training, or historical data with timestamps earlier than the predetermined time are discarded and learning and training is continued based on the remaining data and the newly added data.

[0102] Based on the method of this embodiment, users can preset some data relationships of interest, and let the platform generate models through big data training to monitor newly added data and provide feedback. The functions of this method are as follows: 1) Train models for columns in multiple databases; 2) Monitor subsequent newly entered data, and record and alert data that deviates from the model; 3) Maintain or modify the existing model after the user receives an alert.

[0103] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A database platform indicator monitoring and early warning component based on machine learning, characterized in that: Includes machine learning module, alarm module and feedback module; The machine learning module interacts with the outside world through the machine learning interface, which is used to select data columns, models, and preprocessing methods, and to configure the model; The alarm module interacts with the outside world through the alarm interface, is used to provide a data deviation model and set alarms, is used to calculate the indicators of the deviation model for the input data based on the indicator formula in the data deviation model or the input deviation indicator expression, forms an alarm prediction result, and sends the alarm prediction result to the feedback module; The feedback module is used to modify the machine learning model based on the feedback prediction results.

2. The machine learning-based database platform indicator monitoring and early warning component according to claim 1, characterized in that: The machine learning module is used to perform the following operations: Select the data columns to be monitored, and support data columns from tables in different databases; Selecting a model, including a mathematical model and a machine learning model, wherein the mathematical model includes fitting calculations; configuring the initial parameters of the model; Select available preprocessing methods to precompute selected data columns in the database; Choose how often to update data to trigger model training.

3. The machine learning-based database platform indicator monitoring and early warning component according to claim 2, characterized in that: The machine learning module supports selecting data columns during model training, inserting new data for model training, and supporting modifying configured parameters and update data that triggers model training frequency during model training.

4. The machine learning-based database platform indicator monitoring and early warning component according to claim 1, characterized in that: The alarm module is used to perform the following operations: Select the indicator formula of the data deviation model provided by the alarm module or input the deviation indicator expression; Configure the indicator threshold for generating an alarm. When the deviation indicator value exceeds the threshold, it is considered that the data point deviates from the model. Set the threshold M of the number of times the data continuously deviates from the model required to generate an alarm; Select the alert method, including email and SMS; The indicator formula of the deviation indicator model or the input deviation indicator expression is used to calculate the indicator of the deviation model, and the obtained deviation indicator value is compared with the threshold to form an alarm prediction result. If N consecutive data deviate from the model, an alarm is required, where N is greater than M.

5. The machine learning-based database platform indicator monitoring and early warning component according to claim 1, characterized in that: The feedback module is used as follows: If it is inconvenient to maintain the model based on the alarm prediction results, continue to monitor the data; If the model is modified, the deviated data points are added to the existing model for learning and training, or historical data with timestamps earlier than the predetermined time are discarded and learning and training is continued based on the remaining data and the newly added data.

6. A database platform indicator monitoring and early warning method based on machine learning, characterized in that: The method is used to implement the indicator monitoring and early warning of the database platform by using the database platform indicator monitoring and early warning system based on machine learning as described in any one of claims 1 to 5, and the method comprises the following steps: Data and model selection: select data columns, models, and preprocessing methods, and configure the model; Alarm analysis: Based on the indicator formula in the data deviation model or the input deviation indicator expression, the indicator of the deviation model is calculated for the input data, the alarm prediction result is formed, and the alarm prediction result is sent to the feedback module; Result feedback: Modify the machine learning model based on the feedback prediction results.

7. The method for monitoring and early warning of database platform indicators based on machine learning according to claim 6 is characterized in that: Data and model selection includes the following steps: Select the data columns to be monitored, and support data columns from tables in different databases; Selecting a model, including a mathematical model and a machine learning model, wherein the mathematical model includes fitting calculations; configuring the initial parameters of the model; Select available preprocessing methods to precompute selected data columns in the database; Choose how often to update data to trigger model training.

8. The method for monitoring and early warning of database platform indicators based on machine learning according to claim 7 is characterized in that: Data and model selection also includes the following steps: selecting data columns during model training, inserting new data for model training, and modifying the configuration parameters and update data that triggers the model training frequency during model training.

9. The method for monitoring and early warning of database platform indicators based on machine learning according to claim 6 is characterized in that: Alarm analysis includes the following steps: Select the indicator formula of the data deviation model provided by the alarm module or input the deviation indicator expression; Configure the indicator threshold for generating an alarm. When the deviation indicator value exceeds the threshold, it is considered that the data point deviates from the model. Set the threshold M of the number of times the data continuously deviates from the model required to generate an alarm; Select the alert method, including email and SMS; The indicator formula of the deviation indicator model or the input deviation indicator expression is used to calculate the indicator of the deviation model, and the obtained deviation indicator value is compared with the threshold to form an alarm prediction result. If N consecutive data deviate from the model, an alarm is required, where N is greater than M.

10. The method for monitoring and early warning of database platform indicators based on machine learning according to claim 6, characterized in that: The result feedback includes the following steps: If it is inconvenient to maintain the model based on the alarm prediction results, continue to monitor the data; If the model is modified, the deviated data points are added to the existing model for learning and training, or historical data with timestamps earlier than the predetermined time are discarded and learning and training is continued based on the remaining data and the newly added data.

Images