Alarm processing method, alarm system and monitoring system

Abstract

The invention provides an alarm processing method, an alarm system and a monitoring system, and is suitable for the technical field of alarm of monitoring data. The method comprises the following steps: obtaining index data obtained by carrying out index collection on each business system through an intelligent collector, and obtaining at least one abnormal data in the index data; wherein the abnormal data is obtained by performing data preprocessing on abnormal indexes in the index data through the intelligent collector; determining an alarm mode corresponding to each abnormal data based on each abnormal data, and generating alarm information corresponding to each abnormal data; and for each piece of abnormal data, giving an alarm based on the alarm mode and the alarm information. According to the embodiment of the invention, the false alarm rate can be effectively reduced, an appropriate alarm mode can be selected according to different abnormal data to give an alarm, and the normal work of each service system is ensured.

Description

TECHNICAL FIELD

[0001] The technical field relates to an alarm processing method, an alarm system and a monitoring system. BACKGROUND

[0002] With more and more enterprise business systems, the complexity of the system is higher and higher. The monitoring demand of various business systems such as databases, networks, storages, switches and servers is higher and higher. Moreover, the number of index collectors is increasing, and the number of alarms occurring every day is increasing. Alarms play an important role in early warning of potential problems, can predict the probability of failure in advance, and can reduce the occurrence of failure. At the same time, there is a certain requirement for the accuracy of the alarm, which cannot be false alarm or missed alarm.

[0003] Each server and its related network, service, etc. must report the monitoring status in real time. For example, node_exporter, mysql_exporter, redis_exporter, mongodb_exporter and other index collectors can collect the status of the service of the business system in real time and report it to the data storage system, and the alarm information can be sent to the mailbox through the alarm system.

[0004] However, the indexes collected by the existing index collectors are prone to false alarms due to network jitter and other problems, and all kinds of alarm information are sent through the mailbox, which can also lead to some urgent alarms not being handled in time, affecting the normal work of various business systems. SUMMARY

[0005] Therefore, the embodiments of the present application provide an alarm processing method, an alarm system and a monitoring system to solve the problem that the existing alarm is prone to false alarm or the alarm is prone to not being handled in time.

[0006] The first aspect of the embodiments of the present application provides an alarm processing method, comprising: obtaining index data obtained by an intelligent collector collecting indexes of various business systems, and obtaining at least one abnormal data in the index data; wherein the abnormal data is obtained by the intelligent collector performing data preprocessing on the abnormal indexes in the index data; determining an alarm mode corresponding to each abnormal data based on the abnormal data, and generating alarm information corresponding to each abnormal data; For each abnormal data, alarm is performed based on the alarm mode and the alarm information.

[0007] In a possible implementation, the intelligent collector includes at least one of a first intelligent collector, a second intelligent collector, a third intelligent collector, and a fourth intelligent collector, the business system includes at least one of a relational database, an operating system, a cache database, and a document database, and the index data includes at least one of first index data, second index data, third index data, and fourth index data. The abnormal data are obtained in at least one of the following manners: The first index data of the relational database are collected by the first intelligent collector, and the abnormal index in the first index data is subjected to data preprocessing to obtain the abnormal data; the first index data include first state indexes and slow logs, the first state indexes include at least one of service availability, CPU (Central Processing Unit) usage, memory usage, disk usage, and configuration table information, and the slow logs are used to record query languages of the relational database and execution times of the query languages; The second index data of the operating system are collected by the second intelligent collector, and the abnormal index in the second index data is subjected to data preprocessing to obtain the abnormal data; the second index data include at least one of fan running states, disk usage states, memory running states, and server running states; The third index data of the cache database are collected by the third intelligent collector, and the abnormal index in the third index data is subjected to data preprocessing to obtain the abnormal data; the third index data include at least one of command calling states and command running states; The fourth index data of the document database are collected by the fourth intelligent collector, and the abnormal index in the fourth index data is subjected to data preprocessing to obtain the abnormal data; the fourth index data include at least one of cache hit rates and numbers of cursor timeouts.

[0008] In a possible implementation, the abnormal index in the first index data is subjected to data preprocessing to obtain the abnormal data, including at least one of the following: When it is detected that the first state indexes exist abnormally, the continuous predetermined number of first state indexes are detected, and if the predetermined number of first state indexes all exist abnormally, the first state indexes are taken as the abnormal data; If it is determined based on the slow logs that the execution time of any query language is greater than a first preset time of the query language, the query language and the execution time of the query language are taken as the abnormal data; And / or, the abnormal index in the second index data is subjected to data preprocessing to obtain the abnormal data, including: when it is detected that the server running state is a fault state, the server running state is determined again based on at least one of the fan running states, the disk usage states, and the memory running states, and if the server running state is determined again to be the fault state, the server running state is taken as the abnormal data. and / or, performing data preprocessing on the abnormal indicators in the third indicator data to obtain abnormal data, including: when it is detected that the execution time of any to-be-executed command in the command running state exceeds a first preset threshold, taking the to-be-executed command and the execution time of the to-be-executed command as the abnormal data; and / or, performing data preprocessing on the abnormal indicators in the fourth indicator data to obtain abnormal data, including at least one of the following: when it is detected that there is a cursor timeout abnormality, counting the number of times of cursor timeout, and when the number of times of cursor timeout exceeds a preset number of times, taking the cursor timeout as the abnormal data; when it is detected that the cache hit rate meets a predetermined condition, taking the cache hit rate as the abnormal data; wherein the predetermined condition includes at least one of the following: the cache hit rate is lower than a second preset threshold within a second preset time, and a change rate of the cache hit rate exceeds a third preset threshold within a third preset time.

[0009] In one possible implementation, based on the abnormal data, an alarm mode corresponding to each abnormal data is determined, including: classifying the abnormal data to determine an alarm category corresponding to each abnormal data; determining the alarm mode of each abnormal data based on the alarm category and a preset corresponding relationship between the alarm category and the alarm mode.

[0010] In one possible implementation, the alarm category includes a normal alarm and a critical alarm. classifying the abnormal data to determine an alarm category corresponding to each abnormal data, including: if the abnormal data is a normal alarm indicator, determining that the alarm category corresponding to the abnormal data is a normal alarm; wherein the normal alarm indicator is used to indicate that the abnormal data does not affect the use of the business system and the continuous abnormal change of the abnormal data will affect the use of the business system. if the abnormal data is a critical alarm indicator, determining that the alarm category corresponding to the abnormal data is a critical alarm; wherein the critical alarm indicator is used to indicate that the abnormal data exceeds a corresponding preset threshold and / or the abnormal data causes the business system to be unavailable.

[0011] In one possible implementation, the alarm is performed based on the alarm mode and the alarm information, including: if the alarm category is a normal alarm, sending the alarm information to a preset mailbox address corresponding to the abnormal data; if the alarm category is a critical alarm, sending the alarm information to a preset mailbox address corresponding to the abnormal data, and based on the contact information corresponding to the abnormal data, communicating with a terminal device corresponding to the contact information to push the alarm information to the terminal device.

[0012] In a possible implementation, if the alarm category is a common alarm, the alarm information is sent to a preset mailbox address corresponding to the abnormal data, including: If the alarm category is a common alarm, the prediction information is determined based on the continuous abnormal change trend of the abnormal data, and the prediction information and the alarm information are sent to a preset mailbox address corresponding to the abnormal data.

[0013] The second aspect of the embodiment of the application provides an alarm system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0014] The third aspect of the embodiment of the application provides a monitoring system, including an intelligent collector and the alarm system of the second aspect. The intelligent collector is in communication connection with the alarm system, and the intelligent collector is configured to collect index data of each business system and perform data preprocessing on abnormal indexes in the index data to obtain abnormal data.

[0015] In a possible implementation, the intelligent collector includes a data collection module, a collection confirmation module, a machine learning module, a cache confirmation module, and a sending module; the intelligent collector is configured to perform at least one of the following for each index of the index data collected by the data collection module: When the machine learning module detects that the index has not changed, the index is stored in the distributed stream database at a predetermined time interval through the sending module; When the collection confirmation module detects that the index is abnormal, the index is sent to the machine learning module as abnormal data, so that the machine learning module sends the abnormal data to the cache confirmation module to record the abnormal data, and stores the abnormal data in the distributed stream database based on the record result of the cache confirmation module through the machine learning module; And / or, the alarm system includes a master node and at least two other nodes; the alarm system is configured to send alarm information through the master node based on a preset arbitration mechanism; wherein the arbitration mechanism includes sending the alarm information when more than half of the nodes in the master node and the other nodes vote to confirm.

[0016] Compared with the prior art, the embodiment of the application has at least the following technical effects: The alarm processing method of the first aspect of the embodiments of the present application can obtain the index data obtained by the intelligent collector collecting indexes of each business system, and obtain at least one abnormal data in the index data. Since the abnormal data is obtained by the intelligent collector pre-processing the abnormal indexes in the index data, the accuracy of the abnormal data is high, which can effectively prevent the alarm false alarm caused by network jitter and other problems, and effectively reduce the alarm false alarm rate. Then, the alarm processing method of the embodiments of the present application can also determine the alarm mode corresponding to each abnormal data based on each abnormal data, and generate the alarm information corresponding to each abnormal data; further, for each abnormal data, alarm based on the alarm mode and the alarm information. Therefore, the embodiments of the present application can select appropriate alarm mode for alarm according to different abnormal data, for example, some alarms that need to be handled in time can use urgent alarm mode, so that some urgent alarms can be handled in time, thereby ensuring the normal work of each business system.

[0017] It can be understood that the beneficial effects of the second aspect to the third aspect described above can be referred to the related description in the first aspect described above, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is a schematic diagram of a monitoring system framework provided by the embodiments of the present application; Figure 2 is a schematic diagram of the flow of an alarm processing method provided by the embodiments of the present application; Figure 3 is a schematic diagram of the structure of an intelligent collector provided by the embodiments of the present application; Figure 4 is a schematic diagram of the structure of an alarm processing device provided by the embodiments of the present application; Figure 5 is a schematic diagram of the structure of an alarm system provided by the embodiments of the present application. DETAILED DESCRIPTION

[0020] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0021] It is to be understood that the terminology "includes", "has", "holds", "contains" and / or "comprising", "including", "having" and their conjugates, as used herein, means "including but not limited to", and not to the exclusion of any other term or aspect.

[0022] It is also to be understood that the terminology "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items, and that the term "at least one of A and B" is equivalent to "A, B, or A and B".

[0023] As used in the description of the application and the appended claims, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon [the described condition or event] being detected" or "in response to [the described condition or event] being detected", depending on the context.

[0024] In addition, the terms "first", "second", "third", etc. as used in the description of the application and the appended claims are not used to denote or imply relative importance but are used to distinguish one element from another.

[0025] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in other embodiments" or "in still other embodiments" in various places throughout this specification are not necessarily all referring to the same embodiment, unless otherwise specified. Furthermore, the terms "comprise", "comprises", "comprising", "include", "includes", "including" and the like are synonymous with the term "contain", "contains", "containing" or "has", "has", "having" and are used in the sense of "including but not limited to", unless otherwise specified.

[0026] After research, it is found that Prometheus is a good monitoring data storage system suitable for small and medium-sized systems, but once the number of components is too large and the collected data is too much, there will be a risk of single point failure. The current popular node_exporter, mysql_exporter, redis_exporter, mongodb_exporter and the like can collect the status of the service in real time and report it to Prometheus, and through the alertmanager component, the alarm information can be sent to the mailbox. However, since the index collector only collects in real time, due to the time interval of its collection, and when it does not collect data or appears abnormal, it will not repair or repeatedly collect and judge the results, and false positives often occur. False positives often cause unnecessary trouble for important systems.

[0027] The commonly used technical solution is to deploy a set of prometheus, and deploy node_exporter, redis_exporter, mongodb_exporter, mysql_exporter and other components on each server node. Node_exporter is used to collect various indicators of Linux operating system, redis_expoter collects various indicators of Redis database, mysql_exporter collects various indicators of MySQL database, and mongodb_expoter collects various monitoring indicators of MongoDB database. Data is reported to Prometheus, alertmanager detects abnormal data and sends it to the email system or enterprise internal service system.

[0028] The existing alarm mode has the following disadvantages: (1) The prometheus component is prone to single point failure, the amount of monitoring data that can be stored is limited, and high availability cannot be achieved.

[0029] (2) The node_expoter, redis_exporter, mongodb_exporter, mysql_exporter and other collectors lack intelligent judgment and cannot handle network jitter and other problems, so there is a certain probability of false positives.

[0030] (4) The repeated alarm is only continuously sent without a higher level of alarm handling method, delaying the handling of the problem.

[0031] (5) It is easy to cause too many emails, alarm bombing, and too many emails need to spend too much time to handle.

[0032] The alarm processing method, the alarm system and the monitoring system provided by the embodiments of the present application aim to solve the above technical problems of the prior art.

[0033] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. It should be pointed out that the following embodiments can be mutually referenced, borrowed or combined. For the same terms, similar features and similar implementation steps in different embodiments, they will not be described repeatedly.

[0034] Referring to FIG. 1, Figure 1 As shown in FIG. 1, the embodiments of the present application provide a framework schematic diagram of a monitoring system. As shown in FIG. 1, Figure 1 The monitoring system includes an intelligent collector 11 and an alarm system 12 of the embodiments of the present application. The intelligent collector 11 is in communication connection with the alarm system 12. The intelligent collector 11 is used to collect index data of each business system and to pre-process abnormal indexes in the index data to obtain abnormal data.

[0035] In the embodiments of the present application, the index data is stored in a distributed stream database.

[0036] Optionally, the distributed stream database can use Tdengine. Tdengine is a new generation of relational stream database, which can store a large amount of data. As long as there is data, it can be stored. There is no need to worry about single point failure, and the performance of the database is very good. Massive data can be output in milliseconds. Tdengine supports index access, and each table of Tdengine has a timestamp, which is an index. The intelligent alarm system 12 can predict data growth trend, memory change trend and IO change trend by analyzing the historical data of Tdengine, and provide data support for decision analysis. Tdengine provides more storage saving time than prometheus, and the access speed is faster.

[0037] Referring to FIG. 2, Figure 2 As shown in FIG. 2, the embodiments of the present application provide a flowchart of an alarm processing method. The alarm processing method of the embodiments of the present application can be applied to the alarm system 12. As shown in FIG. 2, Figure 2 The alarm processing method includes steps S201 to S203.

[0038] S201, acquiring index data collected by the intelligent collector 11 from each business system, and acquiring at least one abnormal data in the index data; wherein the abnormal data is obtained by pre-processing abnormal indexes in the index data by the intelligent collector 11.

[0039] Optionally, the data preprocessing can be a preprocessing of the collected indicators, judgment, identification, for example: when an abnormal indicator is collected, it can be collected several times to ensure that each collection is abnormal, rather than directly reporting the abnormal indicator collected once, so as to ensure the accuracy of the collected indicators.

[0040] S202, determining an alarm mode corresponding to each abnormal data based on each abnormal data, and generating alarm information corresponding to each abnormal data.

[0041] In some embodiments, determining an alarm mode corresponding to each abnormal data based on each abnormal data comprises: performing alarm classification on each abnormal data to determine the alarm category corresponding to each abnormal data; determining the alarm mode of each abnormal data based on each alarm category and a preset corresponding relationship between alarm categories and alarm modes.

[0042] In some embodiments, the alarm category includes ordinary alarm and critical alarm; performing alarm classification on each abnormal data to determine the alarm category corresponding to each abnormal data comprises: If the abnormal data is an ordinary alarm indicator, it is determined that the alarm category corresponding to the abnormal data is ordinary alarm; wherein the ordinary alarm indicator is used to indicate that the abnormal data does not affect the use of the business system and the continuous abnormal change of the abnormal data will affect the use of the business system; If the abnormal data is a critical alarm indicator, it is determined that the alarm category corresponding to the abnormal data is critical alarm; wherein the critical alarm indicator is used to indicate that the abnormal data exceeds the corresponding preset threshold and / or the abnormal data causes the business system to be unavailable.

[0043] Optionally, the alarm classification of each abnormal data comprises: classifying based on at least one of whether the abnormal data affects the use of the business system, whether the continuous abnormal change of the abnormal data will affect the use of the business system, whether the abnormal data exceeds the corresponding preset threshold, and whether the abnormal data causes the business system to be unavailable; If the abnormal data does not affect the use of the business system and the continuous abnormal change of the abnormal data will affect the use of the business system, it is determined that the abnormal data is an ordinary alarm indicator; If the abnormal data exceeds the corresponding preset threshold and / or the abnormal data causes the business system to be unavailable, it is determined that the abnormal data is a critical alarm indicator.

[0044] Optionally, the embodiments of the present application can correspondingly set preset classification conditions for at least one of whether each abnormal data affects the use of the business system, whether the continuous abnormal change of the abnormal data will affect the use of the business system, whether the abnormal data exceeds the corresponding preset threshold, and whether the abnormal data causes the business system to be unavailable, and determine whether the abnormal data is an ordinary alarm indicator or a critical alarm indicator based on the preset classification condition corresponding to the abnormal data.

[0045] Optionally, the alarm mode can send an email, call the business responsible person, and the like.

[0046] The embodiment of the present application can alarm classify each abnormal data to determine the alarm category corresponding to each abnormal data, and then determine the alarm mode of each abnormal data based on each alarm category and the preset corresponding relationship between the alarm category and the alarm mode, so as to alarm in a targeted manner.

[0047] S203, for each abnormal data, alarm based on the alarm mode and the alarm information.

[0048] In some embodiments, the alarm based on the alarm mode and the alarm information comprises: If the alarm category is ordinary alarm, the alarm information is sent to the preset mailbox address corresponding to the abnormal data; If the alarm category is critical alarm, the alarm information is sent to the preset mailbox address corresponding to the abnormal data, and based on the contact information corresponding to the abnormal data, the terminal device corresponding to the contact information is connected in communication to push the alarm information to the terminal device.

[0049] The corresponding relationship between the alarm category and the alarm mode comprises: The alarm mode corresponding to the ordinary alarm is to send the alarm information to the preset mailbox address, that is, the email alarm; The alarm mode corresponding to the critical alarm is to send the alarm information to the preset mailbox address and to connect in communication with the terminal device corresponding to the contact information, that is, the email + telephone alarm.

[0050] The alarm processing method of the embodiment of the present application can obtain the index data obtained by the intelligent collector 11 collecting the indexes of each business system, and obtain at least one abnormal data in the index data. Since the abnormal data is obtained by the intelligent collector 11 pre-processing the abnormal indexes in the index data, the accuracy of the abnormal data is high, which can effectively prevent the alarm false alarm caused by network jitter and the like, and effectively reduce the alarm false alarm rate.

[0051] Then, the alarm processing method of the embodiment of the present application can also determine the alarm mode corresponding to each abnormal data based on each abnormal data, and generate the alarm information corresponding to each abnormal data; and then, for each abnormal data, alarm based on the alarm mode and the alarm information. Therefore, the embodiment of the present application can select a suitable alarm mode for alarm according to different abnormal data, for example, an urgent alarm mode can be used for some alarms that need to be handled in time, so that some urgent alarms can be handled in time, thereby ensuring the normal work of each business system.

[0052] In some embodiments, the smart collector 11 comprises at least one of a first smart collector, a second smart collector, a third smart collector, and a fourth smart collector, the business system comprises at least one of a relational database, an operating system, a cache database, and a document database, and the index data comprises at least one of first index data, second index data, third index data, and fourth index data.

[0053] In some embodiments, the abnormal data is obtained by at least one of the following manners: The first index data of the relational database is collected by the first smart collector, and the abnormal index in the first index data is subjected to data preprocessing to obtain the abnormal data; the first index data comprises first state indexes and slow logs, the first state indexes comprise at least one of service availability, CPU usage, memory usage, disk usage, and configuration table information, and the slow logs are used to record query languages of the relational database and execution times of the query languages; The second index data of the operating system is collected by the second smart collector, and the abnormal index in the second index data is subjected to data preprocessing to obtain the abnormal data; the second index data comprises at least one of fan running states, disk usage states, memory running states, and server running states; The third index data of the cache database is collected by the third smart collector, and the abnormal index in the third index data is subjected to data preprocessing to obtain the abnormal data; the third index data comprises at least one of command calling states and command running states; The fourth index data of the document database is collected by the fourth smart collector, and the abnormal index in the fourth index data is subjected to data preprocessing to obtain the abnormal data; the fourth index data comprises at least one of cache hit rates and numbers of cursor timeouts.

[0054] Further, the data preprocessing of the abnormal index in the first index data to obtain the abnormal data comprises at least one of the following: When it is detected that the first state indexes are abnormal, the continuous predetermined number of first state indexes are detected, and if the predetermined number of first state indexes are all abnormal, the first state indexes are taken as the abnormal data; If it is determined based on the slow logs that the execution time of any query language is greater than a first preset time of the query language, the query language and the execution time of the query language are taken as the abnormal data; And / or, the data preprocessing is performed on the abnormal indicators in the second indicator data to obtain abnormal data, including: when it is detected that the server running state is a fault state, the server running state is determined again based on at least one of the fan running state, the disk usage state and the memory running state, and if the server running state determined again is a fault state, the server running state is taken as the abnormal data; And / or, the data preprocessing is performed on the abnormal indicators in the third indicator data to obtain abnormal data, including: when it is detected that the execution time of any to-be-executed command in the command running state exceeds a first preset threshold, the to-be-executed command and the execution time of the to-be-executed command are taken as the abnormal data. And / or, the data preprocessing is performed on the abnormal indicators in the fourth indicator data to obtain abnormal data, including at least one of the following: When it is detected that there is a cursor timeout abnormality, the number of times of cursor timeout is counted, and when the number of times of cursor timeout exceeds a preset number of times, the cursor timeout is taken as the abnormal data. When it is detected that the cache hit rate meets a predetermined condition, the cache hit rate is taken as the abnormal data; the predetermined condition includes at least one of that the cache hit rate is lower than a second preset threshold within a second preset time and that a change rate of the cache hit rate exceeds a third preset threshold within a third preset time.

[0055] The first preset time, the first preset threshold, the second preset time, the second preset threshold, the third preset time and the third preset threshold in the embodiments of the application are all set in advance according to actual application.

[0056] As an example, the first intelligent collector, the second intelligent collector, the third intelligent collector and the fourth intelligent collector correspond to ML_mysql_exporter, ML_node_exporter, ML_redis_exporter and ML_mongodb_exporter respectively. Correspondingly, the relational database, the operating system, the cache database and the document database can be a MySQL database, a Linux operating system, a Redis database and a MongoDB database respectively.

[0057] ML_mysql_exporter is a smart data collector for MySQL. It can accurately collect metrics and pre-process, judge, and identify the collected metrics. Accurate collection means that when abnormal metrics are collected, they can be re-collected to ensure that each collection captures an abnormal metric, rather than reporting it directly after a single abnormal metric. For example, if the environment currently reports health status every minute, and network jitter occurs during the collection process, preventing data from being reported to Tdengine, the machine learning-based smart data collector 11 will collect data five times consecutively. If no healthy metrics are found in five collections, it indicates that the service is indeed down or the entire system is unavailable, requiring reporting of this status and sending emails to relevant stakeholders through subsequent procedures. If the metrics are normal, collection will continue at the previous frequency.

[0058] Simultaneously, it's necessary to exclude duplicate normal indicators. For example: a value of 1 from `mysql_up` indicates a healthy database; `mysql_server_info` represents the database version, and this value won't change unless upgraded; `mysql_transaction_isolation_level` is generally stable unless manually modified; `page_size` is the page size, fixed after database creation; and `mysql_global_variables_wait_timeout` is generally fixed unless manually modified. Record abnormal indicators, such as excessively high network traffic exceeding 150MB / s, `service_unreachable`, `CPU Too High`, excessively high memory usage, and disk usage exceeding 90%. Continuously learn and analyze these abnormal indicators to prevent false alarms and accurately record the frequency of network jitter.

[0059] Currently, existing technology using mysql_exporter only collects and reports transient metrics on anomalies once, such as service availability being collected once per minute. If no result is returned on a particular occasion, this is packet loss. In this case, mysql_exporter will directly report a service failure, and we will receive a "Server_down" alert in the alert email. However, upon checking the service, it is found that the service is still running normally and has not experienced any restart or failure. This occasional packet loss phenomenon is called network jitter, but mysql_exporter mistakenly interprets it as a service failure.

[0060] The ML_mysql_exporter of the embodiment of the present application can repeatedly confirm whether the exception continues to occur through its exception tracking function when sending the exception, so as to make a judgment, is more intelligent than the mysql_exporter, and avoids similar repeated alarms. The slow log of MySQL, the delay of master-slave replication, and the change of connection data are accurately recorded. The slow log is a database-specific indicator. The original mysql_exporter does not collect specific slow SQL logs, but the ML_mysql_exporter intelligent collector not only collects the indicators of the database runtime, but also collects the database slow log. Through threshold setting, the execution SQL higher than the threshold is extracted and stored in Tdengine. Finally, the alarm system 12 sends an email, which is convenient for the administrator of the database to optimize the database and improve the running efficiency.

[0061] ML_node_exporter: is a collector based on machine learning linear regression algorithm (Linear Regression) of Linux operating system, which can collect all the contents collected by node_exporter, and can make a pre-judgment on the collected contents. If it is a downtime class, first confirm whether other services are alive, whether it is only single failure or continuous failure for a period of time. Whether the fan is normal, whether the disk is normal, and whether the memory indicator is normal. Through the confirmation of other indicators, it is confirmed that the machine is down only when it is really down, rather than considering it as down after a failure, ignoring the network jitter situation.

[0062] The confirmation mechanism of the ML_node_exporter of the embodiment of the present application does not require human intervention. The ML_node_exporter will automatically repeatedly confirm the abnormal situation. After multiple rounds of verification, it is confirmed that the exception is very accurate and needs to be immediately intervened. If the exception is not handled within a period of time, the alarm system 12 will call to urge the solution within a certain period of time. The downtime class will immediately send an email and make a call. If the disk usage is too high, an email notification will be sent, and a call will be made when the total disk capacity reaches 95%. At this time, the work of the business system is in a high-risk state. If such an alarm is not handled, the intelligent alarm system 12 will handle it. At the same time, in actual application, human intervention can also be performed according to whether it is a production system or a test system. If it is a test machine, it is not important and does not need to be handled temporarily, and such an alarm can be temporarily closed.

[0063] ML_redis_exporter: is the collector of redis database system, which can collect all the contents that redis_exporter can collect, for example, redis_commands_failed_calls_total{cmd="xxx"}, redis_commands_duration_seconds_total{cmd="set"}, redis_commands_failed_calls_total{cmd="hdel"}, the total duration of command call, the number of failed calls. redis_commands_rejected_calls_total{cmd="xxx"}, the number of rejected calls, such as: redis_commands_rejected_calls_total{cmd="del"}, redis_total_error_replies, which usually means that your Redis client encountered an error reply when executing the command. redis_uptime_in_seconds, the running time of redis. redis_up, whether redis is alive.

[0064] In addition, ML_redis_exporter can also capture slow commands, for example, a command of redis is generally executed in microseconds, if the execution time of a command exceeds 100ms, the command needs to be sent to Tdengine storage, and sent to the mailbox of the operation and maintenance personnel through the intelligent alarm system 12, to prevent the slow command from being found after the fact. From which to provide the basis for optimizing Redis. ML_redis_exporter is more intelligent than redis_exporter, which can pre-process the collected data to prevent false alarms from being sent, while reducing the number of collected indicators sent, and improving the accuracy of collection.

[0065] ML_mongodb_exporter: is the collector of MongoDB database, the abnormality of cache hit rate can be that the cache hit rate is continuously lower than 90% or even 80%. Continuous detection and attention to changes, and if the change rate exceeds 5%, an abnormal index is sent. When the cursor timeout occurs, the number of times of cursor timeout is counted, and after reaching the preset number, an alarm index is recorded.

[0066] Referring to Figure 3 The embodiment of the application provides a structure diagram of an intelligent collector 11. As shown in Figure 3As shown, the intelligent collector 11 includes a data collection module 111, a collection confirmation module 112, a machine learning module 113, a cache confirmation module 114, and a sending module 115. The sending module 115 is configured to send the index data to the distributed streaming database.

[0067] Optionally, the first intelligent collector, the second intelligent collector, the third intelligent collector, and the fourth intelligent collector all adopt the structure of the intelligent collector 11.

[0068] In some embodiments, the intelligent collector 11 is configured to perform at least one of the following for each index of the index data collected by the data collection module 111: When the machine learning module 113 detects that the index has not changed, the index is stored in the distributed streaming database at a predetermined time interval through the sending module 115.

[0069] When the collection confirmation module 112 detects that the index has an anomaly, the index is sent to the machine learning module 113 as abnormal data, so that the machine learning module 113 sends the abnormal data to the cache confirmation module 114 for recording, and stores the abnormal data in the distributed streaming database based on the recording result of the cache confirmation module 114 through the machine learning module.

[0070] The machine learning module 113 can analyze the abnormal index through a machine learning algorithm.

[0071] The data collection module 111 is an index receiving module that periodically collects monitored indexes. After the first collection, the data is directly pushed to the collection confirmation module 112, passes through the machine learning module 113, and is sent to the Tdengine through the sending module 115 for initialization.

[0072] The collection confirmation module 112 confirms the collected monitoring indexes. For abnormal data, the cache confirmation module 114 is used for recording and continuous collection to confirm whether the anomaly persists. If the anomaly persists, an alarm index is first sent to the machine learning module 113. After the index is learned by the machine learning module, the learning result is written to the cache confirmation module 114. Low-level indexes are first sent to the mailbox and continuously monitored, so low-level indexes such as version number, running time, data volume written per second, and data volume sent are regular indexes. If it is a high-level index such as a production system downtime index, it is pushed to the sending module 115 and finally sent to the Tdengine. The normal index is sent to the TDengine for persistence in sequence through the collection confirmation module 112, the machine learning module 113, and the sending module 115.

[0073] Machine learning module 113: collection index machine learning module. The index that does not change in the short term, machine learning module 113 pushes the collection index to the sending module 115 regularly, and then to the Tdengine. When the collection confirmation module 112 identifies that the index is an abnormal index, for the abnormal index, the machine learning module 113 will send it to the cache confirmation module 114 for recording, and continuously monitor the number of times of abnormal occurrence, the duration, whether the alarm has been removed. Continue to collect the same abnormal index, send it only once, if the index continues to deteriorate, and the change rate reaches 5%, resend the alarm once. At this time, the index has not reached the worst. If the monitoring finds that the index has reached the worst warning value, the alarm system 12 immediately notifies the business manager by phone to handle immediately, prevents the fault from worsening, causes the risk of business suspension, and no longer repeats the alarm until the solution. After the fault is solved, the data collection module 111 re-collects the normal index, and after the index returns to normal, the cache confirmation module 114 will mark the index that has returned to normal, indicating that it has been abnormal. The marked abnormal time and number of times can be used to judge the frequency of the occurrence of the abnormal index within a certain period of time. If the abnormal index does not appear for more than 7 days, the record related to the index will be removed from the cache confirmation module 114, that is, it is no longer necessary to continuously pay attention to it. If it is network jitter, for example, the previous second detects an abnormality, and the next second detects success, it can be determined as network jitter. If network jitter occurs multiple times, the host will be marked as network jitter, and a network jitter message will be sent regularly instead of continuously sending the abnormality caused by network jitter, reducing the number of alarm emails and improving the accuracy of the alarm.

[0074] Sending module 115: collection index sending module. Responsible for sending the index and monitoring whether the collector itself is alive. All sent indexes are valid indexes, that is, whether the collected index is abnormal has been repeatedly verified by the collection confirmation module 112 and confirmed by the machine learning module 113. In this way, the indexes sent to Tdengine are accurate indexes.

[0075] TDengine, as a distributed stream database, can store a large amount of data and can retain time as needed. It solves the problem of high availability of index data and is more convenient for operation and maintenance. As a repository of indexes, it has been widely used in Internet of Things systems. Its use of parent tables and child tables is also applicable to monitoring systems. The same component can create a parent table, and each index under each component can create a child table that only saves single index data. In this way, by querying the related child table, the current state of the index can be known. The intelligent alarm system 12 detects the instantaneous value of the index of the child table under each component of Tdengine in real time to confirm whether the index is abnormal. And make a judgment on whether to send an alarm.

[0076] In practical applications, in order to prevent false alarms, the collected indicators must be filtered in advance by a machine learning algorithm, and it is determined whether the abnormal indicators are indeed so. To prevent single point failure of Prometheus, Tdengine distributed cluster can be used to store the collected indicators. Tdengine is a new generation of streaming data storage system, which supports high availability, scalability, high processing performance, large storage capacity, and is suitable for retaining a certain time according to time. The table operation is convenient, and it is suitable for querying using SQL, which is more convenient and intuitive than Prometheus, avoids or reduces single point failure, and reduces the false alarm rate.

[0077] The intelligent collector 11 of the embodiment of the application can at least achieve the following technical effects: (1) The collected abnormal indicators have a repeated confirmation mechanism, which ensures that the results are abnormal multiple times before determining that the indicators are abnormal, and the results are more accurate.

[0078] (2) The collected repeated indicators have a filtering mechanism to avoid repeatedly writing repeated data into storage, and only write indicators that have changed.

[0079] (3) The machine learning module 113 identifies and learns abnormal indicators, analyzes and records abnormal indicators, and predicts the occurrence of the indicators again. After a period of time, the monitoring of the indicators is cancelled.

[0080] Among them, Tdengine is generally distributed as a storage system for indicators. Each node can be accessed, and the same indicator data is queried from any node, and the result is the same. Automatic high availability and load balancing.

[0081] In some embodiments, the alarm system 12 includes a master node and at least two other nodes; the alarm system 12 is configured to send alarm information through the master node based on a preset arbitration mechanism; wherein the arbitration mechanism includes sending alarm information when more than half of the nodes in the master node and the other nodes vote to confirm.

[0082] The alarm system 12 of the embodiment of the application can use an intelligent alarm system AI_NOTICE, which is a new generation of intelligent alarm system and is also distributed. Each node can run independently and form high availability. Generally, only the master node can send alarms. However, each node can query and analyze indicators. When an AI_NOTICE queries an abnormal indicator and needs to send alarm information, the command will be sent to the master node of the AI_NOTICE, and it is confirmed that other AI_NOTICES have not reported the same alarm to avoid repeated sending of alarms.

[0083] Optionally, the AI_NOTICE master node and other nodes form an arbitration mechanism, by deploying an odd number of AI_NOTICE, each transmission needs to be confirmed by other node AI_NOTICE. Minority submits to the majority, the master node sends when it gets more than half of the votes. Whether the time spent on this vote affects the sending of the alarm. We can refer to the voting mechanism of TIDB database, the voting is a very fast process, which can be completed in milliseconds, and has no effect on the entire alarm system.

[0084] Among them, the intelligent alarm system AI_NOTICE is a new generation of intelligent alarm system more intelligent than alertmanager. It not only forwards the alarm, but also can aggregate the alarm. The intelligent alarm system AI_NOTICE is independent of Tdengine, which confirms the content to be sent by querying the data collected by Tdengine every second. For example: when ML_mysql_exporter collects a table configured in the DBLE(database middleware) configuration file, but the backend database does not retrieve the table; or the backend retrieves the table, but the DBLE(database middleware) configuration file in the front end does not configure the table. At this time, the intelligent collector 11 needs to check whether all DBs involve the table according to the configuration item of the configuration file, and store the information to Tdengine.

[0085] The intelligent alarm system AI_NOTICE collects the complete information immediately after collecting the indicators of multiple nodes related to the table in Tdengine, and then sends the alarm. In the past alarm system, when the DBLE(database middleware) configuration file has a table, but the backend database is not created, the alarm will be reported one by one according to the pre-configured data node datan, at this time a large number of alarms will be received, and the intelligent alarm system only sends one through the aggregation function, and prompts which nodes have not created the table. The intelligent alarm system AI_NOTICE can judge whether the current alarm is a repeated alarm or a new alarm. Is it a normal alarm or an urgent alarm? The normal alarm directly sends an email, and the urgent alarm sends an email and calls the business responsible person's phone, so as to intervene in the alarm in time and prevent the business from being unavailable.

[0086] In some embodiments, if the alarm category is a normal alarm, the alarm information is sent to the preset mailbox address corresponding to the abnormal data, including: If the alarm category is a normal alarm, the prediction information is determined based on the continuous abnormal change trend of the abnormal data, and the prediction information and the alarm information are sent to the preset mailbox address corresponding to the abnormal data.

[0087] For example, the intelligent alarm system AI_NOTICE can aggregate historical similar alarms to generate statistical charts. By observing the trend of the same alarms, the monitoring personnel can predict when the same alarm may occur in the future. For example, if the collected indicator is the disk usage, and the growth rate of disk usage is 5% every day in the past week, and the current disk usage has reached 50%, it can be predicted that the disk usage of the server will reach the maximum in the next 10 days, and the prediction information will be written in the remarks and sent to the operation and maintenance personnel together with the alarm information. For this situation, it is necessary to make plans in advance, replace the storage device in advance, or expand, upgrade it to avoid business interruption and cause incalculable losses. In this way, plans can be made in advance, rather than waiting for resources to be insufficient and finding resources on an ad hoc basis. If the resources are not in place, the time cost of purchasing resources and the waiting time cost of applying for resources will affect the timely response to problems and increase the operation and maintenance cost.

[0088] In practical applications, the alarms of the embodiments of the present application can be divided into the following cases: Normal normal indicators: For normal indicators such as version information, kernel version, database version, CPU core number, total memory size, system running time, if there is no change, no alarm will be given.

[0089] Warning level alarm: that is, ordinary alarm, which generally exists potential problems, but does not affect the use of the business system, but continuous deterioration can lead to the unavailability of the business system. This type of alarm is the alarm that needs to be focused on in addition to the Critical type of alarm. Although it does not affect the business system at present, after a period of time, the alarm will be upgraded to the Critical type of alarm. Handling such alarms is also an important way to improve system stability. For example, the disk usage exceeds 80%, the memory usage exceeds 85%, the CPU usage exceeds 70%, the non-uniform memory access (NUMA) function is not closed, and some dependent packages are not installed. This type of alarm is directly sent by email, and the relevant operation and maintenance personnel will handle it after seeing it. If it does not continue, the indicator continues to deteriorate, and the early warning value rises by 5% compared with the last alarm, the alarm will be sent again to prompt.

[0090] Critical class alarm: that is, critical alarm, such as resource usage remaining less than 10%, memory usage more than 90%, CPU usage 80%, disk usage more than 95%; Server_Down (server down), Service_down (service down), Server_Hang (service dead unavailable) and the like not only send email, but also automatically dial the phone of the operation and maintenance personnel every 30 minutes, and only call the relevant person in charge, urging to solve. After the factor causing the alarm is eliminated, no email is sent and no phone call is made. At the same time, before the problem is repaired, in order to prevent such alarms and phone calls from continuing to notify, it can be temporarily disabled for a period of time, which is the alarm processing time. After the problem is handled, the disablement is cancelled.

[0091] In summary, the intelligent alarm system 12 of the embodiments of the present application has the following advantages compared with alertmanager: (1) High availability, load balancing, any node can send an alarm, but the master node needs to arbitrate to ensure that the alarm sent is unique. (2) Alarm classification, through predefined classification, the next action of the alarm is divided. It is divided into regular alarm, which does not notify; ordinary alarm notifies by email; critical alarm notifies by email + phone. Critical alarm will make a phone call again to urge processing if it is not processed within 30 minutes.

[0092] (3) The critical alarm phone notification mechanism can directly reach the corresponding person's mobile phone, rather than dialing to all people in the group, which is more targeted.

[0093] The single server used in the embodiments of the present application generally does not exceed 1T of data volume, Prometheus uses a single machine version, and when the data is too much, the access performance will decrease, while the storage capacity of Tdengine is not limited, the cluster can store as much data as it has, and each table has an index, and the query performance is not affected.

[0094] In actual application, it takes about 17 seconds to write 1000 time series data points each time. While Tdengine uses the same data to write about 500 milliseconds, and using the JNI client reduces to 100 milliseconds, which is obviously better than the former in performance.

[0095] From the perspective of intelligent collection, the intelligent collector 11 of the machine learning type can preprocess, judge and analyze the collected indicators, prevent the same indicators from being reported repeatedly, prevent the collection of an exception from being reported as an error, and must process the error before reporting, prevent false positives, and the like. For example, the node_exporter collects data every second or every 1 minute. If the network is jittered and no result is returned in 1 second, it will immediately report that the system is down, but in fact it is only network jitter, and the next second will return to normal. The intelligent collector can avoid this problem.

[0096] The intelligent alarm system 12 can classify alarms. Compared with the direct forwarding alarm of the alertmanager, the intelligent alarm system can confirm the alarm, generally send an alarm once a period, immediately send a high-risk alarm, and make a telephone call. If the high-risk problem is not solved within a set period of time, the other operation and maintenance personnel in the same group are notified to urge the solution by telephone until the high-risk alarm is eliminated or shielded. The ordinary alarm can be continuously observed, the trend can be predicted, and a trend report can be sent. In the ordinary alarm system, there is no trend report, and only the current instantaneous indicator is reported.

[0097] Based on the above technical content, the embodiments of the present application can at least achieve the following technical effects: (1) The safety of collected data is improved. Tdengine can ensure that data is not lost through its original high availability, and solve the single point failure problem.

[0098] (2) The intelligence of data collection is improved. Through the intelligent collector 11: ML_node_exporter, ML_mysql_exporter, ML_redis_exporter, ML_mongodb_exporter, the collected data is preprocessed, and for the data that may exist network jitter, the collected data of the network is combined and judged to obtain a conclusion and intelligently report.

[0099] (3) The convenience of monitoring and alarming is improved. The intelligent alarm system 12 can send an ordinary alarm to a mail to notify, and immediately call the system administrator for a key alarm to immediately process. The alarm trend report can be sent regularly, and the future risk can be predicted in advance.

[0100] Referring to Figure 4 , the embodiments of the present application provide a structural schematic diagram of an alarm processing device 40. As Figure 4 shown, the alarm processing device 40 includes an acquisition module 401, a processing module 402 and an alarm module 403.

[0101] The acquisition module 401 is configured to acquire index data obtained by the intelligent collector collecting indexes of each business system, and acquire at least one abnormal data in the index data; wherein the abnormal data is obtained by the intelligent collector performing data preprocessing on abnormal indexes in the index data; The processing module 402 is configured to determine an alarm mode corresponding to each abnormal data based on each abnormal data, and generate alarm information corresponding to each abnormal data; The alarm module 403 is configured to, for each abnormal data, perform alarm based on the alarm mode and the alarm information.

[0102] In some embodiments, the intelligent collector includes at least one of a first intelligent collector, a second intelligent collector, a third intelligent collector, and a fourth intelligent collector, the business system includes at least one of a relational database, an operating system, a cache database, and a document database, and the index data includes at least one of first index data, second index data, third index data, and fourth index data; The abnormal data is obtained by at least one of the following ways: The first index data of the relational database is collected by the first intelligent collector, and data preprocessing is performed on abnormal indexes in the first index data to obtain abnormal data; wherein the first index data includes first state indexes and slow logs, the first state indexes include at least one of service availability, CPU usage, memory usage, disk usage, and configuration table information, and the slow logs are used to record query languages of the relational database and execution times of the query languages; The second index data of the operating system is collected by the second intelligent collector, and data preprocessing is performed on abnormal indexes in the second index data to obtain abnormal data; wherein the second index data includes at least one of fan running states, disk usage states, memory running states, and server running states; The third index data of the cache database is collected by the third intelligent collector, and data preprocessing is performed on abnormal indexes in the third index data to obtain abnormal data; wherein the third index data includes at least one of command call states and command running states; The fourth index data of the document database is collected by the fourth intelligent collector, and data preprocessing is performed on abnormal indexes in the fourth index data to obtain abnormal data; wherein the fourth index data includes at least one of cache hit rates and numbers of cursor timeouts.

[0103] In some embodiments, the data preprocessing on the abnormal indexes in the first index data to obtain the abnormal data includes at least one of the following: When it is detected that the first state indicators have an abnormality, the first state indicators in a continuous predetermined number are detected, if the predetermined number of first state indicators all have an abnormality, the first state indicators are taken as abnormal data; If the execution time of any query language is greater than the first preset time of the query language based on the slow log, the query language and the execution time of the query language are taken as abnormal data; And / or, the abnormal indicators in the second indicator data are data pre-processed to obtain abnormal data, including: when it is detected that the server running state is a fault state, the server running state is determined again based on at least one of the fan running state, the disk usage state and the memory running state, if the server running state is determined again as a fault state, the server running state is taken as abnormal data; And / or, the abnormal indicators in the third indicator data are data pre-processed to obtain abnormal data, including: when it is detected that the execution time of any to-be-executed command in the command running state exceeds the first preset threshold, the to-be-executed command and the execution time of the to-be-executed command are taken as abnormal data; And / or, the abnormal indicators in the fourth indicator data are data pre-processed to obtain abnormal data, including at least one of the following: When it is detected that there is an abnormality of cursor timeout, the number of cursor timeouts is counted, and when the number of cursor timeouts exceeds a preset number, the cursor timeout is taken as abnormal data; When it is detected that the cache hit rate meets a predetermined condition, the cache hit rate is taken as abnormal data; wherein the predetermined condition includes at least one of that the cache hit rate is lower than a second preset threshold within a second preset time and that the change rate of the cache hit rate exceeds a third preset threshold within a third preset time.

[0104] In some embodiments, the processing module 402 is configured to classify each abnormal data into an alarm category, and determine an alarm mode of each abnormal data based on the alarm category and a preset correspondence between alarm categories and alarm modes.

[0105] In some embodiments, the alarm categories include common alarms and critical alarms; the processing module 402 is configured to determine that the alarm category corresponding to the abnormal data is a common alarm if the abnormal data is a common alarm indicator; wherein the common alarm indicator is used to indicate that the abnormal data does not affect the use of the business system and the continuous abnormal change of the abnormal data will affect the use of the business system. If the abnormal data is a critical alarm indicator, the alarm category corresponding to the abnormal data is determined to be a critical alarm; wherein the critical alarm indicator is used to indicate that the abnormal data exceeds a corresponding preset threshold and / or the abnormal data causes the business system to be unavailable.

[0106] In some embodiments, the alarm module 403 is configured to send the alarm information to the preset mailbox address corresponding to the abnormal data if the alarm category is a common alarm, and send the alarm information to the preset mailbox address corresponding to the abnormal data and communicate with the terminal device corresponding to the contact information based on the contact information corresponding to the abnormal data to push the alarm information to the terminal device if the alarm category is a critical alarm.

[0107] In some embodiments, the processing module 402 is configured to determine the prediction information based on the continuous abnormal change trend of the abnormal data if the alarm category is a common alarm, and the alarm module 403 is configured to send the prediction information and the alarm information to the preset mailbox address corresponding to the abnormal data.

[0108] In applications, each module in the alarm processing apparatus 40 can be a software program module, can be realized by different logic circuits integrated in a processor, or can be realized by multiple distributed processors.

[0109] The alarm processing apparatus 40 of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the alarm processing apparatus 40 of the embodiments of the present application correspond to the steps in the method of the embodiments of the present application. For the detailed functions of each module of the apparatus, refer to the description of the corresponding method in the foregoing description, which will not be repeated here.

[0110] Referring to Figure 5 FIG. 1 shows a structural schematic diagram of an alarm system 12 provided by an embodiment of the present application. As shown in Figure 5 The alarm system 12 of the embodiments of the present application includes a memory 122, a processor 121, and a computer program 123 stored in the memory 122 and executable on the processor 121. The processor 121 implements the steps of the method of the embodiments of the present application when executing the computer program.

[0111] The alarm system 12 can be a desktop computer, a notebook computer, a palm computer, a cloud server, or other computing devices. The alarm system 12 can include, but is not limited to, the processor 121 and the memory 122. Those skilled in the art can understand that Figure 5 The alarm system 12 is only an example and does not constitute a limitation on the alarm system 12. It can include more or fewer components than those shown in the figure, or combine certain components or different components, for example, it can also include input / output devices, network access devices, etc.

[0112] The processor 121 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0113] The memory 122 can be an internal storage unit, such as a hard disk or a memory, in some embodiments. The memory 122 can be a removable / non-removable, volatile / non-volatile computer system storage medium, such as a non-volatile memory, for reading and writing non-volatile magnetic media. The memory 122 can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., in other embodiments. The memory 122 is used to store an operating system, application programs, a boot loader, data, and other programs, such as program codes of computer programs, etc. The memory 122 can also be used to temporarily store data that has been output or will be output.

[0114] It should be noted that the information interaction, execution process, etc. between the above devices / units, since based on the same concept as the method embodiments of the present application, the specific functions and the technical effects brought by them can be referred to the method embodiments part, and will not be repeated here.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or software. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the application. The specific working process of the unit and module in the system can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

[0116] The embodiment of the application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in each method embodiment.

[0117] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above embodiment methods can be completed by a computer program to instruct related hardware. The computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiment methods can be included. The storage medium can be a magnetic disc, an optical disc, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory, a Hard Dish Drive (HDD) or a Solid-State Drive (SSD), etc. The storage medium can also include a combination of the above-mentioned types of memories.

[0119] The embodiments of the present application provide a computer program product, when the computer program product is executed on a processor, so that the processor executes the steps in the above-mentioned various method embodiments.

[0120] In the above-mentioned embodiments, the description of each embodiment has its own focus. The parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0121] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0122] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / network device and method can be implemented by other ways. For example, the apparatus / network device embodiments described above are only schematic, for example, the division of the above-mentioned modules or units is only a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0123] The units described as separate parts above can or can not be physically separate, and the parts shown as units can or can not be physical units, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0124] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An alarm processing method characterized by comprising: The method comprises: obtaining index data obtained by an intelligent collector collecting indexes of each business system, and obtaining at least one abnormal data in the index data; wherein the abnormal data is obtained by the intelligent collector performing data preprocessing on abnormal indexes in the index data; determining an alarm mode corresponding to each abnormal data based on each abnormal data, and generating alarm information corresponding to each abnormal data; for each abnormal data, performing alarm based on the alarm mode and the alarm information.

2. The alert processing method of claim 1, wherein, The intelligent collector comprises at least one of a first intelligent collector, a second intelligent collector, a third intelligent collector and a fourth intelligent collector, the business system comprises at least one of a relational database, an operating system, a cache database and a document database, and the index data comprises at least one of first index data, second index data, third index data and fourth index data; The abnormal data is obtained by at least one of the following ways: collecting first index data of the relational database by the first intelligent collector, performing data preprocessing on abnormal indexes in the first index data to obtain the abnormal data; wherein the first index data comprises first state indexes and slow logs, the first state indexes comprise at least one of service availability, CPU usage, memory usage, disk usage and configuration table information, and the slow logs are used to record query languages of the relational database and execution times of the query languages; collecting second index data of the operating system by the second intelligent collector, performing data preprocessing on abnormal indexes in the second index data to obtain the abnormal data; wherein the second index data comprises at least one of fan running state, disk usage state, memory running state and server running state; collecting third index data of the cache database by the third intelligent collector, performing data preprocessing on abnormal indexes in the third index data to obtain the abnormal data; wherein the third index data comprises at least one of command calling state and command running state; collecting fourth index data of the document database by the fourth intelligent collector, performing data preprocessing on abnormal indexes in the fourth index data to obtain the abnormal data; wherein the fourth index data comprises at least one of cache hit rate and number of cursor timeouts.

3. The alert processing method of claim 2, wherein, The data preprocessing on the abnormal indexes in the first index data to obtain the abnormal data comprises at least one of the following: when it is detected that the first state indexes exist abnormally, detecting a predetermined number of continuous first state indexes, if a predetermined number of first state indexes all exist abnormally, taking the first state indexes as the abnormal data; if it is determined based on the slow logs that the execution time of any query language is greater than a first preset time of the query language, taking the query language and the execution time of the query language as the abnormal data; And / or, the data preprocessing of the abnormal indicators in the second indicator data to obtain the abnormal data comprises: when it is detected that the server running state is a fault state, determining the server running state again based on at least one of the fan running state, the disk usage state and the memory running state, and if it is determined that the server running state is a fault state again, taking the server running state as the abnormal data; And / or, the data preprocessing of the abnormal indicators in the third indicator data to obtain the abnormal data comprises: when it is detected that the execution time of any to-be-executed command in the command running state exceeds a first preset threshold, taking the to-be-executed command and the execution time of the to-be-executed command as the abnormal data; And / or, the data preprocessing of the abnormal indicators in the fourth indicator data to obtain the abnormal data comprises at least one of the following: When it is detected that there is an abnormality of the cursor timeout, counting the number of times of the cursor timeout, and when the number of times of the cursor timeout exceeds a preset number of times, taking the cursor timeout as the abnormal data; When it is detected that the cache hit rate meets a predetermined condition, taking the cache hit rate as the abnormal data; wherein the predetermined condition comprises at least one of the cache hit rate being lower than a second preset threshold within a second preset time and a change rate of the cache hit rate exceeding a third preset threshold within a third preset time.

4. The alert handling method of any of claims 1-3, wherein, The determination of the alarm mode corresponding to each of the abnormal data based on each of the abnormal data comprises: Alarm classification of each of the abnormal data to determine the alarm category corresponding to each of the abnormal data; Determination of the alarm mode of each of the abnormal data based on each of the alarm categories and a preset correspondence between alarm categories and alarm modes.

5. The alert processing method of claim 4, wherein, The alarm category comprises a normal alarm and a critical alarm; The alarm classification of each of the abnormal data to determine the alarm category corresponding to each of the abnormal data comprises: If the abnormal data is a normal alarm indicator, determining that the alarm category corresponding to the abnormal data is the normal alarm; wherein the normal alarm indicator is used to indicate that the abnormal data does not affect the use of the business system and the continuous abnormal change of the abnormal data will affect the use of the business system; If the abnormal data is a critical alarm indicator, determining that the alarm category corresponding to the abnormal data is the critical alarm; wherein the critical alarm indicator is used to indicate that the abnormal data exceeds a corresponding preset threshold and / or the abnormal data causes the business system to be unavailable.

6. The alert processing method of claim 5, wherein, The alarm based on the alarm mode and the alarm information comprises: If the alarm category is the normal alarm, sending the alarm information to a preset mailbox address corresponding to the abnormal data; If the alarm category is the critical alarm, sending the alarm information to a preset mailbox address corresponding to the abnormal data, and based on the contact information corresponding to the abnormal data, communicating with a terminal device corresponding to the contact information to push the alarm information to the terminal device.

7. The alert processing method of claim 6, wherein, If the alarm category is the common alarm, the alarm information is sent to a preset mailbox address corresponding to the abnormal data, comprising: If the alarm category is the common alarm, prediction information is determined based on the continuous abnormal change trend of the abnormal data, and the prediction information and the alarm information are sent to a preset mailbox address corresponding to the abnormal data.

8. An alert system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

9. A monitoring system, characterized by Comprise: An intelligent collector and the alarm system of claim 8; The intelligent collector is connected with the alarm system, and the intelligent collector is used for collecting index data of each business system and pre-processing abnormal index data in the index data to obtain abnormal data.

10. The monitoring system of claim 9, wherein, The intelligent collector comprises a data collection module, a collection confirmation module, a machine learning module, a cache confirmation module and a sending module; the intelligent collector is used for executing at least one of the following for each index of the index data collected by the data collection module: When the machine learning module detects that the index does not change, the index is stored in a distributed streaming database at a predetermined time interval through the sending module; When the collection confirmation module detects that the index is abnormal, the index is sent to the machine learning module as the abnormal data, so that the machine learning module sends the abnormal data to the cache confirmation module to record the abnormal data, and stores the abnormal data in the distributed streaming database through the machine learning module based on the record result of the cache confirmation module; And / or, the alarm system comprises a master node and at least two other nodes; the alarm system is used for sending the alarm information through the master node based on a preset arbitration mechanism; wherein the arbitration mechanism comprises sending the alarm information when more than half of the nodes in the master node and other nodes vote to confirm.

Images