Monitoring and early warning method, device and system, electronic equipment and storage medium

By pre-processing and risk assessment of monitoring and early warning data, combined with preset alarm strategies and prediction of monitoring and early warning models, the problems of early warning accuracy and timeliness in microservice environments in the existing technology are solved, and efficient and accurate monitoring is achieved and early warning reduces manual intervention and ensures business stability and continuity.

CN118939505APending Publication Date: 2024-11-12CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410989385.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing monitoring and early warning systems are difficult to provide high-accuracy early warnings in complex and changing microservice environments, and manual intervention leads to high time lag, which may lead to business interruptions and production accidents.

Method used

By obtaining the monitoring and early warning data of the target server for pre-processing, inputting the monitoring and early warning module for risk assessment and alarm screening, identifying alarm information using pre-set alarm strategies, and inputting the data to the monitoring and early warning model for alarm prediction of future status, generating predictive alarms information.

Benefits of technology

It improves the accuracy and timeliness of early warning in the case of sparse samples, reduces false alarms and missed reports, improves operation and maintenance efficiency, and can identify potential risks in a timely manner to avoid or reduce the impact of failures on the business.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939505A_ABST
    Figure CN118939505A_ABST
Patent Text Reader

Abstract

The invention provides a monitoring and early warning method, device and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining monitoring and early warning data of a target server, and carrying out the preprocessing of the monitoring and early warning data; inputting the preprocessed monitoring and early warning data into a monitoring and early warning module for risk assessment processing to obtain state assessment information of the target server; performing alarm screening processing on the state evaluation information by using a preset alarm strategy so as to identify alarm information of the target server; and inputting the monitoring and early warning data into a monitoring and early warning model, performing alarm prediction on the operation state of the target server in a future preset duration, and generating prediction alarm information of the target server. Compared with the prior art, according to the embodiment of the invention, the state evaluation information is screened by using the preset alarm strategy, so that the alarm information which really needs to be concerned can be identified more accurately, false alarms and missing alarms are reduced, and the operation and maintenance efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a monitoring and early warning method and device, system, electronic device and storage medium. Background Art

[0002] With the widespread application of microservice architecture in operation and maintenance platforms in recent years, the number of businesses it carries has increased dramatically, and it has become particularly important to monitor and warn the data reported from log collection or data center result tables. Monitoring and warning are the core links in the product life cycle of the operation and maintenance platform, because they can not only quickly reflect the current fault status of the system, but also provide key data support for problem troubleshooting and location, ensuring the stable and reliable operation of the service. However, in the current operation and maintenance platform practice, although the amount of data is growing at an alarming rate, the data samples that can be used for periodic monitoring to determine anomalies are relatively scarce. This leads to a key problem: most of the existing monitoring and warning systems rely on fixed alarm strategies and traditional anomaly detection algorithms (such as static threshold methods), which are difficult to provide high-accuracy warnings in complex and changeable microservice environments.

[0003] In addition, when the system detects an anomaly and triggers an early warning, the traditional approach is to notify monitoring personnel through text messages, emails, etc., and then manually intervene and handle it. This method of manual intervention not only has a high time lag, but may also result in problems not being handled in a timely manner, causing business interruptions and even serious production accidents. Therefore, how to improve the accuracy and timeliness of early warnings when samples are scarce, reduce the need for manual intervention, and thus ensure that the operation and maintenance platform can operate efficiently and stably has become an urgent problem to be solved. Summary of the invention

[0004] The present disclosure provides a monitoring and early warning method and device, system, electronic device and storage medium, the main purpose of which is to improve the accuracy and timeliness of early warning when samples are scarce.

[0005] According to a first aspect of the present disclosure, a method for monitoring early warning is provided, comprising:

[0006] Obtain monitoring and warning data of the target server, and pre-process the monitoring and warning data;

[0007] Input the pre-processed monitoring and warning data into the monitoring and warning module for risk assessment processing to obtain status assessment information of the target server;

[0008] Performing alarm screening processing on the status assessment information using a preset alarm strategy to identify alarm information of the target server;

[0009] The monitoring and warning data is input into a monitoring and warning model, an alarm prediction is performed on the operation status of the target server within a preset time period in the future, and prediction alarm information of the target server is generated.

[0010] In some embodiments, the acquiring of monitoring and warning data of the target server and preprocessing of the monitoring and warning data include:

[0011] Based on the monitoring collection plug-in, obtain the monitoring and early warning data of the target server recorded in the operation and maintenance server, wherein the monitoring and early warning data includes system monitoring data, service monitoring data, network monitoring data, and log monitoring data;

[0012] Parsing the monitoring and warning data, and performing standardization processing based on the parsing result, so as to convert the monitoring and warning data into a preset format;

[0013] The standardized monitoring and early warning data is subjected to time series processing to obtain time series processed monitoring and early warning data.

[0014] In some embodiments, the monitoring and warning module includes a real-time warning engine and an offline warning engine;

[0015] The pre-processed monitoring and warning data is input into the monitoring and warning module for risk assessment processing to obtain the status assessment information of the target server, including:

[0016] Based on the message flow middleware, the pre-processed monitoring warning data is input into the real-time warning engine for real-time risk assessment processing to obtain real-time assessment information;

[0017] Inputting the stored pre-processed monitoring warning data into the offline warning engine for offline risk assessment processing to obtain offline assessment information;

[0018] Based on the offline evaluation information and the real-time evaluation information, status evaluation information of the target server is generated.

[0019] In some embodiments, the using a preset alarm strategy to perform alarm screening processing on the status assessment information to identify the alarm information of the target server includes:

[0020] When the data indicators of each type of status assessment information exceed the first risk threshold and are less than twice the first risk threshold, determining that the alarm assessment information is low-risk alarm information;

[0021] When the data indicators of each type of status assessment information exceed twice the first risk threshold, the alarm assessment information is determined to be high-risk alarm information.

[0022] In some embodiments, after inputting the monitoring warning data into the monitoring warning model, performing an alarm prediction on the operating state of the target server within a preset time period in the future, and generating the predicted alarm information of the target server, the method further includes:

[0023] When the number of times the low-risk warning information is generated exceeds a second risk threshold, the first risk threshold is modified and configured to be updated;

[0024] When the number of times the high-risk warning information is generated exceeds a third risk threshold, the second risk threshold is modified and updated.

[0025] In some embodiments, before inputting the monitoring warning data into the monitoring warning model to perform an alarm prediction on the operation status of the target server within a preset time period in the future, the method further includes:

[0026] Based on the similarity algorithm, similar features between the source domain sample data and the target domain sample data are extracted to obtain auxiliary sample data;

[0027] Constructing source domain sample data for training according to the auxiliary sample data and the target domain sample data;

[0028] The training source domain sample data is input into the monitoring and early warning model for training to obtain a trained monitoring and early warning model.

[0029] According to a second aspect of the present disclosure, a monitoring and early warning device is provided, comprising:

[0030] A preprocessing unit, used to obtain monitoring and early warning data of the target server and preprocess the monitoring and early warning data;

[0031] An evaluation unit, used for inputting the pre-processed monitoring and warning data into the monitoring and warning module for risk evaluation processing to obtain status evaluation information of the target server;

[0032] A screening unit, configured to perform alarm screening processing on the status assessment information using a preset alarm strategy to identify the alarm information of the target server;

[0033] The prediction unit is used to input the monitoring warning data into the monitoring warning model, make an alarm prediction on the operation status of the target server within a preset time period in the future, and generate the prediction alarm information of the target server.

[0034] In some embodiments, the pre-processing unit comprises:

[0035] An acquisition module, used to acquire monitoring and early warning data of the target server recorded in the operation and maintenance server based on the monitoring and collection plug-in, wherein the monitoring and early warning data includes system monitoring data, service monitoring data, network monitoring data, and log monitoring data;

[0036] A parsing module, used to parse the monitoring and warning data, and perform standardization processing based on the parsing results, so as to convert the monitoring and warning data into a preset format;

[0037] The processing module is used to perform time-series processing on the monitoring and early warning data after the standardization processing to obtain the monitoring and early warning data after the time-series processing.

[0038] In some embodiments, the evaluation unit comprises:

[0039] A first evaluation module is used to input the pre-processed monitoring warning data into the real-time warning engine for real-time risk evaluation based on the message flow middleware to obtain real-time evaluation information;

[0040] A second evaluation module is used to input the stored pre-processed monitoring warning data into the offline warning engine for offline risk evaluation processing to obtain offline evaluation information;

[0041] A generating module is used to generate status evaluation information of the target server based on the offline evaluation information and the real-time evaluation information.

[0042] In some embodiments, the screening unit comprises:

[0043] A first determination module, configured to determine that the alarm assessment information is low-risk alarm information when the data indicators of each type of status assessment information exceed a first risk threshold and are less than twice the first risk threshold;

[0044] The second determination module is used to determine that the alarm assessment information is high-risk alarm information when the data indicators of each type of status assessment information exceed twice the first risk threshold.

[0045] In some embodiments, the apparatus further comprises:

[0046] A first correction unit is used for, after the prediction unit inputs the monitoring warning data into the monitoring warning model, performs an alarm prediction on the operation status of the target server within a preset time period in the future, and generates the predicted alarm information of the target server, when the number of times the low-risk alarm information is generated exceeds a second risk threshold, to correct the first risk threshold and configure an update;

[0047] The second correction unit is used to correct and configure an update to the second risk threshold when the number of times the high-risk warning information is generated exceeds a third risk threshold.

[0048] In some embodiments, the apparatus further comprises:

[0049] The extraction unit is used to extract similar features between the source domain sample data and the target domain sample data based on a similarity algorithm to obtain auxiliary sample data before the prediction unit inputs the monitoring warning data into the monitoring warning model to make an alarm prediction on the operation status of the target server within a preset time period in the future;

[0050] A construction unit, configured to construct source domain sample data for training according to the auxiliary sample data and the target domain sample data;

[0051] The training unit is used to input the training source domain sample data into the monitoring and early warning model for training to obtain a trained monitoring and early warning model.

[0052] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0053] at least one processor; and

[0054] a memory communicatively connected to the at least one processor; wherein,

[0055] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0056] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0057] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.

[0058] The present disclosure provides a monitoring and early warning method and device, system, electronic device and storage medium, which obtain the monitoring and early warning data of the target server, and pre-process the monitoring and early warning data; input the pre-processed monitoring and early warning data into the monitoring and early warning module for risk assessment processing to obtain the status assessment information of the target server; use the preset alarm strategy to perform alarm screening processing on the status assessment information to identify the alarm information of the target server; input the monitoring and early warning data into the monitoring and early warning model, and perform alarm prediction on the operation status of the target server within the future preset time period, and generate the predicted alarm information of the target server. Compared with the related art, the embodiment of the present disclosure can more accurately identify the alarm information that really needs attention by screening the status assessment information using the preset alarm strategy, reduce false alarms and missed alarms, and improve the operation and maintenance efficiency; through the analysis of real-time monitoring and early warning data, timely monitoring and early warning can be achieved, which is helpful to maintain the continuity and stability of the business, and potential risks can be identified before or at the early stage of the problem, so as to take measures in advance to avoid or reduce the impact of the failure on the business, and reduce the business interruption or service quality degradation caused by server failure.

[0059] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0061] Figure 1 A flowchart of a monitoring and early warning method provided by an embodiment of the present disclosure;

[0062] Figure 2 A flowchart of another monitoring and early warning method provided by an embodiment of the present disclosure;

[0063] Figure 3 A flowchart of a method for training a monitoring and early warning model provided in an embodiment of the present disclosure;

[0064] Figure 4 A schematic diagram of the structure of a monitoring and early warning device provided in an embodiment of the present disclosure;

[0065] Figure 5 A schematic diagram of the structure of another monitoring and early warning device provided in an embodiment of the present disclosure;

[0066] Figure 6 A schematic block diagram of an exemplary electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0067] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0068] The following describes the monitoring and early warning method and device, system, electronic device and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0069] Figure 1 A flowchart of a monitoring and early warning method provided in an embodiment of the present disclosure.

[0070] like Figure 1 As shown, the method comprises the following steps:

[0071] Step 101: Acquire monitoring and warning data of a target server, and pre-process the monitoring and warning data.

[0072] In the embodiments of the present disclosure, obtaining monitoring and early warning data of the target server generally involves using specialized monitoring tools or software, which can track various performance indicators of the server in real time, such as CPU usage, memory usage, disk I / O, network traffic, etc. Through these tools, detailed data about the server status can be collected, including data during normal operation and possible abnormal or warning information.

[0073] After obtaining these data, these monitoring and warning data need to be preprocessed. The purpose of preprocessing is to clean the data, remove noise, and convert the data into a format suitable for subsequent analysis or machine learning model processing. This may include filling missing values, smoothing data, removing duplicate records, converting data types (such as converting timestamps to a more readable date and time format), and possible data normalization or standardization to ensure that indicators of different magnitudes can be compared within the same analysis framework.

[0074] In some embodiments of the present disclosure, the preprocessing stage may also involve feature engineering, that is, creating new features or indicators based on the original data, which can better reflect the health of the server or predict potential failures. For example, the average response time or request failure rate within a specific time period may be calculated as a new indicator for evaluating server performance.

[0075] Step 102: input the pre-processed monitoring and warning data into a monitoring and warning module for risk assessment processing to obtain status assessment information of the target server.

[0076] In the embodiment of the present disclosure, the monitoring and early warning module is a software system that integrates multiple data analysis algorithms and risk assessment models, which can receive pre-processed monitoring and early warning data and use these algorithms and models to conduct in-depth analysis and mining of the data. In this embodiment, the monitoring and early warning module includes but is not limited to two major parts: a real-time early warning engine and an offline early warning engine.

[0077] Through preset rules, algorithms or machine learning models, data is quickly analyzed to identify any signs that may indicate abnormal server status or performance degradation. Through deeper data analysis and mining of historical monitoring data, possible long-term trends, cyclical changes or potential risk factors can be discovered.

[0078] The monitoring and early warning module can comprehensively evaluate the status of the target server, provide accurate risk assessment information, and help the operation and maintenance team respond to potential problems in a timely manner, thereby ensuring the stability and availability of the server.

[0079] Step 103: Perform alarm screening processing on the status assessment information using a preset alarm strategy to identify the alarm information of the target server.

[0080] In the embodiments of the present disclosure, by applying a series of predefined rules and conditions, the status assessment information is deeply analyzed and screened to identify the alarm information that may exist in the target server. The preset alarm strategies are usually formulated based on the experience of the operation and maintenance team, business requirements and server characteristics. They may include specific performance indicator thresholds, anomaly detection modes, historical fault pattern matching, etc. These strategies are encoded into the monitoring system and used to automatically screen and compare the real-time status assessment data.

[0081] During the alarm screening process, the system checks the indicators and data points in the status assessment information and matches them with the rules and conditions in the preset alarm strategy. If an indicator exceeds the preset threshold or meets a specific abnormal pattern, the system will generate corresponding alarm information. These alarm information may include the level of the alarm (such as urgent, important, general, etc.), the specific content of the alarm, the relevant performance indicators and data points, and the recommended response measures.

[0082] Through alarm screening and processing, the operation and maintenance team can quickly obtain instant feedback on the status of the target server and understand which aspects may have problems or risks. This helps them take appropriate measures in time to prevent potential problems from evolving into serious failures, thereby ensuring the stable operation of the server and business continuity.

[0083] Step 104: input the monitoring warning data into a monitoring warning model, make an alarm prediction on the operation status of the target server within a preset time period in the future, and generate prediction alarm information of the target server.

[0084] In an embodiment of the present disclosure, the monitoring and early warning model can be a prediction model built based on a machine learning or deep learning algorithm, which can receive monitoring and early warning data that has been preprocessed and feature engineered as input, and use the patterns and laws learned within the model to analyze and reason about the data. The monitoring and early warning model will perform time series analysis, trend prediction, anomaly detection, and other processing on the input data to identify features in the data that may indicate future changes in server status. It may consider periodic patterns, trend changes, and data features before known failures in historical data to predict the operating status of the target server within a preset period of time in the future.

[0085] Through alarm prediction processing, the monitoring warning model can generate predictive alarm information about the target server. This information usually includes the predicted time range, potential failure points or risk areas, predicted alarm levels and possibilities, and recommended preventive measures and response plans. Predictive alarm information is extremely valuable to the operation and maintenance team. It can help them understand the possible problems of the server in advance, so that they have enough time to prepare and take appropriate measures to prevent potential problems from evolving into serious failures.

[0086] The present disclosure provides a monitoring and early warning method, which obtains monitoring and early warning data of a target server, and pre-processes the monitoring and early warning data; inputs the pre-processed monitoring and early warning data into a monitoring and early warning module for risk assessment processing to obtain status assessment information of the target server; uses a preset alarm strategy to perform alarm screening processing on the status assessment information to identify the alarm information of the target server; inputs the monitoring and early warning data into a monitoring and early warning model, performs alarm prediction on the operating status of the target server within a preset time period in the future, and generates predicted alarm information of the target server. Compared with the related art, the embodiment of the present disclosure can more accurately identify the alarm information that really needs attention by screening the status assessment information using a preset alarm strategy, reduce false alarms and missed alarms, and improve operation and maintenance efficiency; through real-time monitoring and analysis of early warning data, timely monitoring and early warning can be achieved, which is helpful to maintain the continuity and stability of the business, and potential risks can be identified before or at the early stage of the problem, so as to take measures in advance to avoid or reduce the impact of the failure on the business, and reduce business interruption or service quality degradation caused by server failure.

[0087] In order to clearly illustrate the embodiment of the present disclosure, this embodiment provides a flowchart of another monitoring and early warning method.

[0088] like Figure 2 As shown, the method comprises the following steps:

[0089] Step 201, based on the monitoring collection plug-in, obtain the monitoring and early warning data of the target server recorded in the operation and maintenance server, wherein the monitoring and early warning data includes system monitoring data, service monitoring data, network monitoring data, and log monitoring data.

[0090] Specifically in step 201, the monitoring targets and process information of the monitoring warning data mainly rely on the CMDB (configuration management database) resource pool of the operation and maintenance platform. This resource pool only focuses on storing operation and maintenance related resource data and can be queried through a unified SQL query language, including: system monitoring data, such as CPU load, context switching, memory usage, disk read and write, disk usage, disk usage, etc.; service monitoring data, such as Zookeeper, Kafka, Nginx, Jvm, Kubernates, Redis, MySQL, RabbitMQ and other application operation information; network monitoring data, such as send / receive byte traffic, send / receive packet rate, number of listening / synchronous connections, etc.; log monitoring data, such as the application program generates application access logs, error logs, operation logs, network logs, etc.

[0091] Monitoring and early warning data can be collected in two ways: one is to deploy a set of monitoring and collection plug-ins to the target operation and maintenance server, and actively collect the corresponding physical machine, service process operation or related log information at regular intervals; the other is to customize the data in the result table in the reporting data.

[0092] Step 202: parse the monitoring warning data, and perform standardization processing based on the parsing result, so as to convert the monitoring warning data into a preset format.

[0093] Step 203, performing time series processing on the standardized monitoring and warning data to obtain time series processed monitoring and warning data.

[0094] Specifically, in steps 202 to 203, the data collected by the operation and maintenance monitoring and early warning platform will be put into the resource data center for unified management. It should be noted that the types and formats of monitoring and early warning data such as various types of services, logs or systems may be different, so they need to be parsed and standardized. Data parsing is to convert data of different types or formats into data that can be recognized by the data center, and standardization is to map and convert the parsed data and operation and maintenance indicators, and perform time series processing with time as the reference axis.

[0095] In some embodiments of the present disclosure, in order to facilitate monitoring personnel to understand in real time whether the data collection is successful, the data inspection module also embeds a set of WEB front-end / back-end services. Through the console panel, a graphical interface of the data or operation and maintenance indicators before and after processing of each collection task under the current host instance can be viewed.

[0096] Step 204: Based on the message flow middleware, the pre-processed monitoring warning data is input into the real-time warning engine for real-time risk assessment processing to obtain real-time assessment information.

[0097] Step 205: input the stored pre-processed monitoring warning data into the offline warning engine for offline risk assessment processing to obtain offline assessment information.

[0098] Step 206: Generate status evaluation information of the target server based on the offline evaluation information and the real-time evaluation information.

[0099] Specifically, in steps 204 to 206, the monitoring data features stored in the intermediate processing layer of the resource data center will enter the real-time message data flow pipeline (i.e., message flow middleware), such as message middleware Kafka, etc., and the stored terminal data indicators will enter the offline target data warehouse, such as Hive, etc., and receive the transmitted data information in real time to the distributed processing engine. It should be noted that there are two types of distributed processing engines used in this embodiment: Flink real-time warning task computing engine (real-time warning engine) and Spark offline warning task computing engine (offline warning engine). Both engines will extract and allocate data belonging to various types of warning tasks.

[0100] Step 207: When the data indicators of each type of status assessment information exceed the first risk threshold and are less than twice the first risk threshold, determine that the alarm assessment information is low-risk alarm information.

[0101] Step 208: When the data indicators of each type of status assessment information exceed twice the first risk threshold, determine that the alarm assessment information is high-risk alarm information.

[0102] Specifically, in steps 207 to 208, based on the experience of staff or experts, an initial abnormal risk threshold is pre-set, collectively referred to as the first risk threshold; a cumulative low warning number risk threshold is set, collectively referred to as the second risk threshold; and a cumulative high warning number risk threshold is set, collectively referred to as the third risk threshold. When the data indicators of various types of state assessment information are carefully analyzed, when these data indicators exceed the preset first risk threshold but do not reach twice the first risk threshold, the corresponding alarm assessment information is defined as low-risk alarm information. This definition is based on a prudent assessment of potential risks, which means that although there are certain unfavorable factors or abnormal conditions, these factors have not yet reached a level sufficient to cause serious consequences. Therefore, the corresponding response measures can focus more on prevention and early intervention to avoid further escalation of risks. However, when the data indicators of various types of state assessment information significantly exceed twice the first risk threshold, the situation becomes completely different. At this time, the alarm assessment information is determined to be high-risk alarm information, which indicates that the system or process is facing major risk challenges, and situations that may have or are about to seriously affect the normal operation of the system or business continuity may have occurred or are about to occur. For such high-risk alerts, decisive and effective measures need to be taken immediately to quickly control the risk, prevent the situation from further deteriorating, and ensure the safety of personnel and assets. Through such classification and definition, we can more accurately grasp the urgency and focus of response measures under different risk levels, thereby optimizing resource allocation and improving the efficiency and effectiveness of risk management.

[0103] Step 209: input the monitoring warning data into a monitoring warning model, make an alarm prediction on the operation status of the target server within a preset time period in the future, and generate the prediction alarm information of the target server.

[0104] Specifically, in step 209, the monitoring warning data is input into the initialized DTW-LSTM model (i.e., the monitoring warning model), a corresponding prediction value with high accuracy is output, and prediction warning information of the target server is generated. For the description of step 209, please continue to refer to the above embodiment, which will not be described in detail in this embodiment.

[0105] Furthermore, as an achievable manner of an embodiment of the present disclosure, after the monitoring and warning data is input into a monitoring and warning model, an alarm prediction is made on the operating status of the target server within a preset period of time in the future, and the predicted alarm information of the target server is generated, in order to enable the preset alarm strategy to adapt to the actual situation, it is necessary to adjust the risk threshold in the preset alarm strategy. The method also includes: when the number of times the low-risk alarm information is generated exceeds a second risk threshold, the first risk threshold is corrected and configured to be updated; when the number of times the high-risk alarm information is generated exceeds a third risk threshold, the second risk threshold is corrected and configured to be updated.

[0106] Specifically, the threshold standard is adaptively and continuously revised according to each warning situation, including when there is a data indicator in the monitoring data that is greater than the first risk threshold and less than 2 times the first risk threshold, the cumulative number is recorded. When it exceeds the second risk threshold, the first risk threshold is automatically revised to 1.5 times the first risk threshold; when there is a data indicator in the monitoring data that is directly greater than 2 times the first risk threshold, it will be judged as an abnormal special point. When the accumulated abnormal special points reach the third risk threshold, the second risk threshold will be automatically revised to be equal to (second risk threshold + third risk threshold) / 2; secondly, the policy configuration module also needs to select the monitoring period. The monitoring period defaults to 3 minutes, and it is recommended to be greater than or equal to the collection period; finally, the monitoring granularity can also be selected, such as a data dimension of a specific host instance.

[0107] Before using the monitoring warning model for alarm prediction, the monitoring warning model needs to be trained. Considering that the data samples that can meet the training requirements are relatively scarce, this embodiment adopts the transfer learning method to train the monitoring warning model. Figure 3 As shown, the method comprises the following steps:

[0108] Step 301: Based on a similarity algorithm, similar features between source domain sample data and target domain sample data are extracted to obtain auxiliary sample data.

[0109] Step 302: construct source domain sample data for training based on the auxiliary sample data and the target domain sample data.

[0110] Step 303: input the training source domain sample data into the monitoring and early warning model for training to obtain a trained monitoring and early warning model.

[0111] Specifically, in steps 301 to 303, since the data samples that can meet the training requirements are relatively scarce, this embodiment adopts transfer learning to train the monitoring and early warning model; its core is to mine similar features between the source domain and the target domain to ensure the successful transfer of knowledge relationships. However, in reality, when the relationship between the auxiliary training samples that constitute the source domain and the original small samples is weak, the source domain will produce "negative transfer" to the network prediction model. The key to suppressing the negative transfer problem lies in the reasonable selection of a similarity algorithm.

[0112] In this embodiment, dynamic time warping (DTW) is used to measure the similarity features between the source domain and the target domain. Similarity template matching is performed on the time series data by bending and aligning the time axis. The algorithm execution process is as follows:

[0113] Specifically, assuming that X = {x 1 ,x 2 …,xm} and Y={y 1 ,y 2 …,y n} respectively represent the target domain sample data (i.e., small sample data) and the source domain sample data and other data sources, where m and n are the lengths of their sequences m<<n. These two time series can be combined into an n×n order distance matrix: D m×n , in this distance matrix, define any two points x i and j The distance between them is calculated as d ij =D(x i ,y j ).

[0114] The optimal curved path of the DTW algorithm can be expressed as W = {w 1 ,w 2 …,w k}, represented by D m×n The optimal curved path must satisfy the following three conditions: (a) max(m,n)≤k≤m+n-1; (b) boundary constraints: w 1 =d 11 and w k =d mn ; (c) Two adjacent elements in W are in D m×n The matrix is ​​also adjacent. For example, if w k = {a, b} and w k+1 ={a′,b′}, then it must satisfy 0≤a′-a≤1 and 0≤b′-b≤1.

[0115] Start calculating the DTW distance between the i-th point and the j-th point in DTW: D DTW (i,j). The DTW path can be calculated by dynamic programming as follows:

[0116]

[0117] Note that the D at the end DTW (m,n) is the DTW optimal similarity path value of these two time series.

[0118] This embodiment uses the DTW algorithm template to match auxiliary sample data. The essence of the DTW algorithm is to freeze the small sample data and the data from different data sources in a two-dimensional plane in a time sequence, and use the nonlinear dynamic programming idea to simulate the auxiliary data with the highest similarity. In particular, the feature space distribution of the auxiliary training data selected by the DTW algorithm is more in line with the actual law, and can be well applied in predicting sample data.

[0119] The DTW algorithm is used to measure the template matching degree between small sample data and other monitoring and warning data sources, and the most similar data sources are selected as auxiliary data samples. The small sample data and auxiliary data samples are organically integrated and the source domain sample data to be trained is constructed. Finally, the sample data to be predicted in the prediction task is determined.

[0120] The LSTM neural network model has good medium- and long-term dependencies in learning time series data. Therefore, this paper selects LSTM as a monitoring and early warning model, and uses the obtained training source domain sample data as the input of LSTM for training. The training process mainly inputs the source domain training sample data into the LSTM network, uses 10 layers of cross entropy as the error convergence indicator to initialize the parameters, and finally repeatedly trains the prediction model until convergence.

[0121] This embodiment also defines the feature input and feature output of the training data. Specifically, the data source arranged in time sequence is converted into data that conforms to the model input and output in advance. The main execution process is to use the dynamic window algorithm to divide the training data into In the form of, t represents the characteristic time dimension, Represents the feature space dimension. It can be seen that the former represents the feature input, and the latter represents the feature output. It will also have an important impact on the feature extraction of the entire training process. In the embodiment of the present invention, the prediction time step is determined to be 1, and the feature space dimension is determined to be 6. Finally, 10 layers of cross entropy are selected as the error convergence indicator, and the initialization parameters are determined by grid traversal method and expert experience, which are now specifically explained by examples:

[0122] Assume that the auxiliary training sample data of this network prediction model is (i=1,2,3,…,M), the small sample training sample data is (j=1,2,3,…,N), its initial weight can be set as follows:

[0123]

[0124] The offset initialization formula is as follows:

[0125] B i =random(N,M), where random(*) is a random number in the interval.

[0126] The calculation formula of the ten-layer cross entropy error function δ between the actual output and the expected output of the output layer of the LSTM network prediction model is:

[0127]

[0128] The error result of each training is used to update the weights, and the result of each weight iterative adjustment is as follows:

[0129]

[0130] Where θ is the learning rate, Represents the gradient of the error with respect to the weights between neurons.

[0131] It should be noted that the embodiments of the present disclosure may include multiple steps. For the convenience of description, these steps are numbered, but these numbers do not limit the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.

[0132] Corresponding to the above-mentioned monitoring and early warning method, the present invention also provides a monitoring and early warning device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.

[0133] Figure 4 A schematic diagram of a monitoring and early warning device provided in an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, including:

[0134] The preprocessing unit 41 is used to obtain the monitoring and warning data of the target server and preprocess the monitoring and warning data;

[0135] An evaluation unit 42 is used to input the pre-processed monitoring and warning data into the monitoring and warning module for risk evaluation processing to obtain status evaluation information of the target server;

[0136] A screening unit 43, configured to perform alarm screening processing on the status assessment information using a preset alarm strategy to identify the alarm information of the target server;

[0137] The prediction unit 44 is used to input the monitoring warning data into the monitoring warning model, make an alarm prediction on the operation status of the target server within a preset time period in the future, and generate the prediction alarm information of the target server.

[0138] The present disclosure provides a monitoring and early warning device, which obtains monitoring and early warning data of a target server and pre-processes the monitoring and early warning data; inputs the pre-processed monitoring and early warning data into a monitoring and early warning module for risk assessment processing to obtain status assessment information of the target server; uses a preset alarm strategy to perform alarm screening processing on the status assessment information to identify the alarm information of the target server; inputs the monitoring and early warning data into a monitoring and early warning model, performs alarm prediction on the operating status of the target server within a preset time period in the future, and generates predicted alarm information of the target server. Compared with the related art, the embodiment of the present disclosure can more accurately identify the alarm information that really needs attention by screening the status assessment information using a preset alarm strategy, reduce false alarms and missed alarms, and improve operation and maintenance efficiency; through real-time monitoring and analysis of early warning data, timely monitoring and early warning can be achieved to help maintain the continuity and stability of the business, and potential risks can be identified before or at the early stage of the problem, so as to take measures in advance to avoid or reduce the impact of the failure on the business, and reduce business interruption or service quality degradation caused by server failure.

[0139] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the pre-processing unit 41 includes:

[0140] The acquisition module 411 is used to acquire the monitoring and early warning data of the target server recorded in the operation and maintenance server based on the monitoring and collection plug-in, wherein the monitoring and early warning data includes system monitoring data, service monitoring data, network monitoring data, and log monitoring data;

[0141] The parsing module 412 is used to parse the monitoring and warning data and perform standardization processing based on the parsing results so as to convert the monitoring and warning data into a preset format;

[0142] The processing module 413 is used to perform time series processing on the standardized monitoring and warning data to obtain time series processed monitoring and warning data.

[0143] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the evaluation unit 42 includes:

[0144] The first evaluation module 421 is used to input the pre-processed monitoring warning data into the real-time warning engine for real-time risk evaluation based on the message flow middleware to obtain real-time evaluation information;

[0145] The second evaluation module 422 is used to input the stored pre-processed monitoring warning data into the offline warning engine for offline risk evaluation processing to obtain offline evaluation information;

[0146] The generating module 423 is used to generate the status evaluation information of the target server based on the offline evaluation information and the real-time evaluation information.

[0147] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the screening unit 43 includes:

[0148] A first determination module 431 is used to determine that the alarm assessment information is low-risk alarm information when the data indicators of each type of status assessment information exceed the first risk threshold and are less than twice the first risk threshold;

[0149] The second determination module 432 is used to determine that the alarm assessment information is high-risk alarm information when the data indicators of each type of the status assessment information exceed twice the first risk threshold.

[0150] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the device also includes:

[0151] The first correction unit 45 is used for, after the prediction unit inputs the monitoring warning data into the monitoring warning model, performs an alarm prediction on the operation status of the target server within a preset time in the future, and generates the predicted alarm information of the target server, when the number of times the low-risk alarm information is generated exceeds the second risk threshold, to correct the first risk threshold and configure an update;

[0152] The second correction unit 46 is used to correct and update the second risk threshold when the number of times the high-risk warning information is generated exceeds a third risk threshold.

[0153] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the device also includes:

[0154] The extraction unit 47 is used to extract similar features between the source domain sample data and the target domain sample data based on a similarity algorithm to obtain auxiliary sample data before the prediction unit inputs the monitoring warning data into the monitoring warning model to make an alarm prediction on the operation status of the target server within a preset time period in the future;

[0155] A construction unit 48, configured to construct source domain sample data for training according to the auxiliary sample data and the target domain sample data;

[0156] The training unit 49 is used to input the training source domain sample data into the monitoring and early warning model for training to obtain a trained monitoring and early warning model.

[0157] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and the principle is the same, which is not limited in this embodiment.

[0158] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0159] Figure 6 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0160] like Figure 6 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 502 or a computer program loaded from a storage unit 508 to a RAM (Random Access Memory) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An I / O (Input / Output) interface 505 is also connected to the bus 504.

[0161] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0162] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as a method for monitoring early warning. For example, in some embodiments, the method for monitoring early warning may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the aforementioned monitoring and early warning method in any other appropriate manner (for example, by means of firmware).

[0163] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0165] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory) or a flash memory, an optical fiber, a CD-ROM (Compact Dis sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0167] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0168] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0169] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0170] The various numerical numbers such as first and second involved in the present disclosure are only for the convenience of description and are not used to limit the scope of the embodiments of the present disclosure, but also indicate the order of precedence.

[0171] At least one in the present disclosure may also be described as one or more, and a plurality may be two, three, four or more, which is not limited in the present disclosure. In the embodiments of the present disclosure, for a technical feature, the technical features in the technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", etc., and there is no order of precedence or size between the technical features described by the "first", "second", "third", "A", "B", "C" and "D".

[0172] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0173] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A monitoring and early warning method, It is characterized in that include: Obtain monitoring and warning data of the target server, and pre-process the monitoring and warning data; Input the pre-processed monitoring and warning data into the monitoring and warning module for risk assessment processing to obtain status assessment information of the target server; Performing alarm screening processing on the status assessment information using a preset alarm strategy to identify alarm information of the target server; The monitoring and warning data is input into a monitoring and warning model, an alarm prediction is performed on the operation status of the target server within a preset time period in the future, and prediction alarm information of the target server is generated.

2. The method according to claim 1, It is characterized in that The acquiring of monitoring and early warning data of the target server and preprocessing of the monitoring and early warning data include: Based on the monitoring collection plug-in, obtain the monitoring and early warning data of the target server recorded in the operation and maintenance server, wherein the monitoring and early warning data includes system monitoring data, service monitoring data, network monitoring data, and log monitoring data; Parsing the monitoring and warning data, and performing standardization processing based on the parsing result, so as to convert the monitoring and warning data into a preset format; The standardized monitoring and early warning data is subjected to time series processing to obtain time series processed monitoring and early warning data.

3. The method according to claim 1, It is characterized in that The monitoring and warning module includes a real-time warning engine and an offline warning engine; The pre-processed monitoring and warning data is input into the monitoring and warning module for risk assessment processing to obtain the status assessment information of the target server, including: Based on the message flow middleware, the pre-processed monitoring warning data is input into the real-time warning engine for real-time risk assessment processing to obtain real-time assessment information; Inputting the stored pre-processed monitoring warning data into the offline warning engine for offline risk assessment processing to obtain offline assessment information; Based on the offline evaluation information and the real-time evaluation information, status evaluation information of the target server is generated.

4. The method according to claim 1, It is characterized in that The using a preset alarm strategy to perform alarm screening processing on the status assessment information to identify the alarm information of the target server includes: When the data indicators of each type of status assessment information exceed the first risk threshold and are less than twice the first risk threshold, determining that the alarm assessment information is low-risk alarm information; When the data indicators of each type of status assessment information exceed twice the first risk threshold, the alarm assessment information is determined to be high-risk alarm information.

5. The method according to claim 4, It is characterized in that After inputting the monitoring warning data into the monitoring warning model, performing an alarm prediction on the operation status of the target server within a preset time period in the future, and generating the predicted alarm information of the target server, the method further includes: When the number of times the low-risk warning information is generated exceeds a second risk threshold, the first risk threshold is modified and configured to be updated; When the number of times the high-risk warning information is generated exceeds a third risk threshold, the second risk threshold is modified and updated.

6. The method according to claim 1, It is characterized in that Before inputting the monitoring warning data into the monitoring warning model to perform an alarm prediction on the operation status of the target server within a preset time period in the future, the method further includes: Based on the similarity algorithm, similar features between the source domain sample data and the target domain sample data are extracted to obtain auxiliary sample data; Constructing source domain sample data for training according to the auxiliary sample data and the target domain sample data; The training source domain sample data is input into the monitoring and early warning model for training to obtain a trained monitoring and early warning model.

7. A monitoring and early warning device, It is characterized in that include: Obtain monitoring and warning data of the target server, and pre-process the monitoring and warning data; Input the pre-processed monitoring and warning data into the monitoring and warning module for risk assessment processing to obtain status assessment information of the target server; Performing alarm screening processing on the status assessment information using a preset alarm strategy to identify alarm information of the target server; The monitoring and warning data is input into a monitoring and warning model, an alarm prediction is performed on the operation status of the target server within a preset time period in the future, and prediction alarm information of the target server is generated.

8. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, It is characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, It is characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Citation Information

Cited By

  • Self-adaptive risk assessment adjustment method and system based on penetration supervision

    CN120198213A

  • Risk prevention and control platform for underground comprehensive pipe gallery

    CN120410186A

  • Risk prevention and control platform for underground utility tunnel

    CN120410186B