Data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. A specific embodiment of the method comprises the following steps: acquiring log data associated with each monitoring index, and generating monitoring data of each monitoring index; extracting an index set belonging to the first type, screening monitoring indexes belonging to a target level, determining an abnormal index, and obtaining index monitoring associated with the abnormal index to identify the associated abnormal index; extracting an index set belonging to the second type, detecting the monitoring data of the index set, and determining a corresponding abnormal index; extracting prediction index data of the to-be-predicted index, determining a predicted abnormal index, and completing training of the prediction model based on a plurality of preset data selection strategies; and generating and sending early warning information corresponding to each abnormal index. According to the embodiment of the invention, the problem of false alarm and missing alarm caused by consumption of a large amount of time cost when the value of the monitoring index is compared with the threshold value to determine whether the abnormity occurs can be solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of informatization in various industries, the information systems of major enterprises are becoming increasingly large and complex. In order to ensure the normal operation of the system, the operation and maintenance personnel of the enterprise will monitor the system through the corresponding KPI (Key Performance Indicator). When certain KPIs are abnormal, alarms can be issued in time. In related technologies, system monitoring can be used, such as manually setting rules or thresholds, to determine whether an abnormality occurs by comparing the value of the monitoring indicator with the threshold. However, this method is relatively simple. For medium and large business systems and complex business environments, the indicators that need to be monitored are complex, which not only consumes a lot of time costs, but also easily leads to false alarms and missed alarms. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a data processing method, device, electronic device and storage medium, which can solve the problem of determining whether an abnormality occurs by comparing the value of a monitoring indicator with a threshold, which consumes a lot of time and leads to false alarms and missed alarms in medium and large business systems and complex business environments.

[0004] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for data processing is provided.

[0005] A data processing method of an embodiment of the present invention includes: calling a preset indicator link to obtain log data associated with each monitoring indicator, aggregating the log data, generating monitoring data for each monitoring indicator, and obtaining the detection type of each monitoring indicator; extracting an indicator set whose detection type belongs to a first type, screening monitoring indicators belonging to a target level, obtaining corresponding detection parameters, calling the detection strategy corresponding to the first type to determine abnormal indicators, and obtaining indicator monitoring associated with the abnormal indicators from the indicator set to identify the associated abnormal indicators; extracting an indicator set whose detection type belongs to a second type, obtaining corresponding monitoring data, calling a corresponding detection model to detect the monitoring data of the indicator set, and determining the corresponding abnormal indicators; extracting prediction indicator data of the indicator to be predicted from the indicator data, inputting a preset prediction model and obtaining a prediction result, so as to determine the predicted abnormal indicator according to the prediction result, wherein the prediction model is trained based on multiple preset data selection strategies; and generating and sending warning information corresponding to each abnormal indicator.

[0006] In one embodiment, the monitoring indicators associated with the abnormal indicators are obtained from the indicator set to identify the associated abnormal indicators, including: for each of the abnormal indicators, querying the corresponding indicator list to match with the indicator set, determining the monitoring indicator sequence associated with the abnormal indicator, and obtaining the monitoring data corresponding to the monitoring indicator sequence; calculating the abnormal trend of the abnormal indicator according to the monitoring data of the abnormal indicator and the detection parameters, so as to identify the associated abnormal indicator in combination with the monitoring data corresponding to the monitoring indicator sequence.

[0007] In another embodiment, the identifying of associated abnormal indicators in combination with the monitoring data corresponding to the monitoring indicator sequence includes: obtaining the degree of influence of each monitoring indicator in the monitoring indicator sequence on the abnormal indicator, determining the indicator to be identified in combination with the abnormal trend, and identifying the associated abnormal indicators based on the indicator data of the indicator to be identified.

[0008] In another embodiment, before the preset prediction model is input and the prediction result is obtained, it also includes: obtaining sample data of the indicator to be predicted to input it into the pre-established prediction model to obtain the pending prediction result; screening corresponding candidate data from the pending prediction result according to multiple data selection strategies, obtaining model parameters corresponding to each data selection strategy to update the sample data, so as to input it into the prediction model again, and repeating the above-mentioned update operation until a trained prediction model is obtained.

[0009] In another embodiment, corresponding candidate data are screened from the pending prediction results according to a plurality of the data selection strategies, and model parameters corresponding to each of the data selection strategies are obtained to update the sample data, including: screening the first candidate data from the pending prediction results according to a preset greedy selection strategy, and screening the second candidate data from the pending prediction results according to a preset dynamic programming selection strategy; obtaining the model parameters corresponding to the greedy selection strategy and the dynamic programming selection strategy, respectively, to calculate the selection probability of each candidate data, and determining the target data from the first candidate data and the second candidate data according to the selection probability, to update them to the sample data.

[0010] In another embodiment, the log data is aggregated to generate monitoring data for each of the monitoring indicators, including: parsing the log data, identifying corresponding data and event information based on the identifier of each of the monitoring indicators, so as to generate a data series corresponding to the time of each of the monitoring indicators; obtaining the business logic of each of the monitoring indicators, so as to calculate the monitoring data of each of the monitoring indicators based on the corresponding data series and the business logic.

[0011] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data processing device is provided.

[0012] A data processing device according to an embodiment of the present invention includes: a generating unit, which is used to call a preset indicator link to obtain log data associated with each monitoring indicator, aggregate the log data, generate monitoring data of each monitoring indicator, and obtain the detection type of each monitoring indicator; an identifying unit, which is used to extract an indicator set whose detection type belongs to a first type, screen monitoring indicators belonging to a target level, obtain corresponding detection parameters, call the detection strategy corresponding to the first type to determine abnormal indicators, obtain indicator monitoring associated with the abnormal indicators from the indicator set, and identify the associated abnormal indicators; a determining unit, which is used to extract an indicator set whose detection type belongs to a second type, obtain corresponding monitoring data, call the corresponding detection model to detect the monitoring data of the indicator set, and determine the corresponding abnormal indicators; the determining unit is also used to extract prediction indicator data of the indicator to be predicted from the indicator data, input the preset prediction model and obtain the prediction result, so as to determine the predicted abnormal indicator according to the prediction result, wherein the prediction model is trained based on multiple preset data selection strategies; a sending unit, which is used to generate and send warning information corresponding to each abnormal indicator.

[0013] In one embodiment, the identification unit is specifically used to: for each of the abnormal indicators, query the corresponding indicator list to match with the indicator set, determine the monitoring indicator sequence associated with the abnormal indicator, and obtain the monitoring data corresponding to the monitoring indicator sequence; calculate the abnormal trend of the abnormal indicator based on the monitoring data of the abnormal indicator and the detection parameters, so as to identify the associated abnormal indicator in combination with the monitoring data corresponding to the monitoring indicator sequence.

[0014] In another embodiment, the identification unit is specifically used to: obtain the degree of influence of each monitoring indicator in the monitoring indicator sequence on the abnormal indicator, determine the indicator to be identified in combination with the abnormal trend, and identify the associated abnormal indicator based on the indicator data of the indicator to be identified.

[0015] In another embodiment, the device also includes: an input unit, used to obtain sample data of the indicator to be predicted, so as to input it into a pre-established prediction model to obtain a pending prediction result; a training unit, used to screen corresponding candidate data from the pending prediction result according to a plurality of the data selection strategies, obtain the model parameters corresponding to each of the data selection strategies to update the sample data, so as to input it into the prediction model again, and repeat the above-mentioned update operation until a trained prediction model is obtained.

[0016] In another embodiment, the training unit is specifically used to: screen the first candidate data from the pending prediction results according to a preset greedy selection strategy, and screen the second candidate data from the pending prediction results according to a preset dynamic programming selection strategy; obtain the model parameters corresponding to the greedy selection strategy and the dynamic programming selection strategy, respectively, to calculate the selection probability of each candidate data, and determine the target data from the first candidate data and the second candidate data according to the selection probability to update it to the sample data.

[0017] In another embodiment, the generating unit is specifically used to: parse the log data, identify the corresponding data and event information according to the identifier of each monitoring indicator, so as to generate a data sequence corresponding to the time of each monitoring indicator; obtain the business logic of each monitoring indicator, so as to calculate the monitoring data of each monitoring indicator according to the corresponding data sequence and the business logic.

[0018] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device is provided.

[0019] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided by the embodiment of the present invention.

[0020] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0021] A computer-readable medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, the data processing method provided by the embodiment of the present invention is implemented.

[0022] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer program product is provided.

[0023] A computer program product according to an embodiment of the present invention includes a computer program, and when the program is executed by a processor, the data processing method provided by the embodiment of the present invention is implemented.

[0024] An embodiment of the above invention has the following advantages or beneficial effects: in the embodiment of the present invention, the log data associated with each monitoring indicator is obtained through a preset indicator link to generate monitoring data of each monitoring indicator and obtain the detection type of each monitoring indicator; for monitoring indicators of different detection types, abnormal indicators can be determined in different ways respectively, so as to determine the abnormal indicators in an adaptive way and improve the accuracy of abnormal identification; for the first type, abnormal indicator identification can be performed hierarchically, and for the second type, abnormal identification can also be performed through a preset detection model, so as to meet the needs of complex business and reduce the consumption of time cost; and abnormal prediction can also be performed on the data prediction of the prediction indicator through a prediction model, and the prediction model can be trained based on multiple preset data selection strategies to improve the accuracy and comprehensiveness of abnormal indicator identification. In this way, in the embodiment of the present invention, abnormal identification and abnormal prediction can be performed in an adaptive way according to the generated indicator data, which can not only meet the complex business environment and complex monitoring needs, but also improve the accuracy of abnormal identification.

[0025] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention. Figure 1 is a schematic diagram of a main flow of a data processing method according to an embodiment of the present invention; Figure 2 is a schematic diagram of another main flow of a data processing method according to an embodiment of the present invention; Figure 3 is a schematic diagram of another main flow of a data processing method according to an embodiment of the present invention; Figure 4 is a schematic diagram of main units of a data processing apparatus according to an embodiment of the present invention; Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied; Figure 6 It is a schematic diagram of the structure of a computer system suitable for implementing the embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] It should be pointed out that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict. The acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0029] An embodiment of the present invention provides a data processing system, which can be used in a scenario of data anomaly monitoring, and specifically can be used in a scenario of monitoring abnormal indicators of various business systems.

[0030] The embodiment of the present invention provides a data processing method, which can be executed by a data processing system. Figure 1 As shown, the method includes the following steps.

[0031] S101: calling a preset indicator link to obtain log data associated with each monitoring indicator, aggregating the log data, generating monitoring data of each monitoring indicator, and obtaining a detection type of each monitoring indicator.

[0032] Among them, the indicator link is pre-established and can be used to collect log data in a unified and centralized manner. For example, log data can be stored in Kafka, then parsed and aggregated through real-time processing, and then the obtained data can be dropped into Kafka as the data source of log data to meet different processing requirements.

[0033] Monitoring indicators can be pre-set and can be classified according to needs. For example, they can include system indicators, business indicators, etc. System indicators represent indicators for monitoring system anomalies, such as query per second (QPS) (a common measure of the search traffic received by the information retrieval system within one second), transaction per second (TPS) (the time taken by the client to send a request to the server and the number of completed transactions), response time RT (the total time taken to execute a request from the beginning to the end of receiving the response data), concurrency (the number of requests that the system can handle at the same time, reflecting the load capacity of the system), throughput, job availability (compared to the continuous running time of the job, the number of job restarts), job traffic, CPU, memory, garbage collection and network-related indicators. Business indicators can represent indicators for monitoring anomalies in business processing, which can be set according to the business processing logic.

[0034] In the embodiment of the present invention, users can also set personalized monitoring indicators according to their needs, and can also place points in the job logic to monitor the abnormality of indicators to meet the personalized monitoring needs of users. For example, the monitoring indicators can be related to the business status, such as whether the business job fails, whether the business job survives, whether the business job runs stably, and the risk factors that affect the feasibility of the job; they can also be related to the business job performance, such as the processing delay of the job, data skew, performance bottleneck, etc.; they can also be related to the business logic, such as the upstream data quality, whether there are problems with the new logic, whether there is data loss, etc.

[0035] It should be noted that, in an embodiment of the present invention, log data can also be stored in a database, such as ES, and monitoring indicators can also be stored in a public database, etc., for query use. At the same time, data processing will also be performed based on keywords, such as log-related alarms; at the same time, these data can also be concentrated in a real-time computing platform and displayed to users.

[0036] In the embodiment of the present invention, since each log data does not necessarily directly include the indicator data corresponding to the monitoring indicator, it is necessary to parse, aggregate, and process the log data to obtain the indicator data of each monitoring indicator.

[0037] Specifically, it can be performed as follows: parsing the log data, identifying the corresponding data and time information according to the identification of each monitoring indicator, so as to generate the data sequence corresponding to the time of each monitoring indicator; obtaining the business logic of each monitoring indicator, so as to calculate the monitoring data of each monitoring indicator according to the corresponding data sequence and business logic.

[0038] The log usually records all the data involved in business processing and system operation, so in this step, the log data can be parsed first to identify the corresponding data and time information according to the identification of each monitoring indicator. If the data of a certain monitoring indicator is not included in the log data, the indicator parameters required to generate the data of the monitoring indicator can be determined first to determine it as a monitoring indicator and fuse the monitoring data from the log. For each monitoring indicator, the monitoring data can be generated into a data sequence according to time, so as to fuse the data according to the pre-set business logic to obtain the monitoring data of each monitoring indicator.

[0039] It should be noted that the indicator data of monitoring indicators in different scenarios can be set up with corresponding fusion methods according to the needs. For example, log data can be fused from different angles, granularities, application orientations, etc. to obtain indicator data corresponding to different monitoring needs. The fusion method can be unlimited, such as sum, mean, maximum value, minimum value, rate of change, etc. In data fusion, in order to eliminate statistical errors, the data can be averaged over a moving average or a fixed time to make the data change smoother.

[0040] Due to factors such as the scene requirements and indicator complexity of different abnormal monitoring, different abnormal detection methods can be set for different monitoring indicators, so that adaptive detection methods can be used to improve the accuracy of abnormal detection. Therefore, in an embodiment of the present invention, a corresponding detection type can be set for each monitoring indicator to indicate the abnormal detection method of the monitoring indicator. Specifically, in an embodiment of the present invention, the detection type may include a first type and a second type, the first type may represent a monitoring indicator for abnormal identification through a detection strategy, and the second type may represent a monitoring indicator for abnormal identification through a detection model.

[0041] S102: Extract an indicator set whose detection type belongs to the first type, filter monitoring indicators belonging to the target level, obtain corresponding detection parameters, call the detection strategy corresponding to the first type to determine abnormal indicators, obtain indicator monitoring associated with the abnormal indicators from the indicator set to identify associated abnormal indicators.

[0042] Among them, for the first type of monitoring indicators, corresponding detection strategies and personalized detection parameters can be set. When detecting anomalies, the inspection parameters and detection strategies can be combined to generate rules for detecting anomalies of monitoring indicators, and then perform anomaly detection in combination with the corresponding monitoring data. In the embodiment of the present invention, since some monitoring indicators are obtained by fusion of data of other indicators, each monitoring indicator can be divided into levels in this step, and anomaly identification is first performed on the high-level monitoring indicators; if the high-level monitoring indicators do not identify anomalies, the corresponding low-level monitoring indicators can be exempted from anomaly identification; if the high-level monitoring indicators identify anomalies, the corresponding low-level monitoring indicators can be anomaly identified to improve the efficiency of anomaly identification. In this step, the target level can be set according to demand, and can be the highest-level monitoring indicator or multiple high-level levels. For example, when the number of monitoring indicators at the highest level is small, the two highest levels can be determined as the target level.

[0043] After identifying the monitoring indicators of the target layer, the indicator monitoring associated with the abnormal indicator can be obtained from the indicator set to identify the associated abnormal indicator. Specifically, this step can be performed as follows: for each abnormal indicator, query the corresponding indicator list to match with the indicator set, determine the monitoring indicator sequence associated with the abnormal indicator, and obtain the monitoring data corresponding to the monitoring indicator sequence; calculate the abnormal trend of the abnormal indicator based on the monitoring data and detection parameters of the abnormal indicator, so as to identify the associated abnormal indicator in combination with the monitoring data corresponding to the monitoring indicator sequence.

[0044] The indicator list can represent the list of low-level monitoring indicators corresponding to the abnormal indicators, so by matching it with the indicator set, the monitoring indicator sequence that needs to be monitored in the indicator list can be determined. Since the degree of influence of different low-level monitoring indicators on high-level monitoring indicators is not the same, the abnormal trend of the abnormal indicator can be determined first in this step, such as setting the abnormal level. For low-level abnormal monitoring indicators, it is usually caused by indicators that have a greater degree of influence on it, so the abnormality detection can be performed directly on the indicator with a greater degree of influence; for high-level abnormal monitoring indicators, it may be caused by all indicators that have an impact on it, so all low-level monitoring indicators may need to be detected for abnormalities. Specifically, it can be performed as follows: obtain the degree of influence of each monitoring indicator in the monitoring indicator sequence on the abnormal indicator, determine the indicator to be identified in combination with the abnormal trend, and identify the associated abnormal indicator based on the indicator data of the indicator to be identified.

[0045] S103: extracting an indicator set whose detection type belongs to the second type, obtaining corresponding monitoring data, calling a corresponding detection model to detect the monitoring data of the indicator set, and determining corresponding abnormal indicators.

[0046] The detection model may be pre-trained, and may be a variety of models, such as a machine learning model, and may be LSTM (Long-Short Term Memory, a type of feedback neural network), which may be used for anomaly detection of multiple monitoring indicators. In this step, the monitoring data of the associated monitoring indicators may generate a feature vector and input it into the trained detection model to determine the corresponding anomaly indicator.

[0047] S104: extracting prediction indicator data of the indicator to be predicted from the indicator data, inputting a preset prediction model and obtaining a prediction result, so as to determine the predicted abnormal indicator according to the prediction result.

[0048] Among them, the prediction model is trained based on multiple preset data selection strategies. In order to timely identify anomalies in indicators, anomaly prediction can also be performed on important indicators. The indicators to be predicted are pre-configured, and the prediction indicator data is usually derived from the indicator data to improve the accuracy of the prediction results.

[0049] The prediction model can be set according to needs. For example, the prediction model can be implemented through reinforcement learning. During the prediction model training process, the selection of training data for each round can be performed based on multiple preset data selection strategies.

[0050] The reinforcement learning algorithm can continuously interact with the current indicator environment. Each decision will obtain corresponding rewards. The environment will change accordingly and the corresponding reward feedback and the next round of state will be transmitted back to the model. This iteration is carried out. The goal of the algorithm is to maximize the expectation of the cumulative reward obtained during multiple rounds of interaction, that is, to achieve the purpose of maximizing the prediction of anomalies.

[0051] Specifically, the training of the prediction model in the embodiment of the present invention can be performed as follows: obtaining sample data of the indicator to be predicted and inputting it into the pre-established prediction model to obtain the pending prediction result; screening the corresponding candidate data from the pending prediction result according to multiple data selection strategies, obtaining the model parameters corresponding to each data selection strategy to update the sample data, and inputting it into the prediction model again, and repeating the above-mentioned update operation until a trained prediction model is obtained.

[0052] The pending prediction results may include the prediction data obtained by the prediction model. At this time, the corresponding candidate data can be screened from the pending prediction results according to multiple data selection strategies to determine the sample data for the next round of training, that is, to obtain updated sample data, which is input into the prediction model as input data for the next round of model training. This cycle is repeated and the model training can be stopped after the training end conditions are met. At this time, the trained prediction model can be obtained, which is the prediction model used to execute this step.

[0053] In an embodiment of the present invention, taking the data selection strategy including the greedy selection strategy and the dynamic programming selection strategy as an example, after obtaining the pending prediction results in each round of training, the first candidate data can be screened from the pending prediction results according to the preset greedy selection strategy, and the second candidate data can be screened from the pending prediction results according to the preset dynamic programming selection strategy; the model parameters corresponding to the greedy selection strategy and the dynamic programming selection strategy are obtained to respectively calculate the selection probability of each candidate data, and the target data is determined from the first candidate data and the second candidate data according to the selection probability to update it as sample data. The model parameters are pre-configured, and specifically, the probabilities corresponding to the greedy selection strategy and the dynamic programming selection strategy can be directly set, so as to respectively determine the selection probabilities of the first candidate data and the second candidate data, and then the two can be merged to obtain the overall candidate data set, and at the same time, the selection probability of each candidate data in the candidate data set is obtained, and then the target data can be selected according to the probability to update it as sample data.

[0054] It should be noted that in the training of the reinforcement learning model, samples can be randomly selected, and then the decision on the samples required for the next round of training is made based on the current reward and state. The implementation is relatively simple and has high asymptotic performance, but its disadvantage is that the sampling efficiency is extremely low, resulting in a large amount of interaction with the environment. In order to reduce the cost of interacting with the environment, the greedy algorithm and the dynamic programming algorithm can be used in this step to implement the data selection strategy, that is, the greedy selection strategy and the dynamic programming selection strategy, and the samples with the highest local correlation with the abnormal prediction of the indicator are preferentially selected as the candidate data set, and then these candidate sets are scored through reinforcement learning to determine the subsequent training data. For example, the total size of the pending prediction result is 100kb, and each data has two attributes of correlation and spatial size (unit kb), that is, each data consists of a <correlation, size> binary group, and then the greedy selection strategy and the dynamic programming selection strategy are used to implement the screening of samples. Specifically, two model parameters w and b can be defined, where w represents the probability and b is the bias term (the size of the allowed deviation), providing an adjustable threshold to make the selection flexible. The probability of the first candidate data corresponding to the greedy selection strategy is w, and the probability of the second candidate data corresponding to the dynamic programming selection strategy is 1-w, that is, the combined candidate data set = w * greedy algorithm candidate set + (1-w) dynamic programming candidate set + b, so that the first candidate data and the second candidate data can be selected as target data with a certain probability, ensuring the sufficiency of the exploration sample space, making full use of their respective advantages, and maintaining the flexibility of adjustment of the bias term, thereby improving the overall prediction ability of the model. At the same time, the parameters w and b are also used as a parameter of reinforcement learning. For example, w=0.5 and b=-0.01 are set during initialization. As the reinforcement learning training proceeds, the values ​​of the parameters w and b will become more and more in line with the current environment.

[0055] It should be noted that the prediction model in the embodiment of the present invention can also be used as the detection model in step S103 to make predictions based on the historical indicator data of the monitoring indicators. After the prediction results are obtained, they are compared with the monitoring data generated in step S101 to determine whether the two are consistent, and then determine whether the indicators are abnormal.

[0056] S105: Generate and send warning information corresponding to each abnormal indicator.

[0057] For the abnormal indicators determined in the above steps, corresponding warning information can be generated to warn of the abnormality. The warning information can include the required content, such as the identification, data, time, etc. of the abnormal indicator. The sending method of the warning information can be unlimited.

[0058] It should be noted that the execution order of step S102, step S103 and step S104 in the embodiment of the present invention is not limited.

[0059] In the embodiment of the present invention, the log data associated with each monitoring indicator is obtained through a preset indicator link to generate monitoring data of each monitoring indicator and obtain the detection type of each monitoring indicator; for monitoring indicators of different detection types, abnormal indicators can be determined in different ways respectively, so as to determine the abnormal indicators in an adaptive way and improve the accuracy of abnormal identification; for the first type, abnormal indicator identification can be performed in a hierarchical manner, and for the second type, abnormal identification can also be performed through a preset detection model, so as to meet the needs of complex business and reduce the consumption of time cost; and abnormal prediction can also be performed on the data prediction of the prediction indicator through a prediction model, and the prediction model can be completed based on multiple preset data selection strategies to improve the accuracy and comprehensiveness of abnormal indicator identification. In this way, in the embodiment of the present invention, abnormal identification and abnormal prediction can be performed in an adaptive way according to the generated indicator data, which can not only meet the complex business environment and complex monitoring needs, but also improve the accuracy of abnormal identification.

[0060] Combine the following Figure 1 The embodiment shown in the figure specifically describes the method for data processing in the embodiment of the present invention. Figure 2 As shown, the method includes the following steps.

[0061] S201: calling a preset indicator link to obtain log data associated with each monitoring indicator, aggregating the log data, and generating monitoring data of each monitoring indicator.

[0062] S202: Obtain the detection type of each monitoring indicator, extract the indicator set whose detection type belongs to the first type, filter the monitoring indicators belonging to the target level, obtain the corresponding detection parameters, and call the detection strategy corresponding to the first type to determine the abnormal indicators.

[0063] S203: For each abnormal indicator, query the corresponding indicator list to match it with the indicator set, determine the monitoring indicator sequence associated with the abnormal indicator, and obtain monitoring data corresponding to the monitoring indicator sequence.

[0064] S204: Calculate the abnormal trend of the abnormal indicator according to the monitoring data and detection parameters of the abnormal indicator.

[0065] S205: Obtain the influence degree of each monitoring indicator in the monitoring indicator sequence on the abnormal indicator, determine the indicator to be identified in combination with the abnormal trend, and identify the associated abnormal indicator based on the indicator data of the indicator to be identified.

[0066] S206: Obtain the detection type of each monitoring indicator, extract the indicator set whose detection type belongs to the second type, obtain the corresponding monitoring data, call the corresponding detection model to detect the monitoring data of the indicator set, and determine the corresponding abnormal indicator.

[0067] S207: Generate and send warning information corresponding to each abnormal indicator.

[0068] It should be noted that the data processing principle in the embodiment of the present invention is the same as Figure 1 The corresponding data processing principles in the illustrated embodiments are the same and will not be described in detail here.

[0069] Combine the following Figure 1 The embodiment shown in the figure specifically describes the method for data processing in the embodiment of the present invention. Figure 3 As shown, the method includes the following steps.

[0070] S301: Obtain sample data of the indicator to be predicted and input it into a pre-established prediction model to obtain a pending prediction result.

[0071] S302: Filtering first candidate data from pending prediction results according to a preset greedy selection strategy, and filtering second candidate data from pending prediction results according to a preset dynamic programming selection strategy.

[0072] S303: Obtain model parameters corresponding to the greedy selection strategy and the dynamic programming selection strategy respectively to calculate the selection probability of each candidate data respectively, and determine the target data from the first candidate data and the second candidate data according to the selection probability to update it into sample data.

[0073] S304: input the updated sample data into the prediction model again, and repeat the above updating operation until a trained prediction model is obtained.

[0074] It should be noted that the data processing principle in the embodiment of the present invention is the same as Figure 1 The corresponding data processing principles in the illustrated embodiments are the same and will not be described in detail here.

[0075] In order to solve the problems existing in the prior art, an embodiment of the present invention provides a data processing device 400, such as Figure 4 As shown, the device 400 includes: a generating unit 401, which is used to call a preset indicator link to obtain log data associated with each monitoring indicator, aggregate the log data, generate monitoring data of each monitoring indicator, and obtain the detection type of each monitoring indicator; an identifying unit 402, which is used to extract an indicator set whose detection type belongs to the first type, screen the monitoring indicators belonging to the target level, obtain corresponding detection parameters, call the detection strategy corresponding to the first type to determine the abnormal indicator, and obtain the indicator monitoring associated with the abnormal indicator from the indicator set to identify the associated abnormal indicator; a determining unit 403, which is used to extract an indicator set whose detection type belongs to the second type, obtain the corresponding monitoring data, call the corresponding detection model to detect the monitoring data of the indicator set, and determine the corresponding abnormal indicator; the determining unit 403 is also used to extract the prediction indicator data of the indicator to be predicted from the indicator data, input the preset prediction model and obtain the prediction result, so as to determine the predicted abnormal indicator according to the prediction result, wherein the prediction model is trained based on multiple preset data selection strategies; a sending unit 404, which is used to generate and send the warning information corresponding to each of the abnormal indicators.

[0076] It should be understood that the manner of implementing the embodiments of the present invention is different from the Figure 1 The method of the illustrated embodiment is the same and will not be repeated here.

[0077] In one embodiment, the identification unit 402 is specifically used to: for each of the abnormal indicators, query the corresponding indicator list to match the indicator set, determine the monitoring indicator sequence associated with the abnormal indicator, and obtain the monitoring data corresponding to the monitoring indicator sequence; calculate the abnormal trend of the abnormal indicator based on the monitoring data of the abnormal indicator and the detection parameters, so as to identify the associated abnormal indicator in combination with the monitoring data corresponding to the monitoring indicator sequence.

[0078] In another embodiment, the identification unit 402 is specifically used to: obtain the degree of influence of each monitoring indicator in the monitoring indicator sequence on the abnormal indicator, determine the indicator to be identified in combination with the abnormal trend, and identify the associated abnormal indicator based on the indicator data of the indicator to be identified.

[0079] In another embodiment, the device 400 also includes: an input unit, used to obtain sample data of the indicator to be predicted, so as to input it into a pre-established prediction model to obtain a pending prediction result; a training unit, used to screen corresponding candidate data from the pending prediction result according to a plurality of the data selection strategies, obtain the model parameters corresponding to each of the data selection strategies to update the sample data, so as to input it into the prediction model again, and repeat the above-mentioned update operation until a trained prediction model is obtained.

[0080] In another embodiment, the training unit is specifically used to: screen the first candidate data from the pending prediction results according to a preset greedy selection strategy, and screen the second candidate data from the pending prediction results according to a preset dynamic programming selection strategy; obtain the model parameters corresponding to the greedy selection strategy and the dynamic programming selection strategy, respectively, to calculate the selection probability of each candidate data, and determine the target data from the first candidate data and the second candidate data according to the selection probability to update it to the sample data.

[0081] In another embodiment, the generation unit 401 is specifically used to: parse the log data, identify the corresponding data and event information according to the identifier of each monitoring indicator, so as to generate a data sequence corresponding to the time of each monitoring indicator; obtain the business logic of each monitoring indicator, so as to calculate the monitoring data of each monitoring indicator according to the corresponding data sequence and the business logic.

[0082] It should be understood that the manner of implementing the embodiments of the present invention is different from the Figure 1 , 2 , and 3 are the same as the embodiments shown in FIG. 3 and will not be described again here.

[0083] In the embodiment of the present invention, the log data associated with each monitoring indicator is obtained through a preset indicator link to generate monitoring data of each monitoring indicator and obtain the detection type of each monitoring indicator; for monitoring indicators of different detection types, abnormal indicators can be determined in different ways respectively, so as to determine the abnormal indicators in an adaptive way and improve the accuracy of abnormal identification; for the first type, abnormal indicator identification can be performed in a hierarchical manner, and for the second type, abnormal identification can also be performed through a preset detection model, so as to meet the needs of complex business and reduce the consumption of time cost; and abnormal prediction can also be performed on the data prediction of the prediction indicator through a prediction model, and the prediction model can be completed based on multiple preset data selection strategies to improve the accuracy and comprehensiveness of abnormal indicator identification. In this way, in the embodiment of the present invention, abnormal identification and abnormal prediction can be performed in an adaptive way according to the generated indicator data, which can not only meet the complex business environment and complex monitoring needs, but also improve the accuracy of abnormal identification.

[0084] According to an embodiment of the present invention, the embodiment of the present invention further provides an electronic device and a readable storage medium.

[0085] The electronic device of an embodiment of the present invention comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the data processing method provided by the embodiment of the present invention.

[0086] Figure 5 An exemplary system architecture 500 is shown to which the data processing method or data processing apparatus according to the embodiment of the present invention can be applied.

[0087] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0088] The user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various client applications can be installed on the terminal devices 501, 502, 503.

[0089] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0090] The server 505 may be a server that provides various services. The server may analyze and process the received data such as the product information query request, and feed back the processing result (such as product information - just an example) to the terminal device.

[0091] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data processing device is generally set in the server 505 .

[0092] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0093] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 suitable for implementing an embodiment of the present invention. Figure 6The computer system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0094] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0095] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0096] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0097] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0098] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a unit, a program segment, or a part of a code, and the above-mentioned unit, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The units involved in the embodiments of the present invention may be implemented by software or hardware. The units described may also be arranged in a processor, for example, it may be described as: a processor includes a generating unit, an identifying unit, a determining unit, and a sending unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the generating unit may also be described as a "unit for generating functions".

[0100] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device executes the data processing method provided by the present invention.

[0101] As another aspect, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method provided by an embodiment of the present invention.

[0102] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data processing, characterized in that: include: Calling a preset indicator link to obtain log data associated with each monitoring indicator, aggregating the log data, generating monitoring data for each monitoring indicator, and obtaining a detection type for each monitoring indicator; Extracting the indicator set of the detection type belonging to the first type, screening the monitoring indicators belonging to the target level, obtaining the corresponding detection parameters, calling the detection strategy corresponding to the first type to determine the abnormal indicator, and obtaining the indicator monitoring associated with the abnormal indicator from the indicator set to identify the associated abnormal indicator; Extracting the indicator set of the detection type belonging to the second type, obtaining corresponding monitoring data, calling the corresponding detection model to detect the monitoring data of the indicator set, and determining the corresponding abnormal indicator; Extracting prediction index data of the index to be predicted from the index data, inputting a preset prediction model and obtaining a prediction result, so as to determine the predicted abnormal index according to the prediction result, wherein the prediction model is trained based on a plurality of preset data selection strategies; The warning information corresponding to each abnormal indicator is generated and sent respectively.

2. The method according to claim 1, characterized in that Acquiring a monitoring indicator associated with the abnormal indicator from the indicator set to identify the associated abnormal indicator includes: For each of the abnormal indicators, query the corresponding indicator list to match with the indicator set, determine the monitoring indicator sequence associated with the abnormal indicator, and obtain the monitoring data corresponding to the monitoring indicator sequence; The abnormal trend of the abnormal indicator is calculated according to the monitoring data of the abnormal indicator and the detection parameter, so as to identify the associated abnormal indicator in combination with the monitoring data corresponding to the monitoring indicator sequence.

3. The method according to claim 2, characterized in that The step of identifying the associated abnormal indicators in combination with the monitoring data corresponding to the monitoring indicator sequence includes: The influence degree of each monitoring indicator in the monitoring indicator sequence on the abnormal indicator is obtained to determine the indicator to be identified in combination with the abnormal trend, and the associated abnormal indicator is identified based on the indicator data of the indicator to be identified.

4. The method according to claim 1, characterized in that Before inputting the preset prediction model and obtaining the prediction result, the method further includes: Obtaining sample data of the indicator to be predicted, and inputting it into a pre-established prediction model to obtain a pending prediction result; According to the multiple data selection strategies, the corresponding candidate data are screened from the pending prediction results, and the model parameters corresponding to each data selection strategy are obtained to update the sample data to input the prediction model again, and the above-mentioned update operation is repeated until a trained prediction model is obtained.

5. The method according to claim 4, characterized in that Screening corresponding candidate data from the pending prediction results according to the plurality of data selection strategies, and obtaining model parameters corresponding to each of the data selection strategies to update the sample data, including: Filtering the first candidate data from the pending prediction results according to a preset greedy selection strategy, and filtering the second candidate data from the pending prediction results according to a preset dynamic programming selection strategy; The model parameters corresponding to the greedy selection strategy and the dynamic programming selection strategy are obtained to respectively calculate the selection probability of each candidate data, and the target data is determined from the first candidate data and the second candidate data according to the selection probability to update the target data to the sample data.

6. The method according to claim 1, characterized in that Aggregating the log data to generate monitoring data of each monitoring indicator includes: Parsing the log data, identifying the corresponding data and time information according to the identifier of each monitoring indicator, so as to generate a data sequence corresponding to the time of each monitoring indicator; The business logic of each monitoring indicator is obtained to calculate the monitoring data of each monitoring indicator according to the corresponding data sequence and the business logic.

7. A data processing device, characterized in that: include: A generating unit, used to call a preset indicator link to obtain log data associated with each monitoring indicator, aggregate the log data, generate monitoring data of each monitoring indicator, and obtain a detection type of each monitoring indicator; an identification unit, configured to extract an indicator set of the detection type belonging to the first type, screen monitoring indicators belonging to the target level, obtain corresponding detection parameters, call the detection strategy corresponding to the first type to determine abnormal indicators, obtain indicator monitoring associated with the abnormal indicators from the indicator set, so as to identify the associated abnormal indicators; A determination unit, configured to extract the indicator set of the detection type belonging to the second type, obtain corresponding monitoring data, call the corresponding detection model to detect the monitoring data of the indicator set, and determine the corresponding abnormal indicator; The determination unit is further used to extract prediction indicator data of the indicator to be predicted from the indicator data, input a preset prediction model and obtain a prediction result, so as to determine the predicted abnormal indicator according to the prediction result, wherein the prediction model is trained based on multiple preset data selection strategies; The sending unit is used to generate and send warning information corresponding to each of the abnormal indicators.

8. The device according to claim 7, characterized in that The identification unit is specifically used for: For each of the abnormal indicators, query the corresponding indicator list to match with the indicator set, determine the monitoring indicator sequence associated with the abnormal indicator, and obtain the monitoring data corresponding to the monitoring indicator sequence; The abnormal trend of the abnormal indicator is calculated according to the monitoring data of the abnormal indicator and the detection parameter, so as to identify the associated abnormal indicator in combination with the monitoring data corresponding to the monitoring indicator sequence.

9. The device according to claim 8, characterized in that The identification unit is specifically used for: The influence degree of each monitoring indicator in the monitoring indicator sequence on the abnormal indicator is obtained to determine the indicator to be identified in combination with the abnormal trend, and the associated abnormal indicator is identified based on the indicator data of the indicator to be identified.

10. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images