Data backlog early warning method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data backlog early warning method, electronic equipment, a storage medium and a program product, and is applied to the technical field of computers. The method comprises the steps of obtaining performance index data of a data processing platform in a current time period, wherein the performance index data is used for representing a data processing speed of the data processing platform in the current time period; according to the performance index data in the current time period, predicting a data backlog state of the data processing platform at a future moment; and if the data backlog state indicates that the data backlog problem exists, outputting early warning information. And the effect of improving the data exception alarm real-time performance is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data backlog early warning method, electronic equipment, storage medium and program product. Background Art

[0002] As the scale of business processing in data centers continues to increase or decrease, in order to ensure the stability and security of business processing, it is necessary to monitor each infrastructure in the data center in real time and provide timely feedback on the current operating status of the infrastructure.

[0003] In the prior art, the infrastructure in the data center needs to be fully connected to the same data processing platform for monitoring and alarming; based on the data processing platform, a large amount of monitoring data generated by the data center about the infrastructure needs to be obtained, and these monitoring data need to use the same monitoring strategy standard, so as to achieve standardized alarm processing of the data. In order to achieve rapid alarm response to data, the prior art uses a data processing method that combines Flink distributed streaming and Kafka queues to process the currently collected monitoring data. Specifically: use the distributed Flink framework and Kafka queues to build a data processing platform for monitoring and alarming, perform data analysis and processing based on the monitoring data received in real time, monitor the monitoring data used to characterize the anomaly, and then perform data anomaly alarms.

[0004] Therefore, there is a technical problem in the prior art that the real-time performance of data anomaly alarm is low. Summary of the invention

[0005] The embodiments of the present application provide a data backlog warning method, an electronic device, a storage medium, and a program product to achieve the effect of improving the real-time performance of data anomaly warnings.

[0006] In a first aspect, an embodiment of the present application provides a data backlog warning method, comprising:

[0007] Obtaining performance indicator data of the data processing platform in the current time period, where the performance indicator data is used to characterize the data processing speed of the data processing platform in the current time period;

[0008] Based on the performance indicator data in the current time period, predict the data backlog status of the data processing platform at a future time;

[0009] If the data backlog status indicates that there is a data backlog problem, a warning message is output.

[0010] In a possible implementation, the performance indicator data includes data backlog and / or data waiting time for processing.

[0011] In a possible implementation, obtaining performance indicator data of a data processing platform in a current time period includes:

[0012] At each collection moment, collect the processing status of each data stored in the Kafka queue. The processing status is used to indicate whether the data has been processed;

[0013] Determine the amount of unprocessed data backlog at the time of collection based on the processing status of each data;

[0014] and / or,

[0015] For each collection moment, collect the arrival time and processing time of each data stored in the Kafka queue;

[0016] Determine the waiting time for data processing at the time of collection based on the arrival time and processing time of each data;

[0017] Among them, the Kafka queue is used to store the data that needs to be processed in the data processing platform, and the collection time is determined according to the current time period and the collection frequency.

[0018] In a possible implementation, predicting the data backlog state of the data processing platform at a future time according to the performance indicator data in the current time period includes:

[0019] According to the data waiting time in the current time period, a linear prediction algorithm is used to predict the data waiting time of the data processing platform at a future time.

[0020] According to the waiting time for data processing at a future moment and the preset waiting time, predict the data backlog status of the data processing platform at a future moment;

[0021] and / or;

[0022] According to the data backlog in the current time period, determine the data backlog change trend of the data processing platform at the future moment;

[0023] According to the changing trend of data backlog, the data backlog status of the data processing platform at future moments is predicted.

[0024] In a possible implementation, predicting the data backlog state of the data processing platform at a future moment according to the waiting time for data processing at a future moment and the preset waiting time includes:

[0025] If the waiting time for data processing at the future moment is longer than the preset waiting time, a data backlog state indicating that there is a data backlog problem is predicted;

[0026] If the data waiting time at the future moment is less than or equal to the preset waiting time, a data backlog state indicating that there is no data backlog problem is predicted;

[0027] Accordingly, based on the trend of data backlog changes, the data backlog status of the data processing platform at a future moment is predicted, including:

[0028] If the trend of the data backlog amount change is positive growth, a data backlog state indicating that there is a data backlog problem is predicted;

[0029] If the data backlog amount change trend is negative growth, then the prediction obtains a data backlog state indicating that there is no data backlog problem.

[0030] In a possible implementation, after outputting the warning information, the method further includes:

[0031] Based on the performance indicator data in the current time period, predict the target data backlog level of the data processing platform at a future time;

[0032] According to the target data backlog level and the preset mapping relationship, the target processing strategy corresponding to the target data backlog level is determined. The preset mapping relationship is used to represent the mapping relationship between the data backlog level and the processing strategy. The processing strategy includes adjusting the data processing order, business downgrade, fuse or dump.

[0033] In a second aspect, an embodiment of the present application provides a data backlog warning device, comprising:

[0034] An acquisition module is used to acquire performance indicator data of the data processing platform in the current time period, where the performance indicator data is used to characterize the data processing speed of the data processing platform in the current time period;

[0035] The first processing module is used to predict the data backlog state of the data processing platform at a future time according to the performance indicator data in the current time period;

[0036] The second processing module is used to output warning information when the data backlog status indicates that there is a data backlog problem.

[0037] In a possible implementation, the performance indicator data includes data backlog and / or data waiting time for processing.

[0038] In a possible implementation, the acquisition module is further used to:

[0039] At each collection moment, collect the processing status of each data stored in the Kafka queue. The processing status is used to indicate whether the data has been processed;

[0040] Determine the amount of unprocessed data backlog at the time of collection based on the processing status of each data;

[0041] and / or,

[0042] For each collection moment, collect the arrival time and processing time of each data stored in the Kafka queue;

[0043] Determine the waiting time for data processing at the time of collection based on the arrival time and processing time of each data;

[0044] Among them, the Kafka queue is used to store the data that needs to be processed in the data processing platform, and the collection time is determined according to the current time period and the collection frequency.

[0045] In a possible implementation manner, the first processing module is further configured to:

[0046] According to the data waiting time in the current time period, a linear prediction algorithm is used to predict the data waiting time of the data processing platform at a future time.

[0047] According to the waiting time for data processing at a future moment and the preset waiting time, predict the data backlog status of the data processing platform at a future moment;

[0048] and / or;

[0049] According to the data backlog in the current time period, determine the data backlog change trend of the data processing platform at the future moment;

[0050] According to the changing trend of data backlog, the data backlog status of the data processing platform at future moments is predicted.

[0051] In a possible implementation manner, the first processing module is further configured to:

[0052] When the waiting time for data processing at a future moment is longer than a preset waiting time, a data backlog state indicating that there is a data backlog problem is predicted;

[0053] When the data waiting time for processing at the future moment is less than or equal to the preset waiting time, a data backlog state indicating that there is no data backlog problem is predicted;

[0054] Accordingly, based on the trend of data backlog changes, the data backlog status of the data processing platform at a future moment is predicted, including:

[0055] When the data backlog amount change trend is positive growth, a data backlog state indicating that there is a data backlog problem is predicted;

[0056] When the data backlog amount change trend is negative growth, a data backlog state indicating that there is no data backlog problem is predicted.

[0057] In a possible implementation manner, the second processing module is further used for:

[0058] Based on the performance indicator data in the current time period, predict the target data backlog level of the data processing platform at a future time;

[0059] According to the target data backlog level and the preset mapping relationship, the target processing strategy corresponding to the target data backlog level is determined. The preset mapping relationship is used to represent the mapping relationship between the data backlog level and the processing strategy. The processing strategy includes adjusting the data processing order, business downgrade, fuse or dump.

[0060] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0061] Memory stores computer-executable instructions;

[0062] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0064] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0065] The data backlog warning method, device, storage medium and program product provided in the embodiment of the present application obtain the performance indicator data of the data platform in the current time period, determine the data processing speed of the data processing platform in the current time period based on the performance indicator data; predict the data backlog state of the data processing platform at a future moment according to the data processing speed, and judge whether the current data processing platform has a data backlog problem according to the data backlog state; if it is determined that the data processing platform has a data backlog problem, output the warning information corresponding to the data backlog. Compared with the prior art, when the present application performs data warning for the performance indicator data of the data center, it will predict whether there will be a data backlog problem at a future moment according to the data backlog state in the current time period. When it is predicted that there is a data backlog problem, a warning is made in time, achieving the effect of early warning; at the same time, it avoids the inability to quickly identify the abnormal performance indicator data in the backlog data when a large amount of data is backlogged, thereby achieving the effect of improving the real-time nature of data anomaly alarms. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0067] Figure 1 Schematic diagram of the process of the data backlog warning method provided for this application Figure 1 ;

[0068] Figure 2 Schematic diagram of the process of the data backlog warning method provided for this application Figure 2 ;

[0069] Figure 3 Schematic diagram of the process of the data backlog warning method provided for this application Figure 3 ;

[0070] Figure 4 A schematic diagram of the structure of the data backlog early warning system provided for this application;

[0071] Figure 5 A schematic diagram of the structure of the data backlog warning device provided for this application;

[0072] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0073] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0074] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0075] First, the terms involved in this application are explained:

[0076] Flink (Apache Flink): refers to an open source stream processing framework specifically for distributed data processing. Flink is designed to provide high-throughput, low-latency data processing capabilities and support event-driven applications. It can process unbounded and bounded data streams and is suitable for real-time data analysis, complex event processing, and batch processing tasks.

[0077] Kafka (Apache Kafka): refers to a distributed stream processing platform, mainly used to build real-time data pipelines and stream applications. Kafka is designed to provide high-throughput, low-latency data transmission capabilities, suitable for processing large-scale real-time data streams.

[0078] Flink metric: refers to a method provided by the Flink framework that collects internal indicators of tasks based on metrics and exposes related indicators to external systems. It supports multiple types of indicators, such as counters, single performance indicators, histograms, or user-defined indicators. Metric monitoring parameters can be used to optimize specific processes or assist in performance analysis.

[0079] In the prior art, when data is monitored on the infrastructure of a data center, it is necessary to collect monitoring data of the infrastructure in real time; use a unified data format to manage the monitoring data, and perform data processing based on the currently collected monitoring data, perform data anomaly analysis on it, and determine whether there is abnormal monitoring data; when it is determined that the monitoring data is abnormal monitoring data, an abnormal alarm is issued for the abnormal monitoring data.

[0080] However, when the data processing platform actually processes the monitoring data generated by various infrastructures in the data center, the data processing platform will accumulate a large amount of unprocessed monitoring data in a short period of time due to network jitter and changes in related systems. Among these accumulated unprocessed monitoring data, there may be a large amount of abnormal monitoring data, and the abnormal monitoring data that needs to be processed by alarm cannot be processed in time due to the data backlog, and the corresponding abnormal feedback cannot be made in real time based on the collected monitoring data; therefore, the existing technology has the technical problem of low real-time performance of data abnormality alarm.

[0081] In response to the above technical problems, the present application proposes the following technical concept: the problem of low real-time performance of data anomaly alarms due to data backlogs in the prior art. Therefore, it can be considered to timely predict the backlog status and make corresponding backlog warnings when there may be data backlogs. Specifically: obtain the performance indicator data of the data processing platform in the current time period, determine the data processing speed in the current data processing platform based on the performance indicator data, and predict the data backlog status of the data processing platform at a future moment based on the performance indicator data; thereby judging whether a backlog warning is needed based on the predicted data backlog status. When a backlog warning is needed, the backlog data in the data processing platform will be processed accordingly according to the backlog warning, so as to avoid the situation where abnormal monitoring data cannot be processed in time due to a large amount of data backlogs, thereby achieving the technical effect of improving the real-time performance of data anomaly alarms.

[0082] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0083] Figure 1 Schematic diagram of the process of the data backlog warning method provided for this application Figure 1 .like Figure 1 As shown, the method includes:

[0084] S101. Obtain performance indicator data of a data processing platform in a current time period.

[0085] In this step, the performance indicator data is used to characterize the data processing speed of the data processing platform in the current time period.

[0086] Optionally, in a possible implementation manner, the performance indicator data includes: data backlog and / or data waiting time for processing.

[0087] A possible implementation method for obtaining performance indicator data in this step is proposed here, specifically:

[0088] S1011. Determine the performance indicator interface provided by the data processing platform.

[0089] For example, when the data processing platform is implemented based on the Flink stream processing framework, the performance indicator interface for implementing performance data collection may be a Flink metric. Based on the interface, the internal performance indicator tree in the data processing platform may be collected, and the performance indicator data may be transmitted to an external system of the processing platform, thereby implementing performance data collection.

[0090] S1012: Call the performance indicator interface based on the preset collection program to obtain the data backlog and / or data waiting time of the data processing platform in the current time period.

[0091] S1013. Record the collection time of the performance indicator data, and store the performance indicator data and its corresponding collection time in a database.

[0092] The fields of the data table storing the performance indicator data in the database include: timestamp, data processing time difference and data backlog.

[0093] S102. Predicting the data backlog state of the data processing platform at a future time based on the performance indicator data in the current time period.

[0094] In this step, the method of predicting the data backlog status at a future time for the performance indicator data of the current time period may be: using a machine learning model to predict the data backlog status, or using time series analysis to predict the data backlog status.

[0095] For example, if the prediction of data backlog status is implemented based on a machine learning model, the specific implementation method includes:

[0096] S1021. Data collection and data preparation: collect historical performance indicator data, and normalize the historical performance indicator data to determine it as sample data in a unified format.

[0097] S1022. Extracting eigenvalues ​​from the sample data. The extraction of eigenvalues ​​needs to take into account the time series characteristics of the data itself. Determine a training data set for model training based on the eigenvalues ​​of the sample data.

[0098] S1023. Select a suitable model as the prediction model.

[0099] S1024. Train a prediction model based on the training data set, and generate a target prediction model that can be used to achieve data backlog status prediction.

[0100] S1025. Extract features based on the performance indicator data acquired in the current time period and determine them as model input parameters.

[0101] S1026. Input the model input parameters into the target prediction model to obtain the prediction data backlog status.

[0102] If it is necessary to explain, the data backlog status prediction in this step is only for exemplary purposes and is not specifically limited.

[0103] S103: If the data backlog status indicates that there is a data backlog problem, output a warning message.

[0104] In this step, the warning information may include: the severity of the data backlog, the time information when the data backlog may occur, the impact range of the data backlog, and the reasons causing the data backlog.

[0105] The impact of data backlog includes: business processes affected by data backlog; the causes of data backlog include: hardware failure and network jitter.

[0106] Optionally, after outputting the warning information, the existing data backlog problem can also be processed based on the performance indicator data in the current time period. A possible implementation method for processing the data backlog problem is:

[0107] S1031. Predict the target data backlog level of the data processing platform at a future time based on the performance indicator data in the current time period.

[0108] In this step, the target data backlog level is used to indicate the severity of the data backlog in the current data processing platform. According to the classification of the data backlog level, you can choose to modify the data processing order in the data processing platform, or reduce the quality of business operations in the data platform, or choose to suspend some of the businesses being processed in the data processing platform.

[0109] S1032. Determine a target processing strategy corresponding to the target data backlog level according to the target data backlog level and a preset mapping relationship.

[0110] In this step, the preset mapping relationship is used to represent the mapping relationship between the data backlog level and the processing strategy, and the processing strategy includes adjusting the data processing order, service degradation, fuse or dump.

[0111] Exemplarily, the target data backlog level is divided into mild, moderate and severe. Data backlog processing is performed according to the target backlog level and the type of data to be processed in the data processing platform, specifically:

[0112] a1. Classify the data to be processed in the current data processing platform into the following categories: available data, key business data, and general business data; prioritize the data to be processed according to the data category, where available data corresponds to priority 3, key business data corresponds to priority 2, and general business data corresponds to priority 1.

[0113] Among them, available data refers to whether the infrastructure corresponding to the current data to be processed is available; key business data refers to whether the current data to be processed is related to the core business; general business data refers to whether the current data to be processed has a low impact on the business.

[0114] a2. Select the appropriate target processing strategy based on data priority and current data backlog level.

[0115] a3. If the current data backlog level is mild, the data processing order of the data processing platform can be adjusted to give priority to processing the pending data of priority 3, so that the operation and maintenance personnel can quickly resolve problems that may cause system unavailability.

[0116] a3. If the current data backlog level is moderate, the business processes corresponding to some low-priority data to be processed can be suspended according to the priority of the data to be processed, so as to free up resources to process high-priority data to be processed.

[0117] a4. If the current data backlog level is severe, data processing can be stopped and only simple logging and data transmission operations can be performed; fuse processing can be performed to stop some business processes to protect the normal operation of the data processing platform; at the same time, the data to be processed can also be temporarily dumped into the database for subsequent processing.

[0118] For example, when the problem of data backlog is solved, data can be truncated according to the data dependency relationship of the current data to be processed to avoid generating more data to be processed. Specifically:

[0119] b1. Pre-configure the data dependencies between the businesses to be processed in the data processing platform.

[0120] b2. When it is predicted that the data processing platform will have a data backlog problem in the future based on the performance indicator data of the current time period, the dependency relationship between the data to be processed in the current data processing platform is determined based on the pre-configured data dependency relationship, and the production source of the data to be processed is determined based on the dependency relationship.

[0121] b3. Data truncation is performed based on the production source of the data to be processed, thereby reducing the continuous backlog of these data on the data processing platform.

[0122] Data truncation can be done by limiting the frequency of database queries and optimizing database query statements.

[0123] The data backlog warning method provided in the embodiment of the present application obtains the performance indicator data of the data processing platform in the current time period, determines the data processing speed in the current data processing platform according to the performance indicator data, and predicts the data backlog state of the data processing platform at a future moment according to the performance indicator data; thereby judging whether a backlog warning is needed based on the predicted data backlog state. When a backlog warning is needed, the backlog data in the data processing platform will be processed accordingly according to the backlog warning, so as to avoid the situation where the abnormal monitoring data cannot be processed in time due to a large amount of data backlog, thereby achieving the technical effect of improving the real-time performance of data abnormality alarms.

[0124] Figure 2 Schematic diagram of the process of the data backlog warning method provided for this application Figure 2 .like Figure 2 As shown, the method includes:

[0125] S201. At each collection moment, collect the processing status of each data stored in the Kafka queue.

[0126] In this step, the processing status is used to indicate whether the data has been processed. The Kafka queue is used to store the data that needs to be processed in the data processing platform. The collection time is determined according to the current time period and the collection frequency.

[0127] Exemplarily, the current time period refers to the length of time between the moment when the collection is started and the moment when the collection is stopped. The time length is set to 30 seconds, each second represents a collection moment, and the processing status corresponding to the data of 3 moments is finally collected; the processing status of the data is unprocessed or processed.

[0128] S202: Determine the amount of unprocessed data backlog at the time of collection according to the processing status of each data.

[0129] Exemplarily, assuming that data at 60 moments are collected, of which data at 40 moments are in an unprocessed state, the current data backlog is the total amount of data at these 40 moments.

[0130] S203: Determine the change trend of the data backlog of the data processing platform at a future time according to the data backlog in the current time period.

[0131] Exemplarily, a method for determining a changing trend of a data backlog based on a data backlog in a current time period may be as follows: there are moment one and moment two, and based on the data backlogs corresponding to the two moments and the time difference between the two moments, the changing trend of the data backlog is calculated.

[0132] Specifically: backlog change value = data backlog at moment two - data backlog at moment one; time change value = moment two - moment one; changing trend of data backlog = backlog change value divided by time change value.

[0133] S204. If the data backlog amount change trend is positive growth, a data backlog state indicating that there is a data backlog problem is predicted; if the data backlog amount change trend is negative growth, a data backlog state indicating that there is no data backlog problem is predicted.

[0134] Exemplarily, the changing trend of the data backlog is calculated based on the changing values ​​in multiple consecutive unit times. For example, the changing trend k = Δy / Δx; Δy = the backlog at time t2 - the backlog at time t1, Δx = t2-t1. When the changing trend of the data backlog shows a positive growth, it indicates that the data backlog corresponding to the current business is in a deteriorating state, indicating that there is a data backlog problem; when the changing trend of the data backlog shows a negative growth, it indicates that the data backlog corresponding to the current business is in a state of relief, indicating that there is no data backlog problem.

[0135] In this embodiment, the amount of data backlog in the current time period is collected, and based on the changing trend of the data backlog in the current time period, it is predicted whether there is a data backlog problem. When the data processing platform operates normally, it can be predicted whether there will be excessive data backlog in the future based on the current data backlog changing trend, so as to make corresponding warnings.

[0136] Figure 3 Schematic diagram of the process of the data backlog warning method provided for this application Figure 3 .like Figure 3 As shown, the method includes:

[0137] S301. For each collection moment, the arrival time and processing time of each data stored in the Kafka queue are collected.

[0138] In this step, a possible implementation method for recording the arrival time and processing time of each data in the Kafka queue at each collection time is:

[0139] S3011. Kafka producers record arrival time: When data is generated, Kafka producers can attach a timestamp to the message to mark the generation or arrival time of the data. The timestamp can be stored in the header field of the Kafka message or as part of the message.

[0140] S3012. Kafka consumer records processing time: In Kafka consumer, when processing each message, the current processing time is recorded. You can also use log records or attach the processing time to the processing result.

[0141] S3013. Data storage and analysis: The arrival time and processing time of each message are stored in the database for subsequent analysis. These timestamps can be stored using a relational database, a NoSQL database, or a dedicated time series database.

[0142] S302: Determine the waiting time for processing of the data at the collection time according to the arrival time and processing time of each data.

[0143] In this step, the data waiting time for processing refers to the processing delay of each piece of data, that is, the difference between the processing time and the arrival time.

[0144] Optionally, the data processing waiting time can be aggregated and analyzed according to the data waiting processing time of each data, specifically: at each collection moment, aggregate the waiting processing time of all messages processed in the time period; and calculate the average waiting time, maximum waiting time, and minimum waiting time.

[0145] S303: According to the data waiting time in the current time period, a linear prediction algorithm is used to predict the data waiting time at the data processing platform at a future time.

[0146] In this step, the linear prediction algorithm is used to predict the waiting time for data processing at a future time based on the waiting time for data processing collected in the current time period.

[0147] For example, a possible implementation method of predicting the waiting time for data processing at a future moment based on a linear prediction algorithm is as follows:

[0148] S3031. Collect historical data of the waiting time for data processing in multiple time periods.

[0149] S3032. Use a linear regression algorithm to build a prediction model, taking time as the independent variable and waiting time as the dependent variable.

[0150] S3033. Use the collected historical data to train the constructed prediction model to obtain a trained model.

[0151] S3034. Use the trained prediction model to predict the waiting time for data processing in the future.

[0152] S304. If the data waiting time for processing at a future moment is greater than the preset waiting time, a data backlog state indicating that there is a data backlog problem is predicted; if the data waiting time for processing at a future moment is less than or equal to the preset waiting time, a data backlog state indicating that there is no data backlog problem is predicted.

[0153] In this embodiment, based on the data waiting time collected in the current time period, the data waiting time at a future moment is predicted, so that when the data processing platform is operating normally, it is predicted whether the data processing platform will generate data congestion at a future moment, and the waiting state of the data to be processed is avoided for a long time. When it is predicted that a data backlog problem may occur in the future, the current data to be processed can be backlogged to avoid the data used to characterize abnormalities in the data to be processed not being processed in time.

[0154] Based on the above embodiments, the present application also provides a data backlog warning system. Figure 4 The structural diagram of the data backlog early warning system provided for this application is as follows: Figure 4 As shown, the data backlog warning system provided in this embodiment includes: a service registration module 401, a policy orchestration module 402, a backlog warning module 403, a parameter configuration module 404, a backlog coordination center 405, an indicator collection module 406, an execution record module 407, and a data processing platform 408.

[0155] The data processing platform 408 includes a plurality of business service units for processing monitoring data obtained from the data center.

[0156] The service registration module 402 is used to register the business services processed in the data processing platform 408 with the backlog coordination center 405 .

[0157] The policy arrangement module 402 is used to send the processing strategy corresponding to each data backlog state to the backlog coordination center 405 .

[0158] When backlog warning is performed on the data processing platform 408 , the indicator collection module 406 collects the performance indicator data generated when the data processing platform 408 processes the business in the current time period and sends the collected performance indicator data to the backlog warning module 403 .

[0159] The backlog warning module 403 predicts the data backlog status at a future time based on the performance indicator data, and determines whether the predicted data backlog status indicates the existence of a data backlog problem based on the preset judgment threshold pre-configured in the parameter configuration module 404. When the data backlog status indicates the existence of a data backlog problem, the backlog warning module 403 sends a warning message to the backlog coordination center 405.

[0160] The backlog coordination center 405 is used to send the backlog processing strategy to the data processing platform 408 based on the mapping relationship between the data backlog status corresponding to the warning information and the backlog processing strategy.

[0161] When receiving the backlog handling strategy, the data processing platform 408 executes the corresponding backlog handling strategy; at the same time, the execution recording module 407 records the process of the data processing platform 408 executing the backlog handling strategy.

[0162] Figure 5 The structural diagram of the data backlog warning device provided for this application is as follows: Figure 5 As shown, the data backlog warning device 50 provided in this embodiment includes:

[0163] An acquisition module 501 is used to acquire performance indicator data of the data processing platform in the current time period, where the performance indicator data is used to characterize the data processing speed of the data processing platform in the current time period;

[0164] The first processing module 502 is used to predict the data backlog state of the data processing platform at a future time according to the performance indicator data in the current time period;

[0165] The second processing module 503 is used to output warning information when the data backlog status indicates that there is a data backlog problem.

[0166] In a possible implementation, the performance indicator data includes data backlog and / or data waiting time for processing.

[0167] In a possible implementation, the acquisition module 501 is further configured to:

[0168] At each collection moment, collect the processing status of each data stored in the Kafka queue. The processing status is used to indicate whether the data has been processed;

[0169] Determine the amount of unprocessed data backlog at the time of collection based on the processing status of each data;

[0170] and / or,

[0171] For each collection moment, collect the arrival time and processing time of each data stored in the Kafka queue;

[0172] Determine the waiting time for data processing at the time of collection based on the arrival time and processing time of each data;

[0173] Among them, the Kafka queue is used to store the data that needs to be processed in the data processing platform, and the collection time is determined according to the current time period and the collection frequency.

[0174] In a possible implementation, the first processing module 502 is further configured to:

[0175] According to the data waiting time in the current time period, a linear prediction algorithm is used to predict the data waiting time of the data processing platform at a future time.

[0176] According to the waiting time for data processing at a future moment and the preset waiting time, predict the data backlog status of the data processing platform at a future moment;

[0177] and / or;

[0178] According to the data backlog in the current time period, determine the data backlog change trend of the data processing platform at the future moment;

[0179] According to the changing trend of data backlog, the data backlog status of the data processing platform at future moments is predicted.

[0180] In a possible implementation, the first processing module 502 is further configured to:

[0181] When the waiting time for data processing at a future moment is longer than a preset waiting time, a data backlog state indicating that there is a data backlog problem is predicted;

[0182] When the data waiting time for processing at the future moment is less than or equal to the preset waiting time, a data backlog state indicating that there is no data backlog problem is predicted;

[0183] Accordingly, based on the trend of data backlog changes, the data backlog status of the data processing platform at a future moment is predicted, including:

[0184] When the data backlog amount change trend is positive growth, a data backlog state indicating that there is a data backlog problem is predicted;

[0185] When the data backlog amount change trend is negative growth, a data backlog state indicating that there is no data backlog problem is predicted.

[0186] In a possible implementation, the second processing module 503 is further configured to:

[0187] Based on the performance indicator data in the current time period, predict the target data backlog level of the data processing platform at a future time;

[0188] According to the target data backlog level and the preset mapping relationship, the target processing strategy corresponding to the target data backlog level is determined. The preset mapping relationship is used to represent the mapping relationship between the data backlog level and the processing strategy. The processing strategy includes adjusting the data processing order, business downgrade, fuse or dump.

[0189] The data backlog warning device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail in this embodiment.

[0190] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0191] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.

[0192] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0193] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0194] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0195] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0196] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0197] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0198] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0199] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0200] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0203] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0204] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0205] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data backlog early warning method, characterized in that: include: Acquire performance indicator data of the data processing platform in the current time period, wherein the performance indicator data is used to characterize the data processing speed of the data processing platform in the current time period; Predicting the data backlog state of the data processing platform at a future time based on the performance indicator data in the current time period; If the data backlog status indicates that there is a data backlog problem, a warning message is output.

2. The method according to claim 1, characterized in that The performance indicator data includes data backlog and / or data waiting time for processing.

3. The method according to claim 2, characterized in that The obtaining of performance indicator data of the data processing platform in the current time period includes: At each collection moment, the processing status of each data stored in the Kafka queue is collected, and the processing status is used to indicate whether the data has been processed; Determine the amount of unprocessed data backlog at the collection time according to the processing status of each data; and / or, For each collection moment, the arrival time and processing time of each data stored in the Kafka queue are collected; Determine the waiting time for processing of the data at the acquisition time according to the arrival time and processing time of each data; The Kafka queue is used to store the data to be processed in the data processing platform, and the collection time is determined according to the current time period and the collection frequency.

4. The method according to claim 3, characterized in that The predicting, based on the performance indicator data in the current time period, the data backlog state of the data processing platform at a future time comprises: According to the data waiting time in the current time period, a linear prediction algorithm is used to predict the data waiting time of the data processing platform at the future time; According to the waiting time for data processing at the future moment and the preset waiting time, predicting the data backlog state of the data processing platform at the future moment; and / or; Determine, based on the data backlog in the current time period, a change trend of the data backlog of the data processing platform at a future time; The data backlog state of the data processing platform at a future moment is predicted according to the changing trend of the data backlog amount.

5. The method according to claim 4, characterized in that The predicting of the data backlog state of the data processing platform at a future moment according to the data waiting processing time at the future moment and the preset waiting time includes: If the data waiting time for processing at the future moment is longer than the preset waiting time, a data backlog state indicating that there is a data backlog problem is predicted; If the data waiting time for processing at the future moment is less than or equal to the preset waiting time, a data backlog state indicating that there is no data backlog problem is predicted; Accordingly, predicting the data backlog state of the data processing platform at a future time according to the data backlog change trend includes: If the data backlog amount change trend is positive growth, a data backlog state indicating that there is a data backlog problem is predicted; If the data backlog amount change trend is negative growth, then a data backlog state indicating that there is no data backlog problem is predicted.

6. The method according to any one of claims 3 to 5, characterized in that: After outputting the warning information, the method further includes: Predicting a target data backlog level of the data processing platform at a future time based on the performance indicator data in the current time period; According to the target data backlog level and the preset mapping relationship, the target processing strategy corresponding to the target data backlog level is determined. The preset mapping relationship is used to represent the mapping relationship between the data backlog level and the processing strategy. The processing strategy includes adjusting the data processing order, service degradation, fuse or dump.

7. A data backlog early warning device, characterized in that: Applied to a BMC server, the method includes: An acquisition module, used to acquire performance indicator data of the data processing platform in a current time period, wherein the performance indicator data is used to characterize the data processing speed of the data processing platform in the current time period; A first processing module, configured to predict the data backlog state of the data processing platform at a future time according to the performance indicator data in the current time period; The second processing module is used to output warning information when the data backlog status indicates that there is a data backlog problem.

8. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images