Job operation early warning method and device, medium and electronic equipment

Abstract

This application belongs to the field of artificial intelligence technology, specifically relating to a job operation early warning method, a job operation early warning device, a computer-readable medium, an electronic device, and a computer program product. The method includes: parsing and processing job operation logs to obtain monitoring data associated with timestamps; grouping the monitoring data according to multiple time windows with different window scales to obtain multiple sets of window data corresponding to the multiple time windows; performing feature mapping processing on the multiple sets of window data to obtain predicted data corresponding to the time window to be predicted; and triggering an early warning message for job operation anomalies based on the predicted data when the number of predicted data exceeding a threshold range exceeds a quantity threshold. Embodiments of this application can provide accurate early warnings for the job operation status of a big data job platform.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a job operation early warning method, a job operation early warning device, a computer-readable medium, an electronic device, and a computer program product. Background Technology

[0002] As the information age continues to advance and improve, the technology in the internet industry is also keeping pace with the times, resulting in an increasing amount of data in various fields and a growing demand for massive data processing.

[0003] Monitoring the operational status of big data platforms, predicting the cluster storage, node operation, CPU, memory, virtual resources, and the usage of business job execution for various tasks, and anticipating potential failures to implement countermeasures and safeguards before they occur are crucial for improving platform stability and are also an important aspect of data governance. Therefore, accurately predicting the operational status of big data platforms is a pressing issue that needs to be addressed in this field. Summary of the Invention

[0004] This application provides a job operation early warning method, a job operation early warning device, a computer-readable medium, an electronic device, and a computer program product, with the aim of providing accurate early warning of the job operation status of a big data job platform.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of the embodiments of this application, a job operation early warning method is provided, the method comprising:

[0007] The job execution logs are parsed and processed to obtain monitoring data associated with timestamps;

[0008] The monitoring data is grouped according to multiple time windows with different window scales to obtain multiple sets of window data corresponding to the multiple time windows;

[0009] The multiple sets of window data are subjected to feature mapping processing to obtain the prediction data corresponding to the time window to be predicted;

[0010] When the number of predicted data exceeding the threshold range exceeds the quantity threshold, an early warning message for abnormal operation is triggered based on the predicted data.

[0011] According to one aspect of the embodiments of this application, a job operation early warning device is provided, the device comprising:

[0012] The parsing module is configured to parse and process job execution logs to obtain monitoring data associated with timestamps;

[0013] The grouping module is configured to group the monitoring data according to multiple time windows with different window scales, so as to obtain multiple sets of window data corresponding to the multiple time windows;

[0014] The mapping module is configured to perform feature mapping processing on the multiple sets of window data to obtain prediction data corresponding to the time window to be predicted;

[0015] The early warning module is configured to trigger an early warning message for abnormal operation based on the predicted data when the number of predicted data exceeding the threshold range exceeds the quantity threshold.

[0016] According to one aspect of the embodiments of this application, a computer-readable medium is provided, on which a computer program is stored, which, when executed by a processor, implements the job operation early warning method as described above.

[0017] According to one aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the job operation early warning method as described above.

[0018] According to one aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the job operation early warning method as described in the above technical solutions.

[0019] In the technical solution provided in this application embodiment, monitoring data associated with timestamps is obtained by parsing the job operation log, and then the data is grouped according to different window scales to obtain window data corresponding to multiple time windows. Furthermore, feature mapping is performed on multiple groups of window data to obtain prediction data. By using data under multiple time windows to predict the job operation status, the early warning accuracy of job operation anomalies can be significantly improved.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 A flowchart illustrating the steps of a job operation early warning method according to one embodiment of this application is shown.

[0023] Figure 2 A flowchart illustrating the steps of a data preprocessing-based job execution early warning method in one embodiment of this application is shown.

[0024] Figure 3 The flowchart illustrates the steps of a job operation early warning method based on a long short-term memory network and a residual network in one embodiment of this application.

[0025] Figure 4 The model structure of the ResNet-LSTM fusion model in one embodiment of this application is shown.

[0026] Figure 5 The residual block structure in one embodiment of this application is shown.

[0027] Figure 6 A schematic diagram of the operation early warning device provided in the embodiments of this application is shown.

[0028] Figure 7 A schematic diagram of a computer system architecture suitable for implementing the embodiments of this application is shown. Detailed Implementation

[0029] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0030] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] Against the backdrop of the development of big data and data mining technologies, single-machine independent computing technology can no longer meet current data analysis needs, and parallel and distributed computing technologies are becoming increasingly prevalent in various fields. Distributed frameworks can not only solve the problem of storing massive amounts of data, but also integrate data from multiple data sources and at multiple levels. However, they also face the dynamic and complex nature of data management, which can bring certain monitoring difficulties to big data platforms.

[0034] Currently, most job scheduling systems on the market, such as Airflow and Oflow, generate alerts by monitoring task attributes; that is, alerts are triggered when a job fails or times out. Alerts based on job attributes have the following problems:

[0035] 1) Lack of predictability for job scheduling results. For example, when running job scheduling, the company usually chooses the early morning when CPU load is low and processes are idle. When executing tasks, execution begins according to the task's cycle, execution time, and pre-dependent relationships. If a failure occurs at a certain execution node, operations and maintenance personnel can only be notified via SMS. If operations and maintenance personnel do not see and resolve the issue immediately, it can easily cause delays in the overall business process.

[0036] 2) The alarm methods are limited, and the number and frequency of alarms are high. For example, if certain jobs fail, the only way to notify operations personnel is by sending SMS messages. Sending a large number of frequent alarm SMS messages means that the cause of the job failure still requires logging into the system to check the operation logs and tracing the processing logic. The SMS alarms only disappear after the task is forcibly approved. In short, as business expands, the alarm content will also increase, and sending a large number of alarm messages from different business processes can lead to the omission or oversight of critical alarm information.

[0037] 3) Job execution logs are distributed across different platforms, making data collection difficult. For example, business scheduling logs are typically stored on different devices. As the types of business continue to expand, the logs will accumulate to a massive amount over time. If the method of sequentially logging into machines is still used to view and retrieve logs, the operation is complex and inefficient.

[0038] In the related technologies of this application, artificial intelligence can be used to train predictive models for fault early warning; however, these predictive model-based methods still have the following drawbacks.

[0039] 1) The training set reflects information about anomalous events that have already occurred, requiring manual data collection and filtering, and consuming a significant amount of time for feature extraction. In reality, the number of anomalous events is far less than that of normal data. Even with manual annotation of the dataset, oversights can lead to labeling errors or omissions, causing the model to miss the optimal opportunity to learn about the event.

[0040] 2) The proposed business operation monitoring model and individual parameter values ​​consistently result in mismatches between predicted and actual values. In reality, current job scheduling is affected by factors such as cluster storage, node operating status, CPU, memory, virtual resources, and business operation execution status. Relying solely on traditional models applicable to all job scheduling and monitoring scenarios will lead to poor prediction results and make the model difficult to apply.

[0041] 3) Predictions can only be made for abnormal operations within a certain timeframe, making it difficult to effectively predict operations with varying time windows. Current prediction methods only cover a single window, failing to simultaneously address faults within minute, hour, day, or month-long time windows. While some proposed methods show good predictive performance for short-term, high-time-sensitivity data, effectively assessing indicator changes within minutes or hours, they struggle to predict rapidly changing indicators or those that change slowly over several days.

[0042] Furthermore, most predictive model-based early warning methods in related technologies rely on clustering and threshold setting to determine whether monitoring data exhibits anomalies. These methods require guidance from domain experts to define rules, leading to high maintenance costs and poor predictive performance.

[0043] To address the problems and shortcomings of the aforementioned related technologies, this application provides a method for monitoring and scheduling jobs applicable to the operation and maintenance of big data platforms, which analyzes factors such as cluster storage, node operation status, CPU, memory, virtual resources, and job execution status for prediction.

[0044] Especially in certain application scenarios, this application proposes a big data platform job operation early warning system and method based on hierarchical residual connection LSTM. The hierarchical residual connection LSTM model proposed in this application has a strong fitting ability for different job monitoring indicators. Combined with the multi-window prediction method proposed in this application, it can take into account short-term, medium-term, and long-term monitoring indicator prediction tasks, assess the possible factors of anomalies, effectively identify fluctuations in job scheduling anomalies, and issue alarms.

[0045] The following detailed description, in conjunction with specific embodiments, provides a detailed explanation of the technical solutions provided in this application, including the operation early warning method, operation early warning device, computer-readable medium, electronic device, and computer program product.

[0046] Figure 1 A flowchart illustrating the steps of a job execution early warning method according to one embodiment of this application is shown. Figure 1 As shown, the operation early warning method may include the following steps S110 to S140.

[0047] S110: Parse and process the job operation log to obtain monitoring data associated with timestamps.

[0048] Job execution logs are runtime data collected in real-time from the job node cluster of a big data platform over a period of time. This data is typically semi-structured streaming logs. By parsing and processing the job execution logs, a uniformly formatted timestamp and the monitoring data associated with that timestamp can be obtained.

[0049] S120: The monitoring data is grouped according to multiple time windows with different window scales to obtain multiple sets of window data corresponding to multiple time windows.

[0050] By grouping the monitoring data into multiple time windows with different window scales, a set of window data corresponding to each time window can be obtained.

[0051] In one embodiment of this application, a window sequence comprising multiple time windows with different window scales is obtained, wherein the window scale of each time window in the window sequence increases sequentially; the monitoring data is grouped according to each time window in the window sequence to obtain multiple sets of window data corresponding to multiple time windows.

[0052] For example, in this embodiment of the application, monitoring data can be grouped into three groups using three time windows with different window scales, thus obtaining three groups of window data. The first group of window data has a window scale of 5 minutes, in which the timestamps between any two adjacent monitoring data items are 5 minutes apart. The second group of window data has a window scale of 10 minutes, in which the timestamps between any two adjacent monitoring data items are 10 minutes apart. The third group of window data has a window scale of 30 minutes, in which the timestamps between any two adjacent monitoring data items are 30 minutes apart.

[0053] S130: Perform feature mapping processing on multiple sets of window data to obtain prediction data corresponding to the time window to be predicted.

[0054] The time window to be predicted is one or more of multiple time windows for grouping and processing monitoring data. The method for feature mapping processing of multiple sets of window data can include pre-training a prediction model corresponding to each time window to be predicted, and then, based on the matching relationship of window scales, inputting each set of window data into a prediction model with the same window scale, thereby obtaining the predicted data output by the prediction model. For example, in the embodiments of this application, the time windows to be predicted include three window scales: 5 minutes, 10 minutes, and 30 minutes. Inputting the first set of window data with a window scale of 5 minutes into the first prediction model yields the predicted data for the next 5 minutes output by the first prediction model; inputting the second set of window data with a window scale of 10 minutes into the second prediction model yields the predicted data for the next 10 minutes output by the second prediction model; and inputting the third set of window data with a window scale of 30 minutes into the third prediction model yields the predicted data for the next 30 minutes output by the third prediction model.

[0055] S140: When the number of predicted data exceeding the threshold range exceeds the quantity threshold, an early warning message for abnormal operation is triggered based on the predicted data.

[0056] The threshold range is a pre-set numerical range representing the normal operating status of a job. When the predicted data value is within the threshold range, the predicted job operating status is normal. When the predicted data value exceeds the threshold range, the number of predicted data exceeding the threshold range can be recorded. If this number also exceeds the quantity threshold, an early warning message for job operation anomalies can be triggered based on the predicted data. For example, if three predicted data points A, B, and C are obtained from step S130, and the quantity threshold is set to two, then if any two of the three predicted data points A, B, and C exceed the threshold range, it can be determined that there is an abnormal risk in the job operation, and an early warning message for job operation anomalies can be triggered based on the predicted data exceeding the threshold range.

[0057] In the job operation early warning method provided in this application embodiment, monitoring data associated with timestamps is obtained by parsing the job operation log, and then the data is grouped according to different window scales to obtain window data corresponding to multiple time windows. Furthermore, feature mapping is performed on multiple groups of window data to obtain prediction data. By using data under multiple time windows to predict the job operation status, the early warning accuracy of job operation anomalies can be significantly improved.

[0058] Figure 2 A flowchart illustrating the steps of a data preprocessing-based job execution early warning method according to one embodiment of this application is shown. Figure 2 As shown, the operation early warning method may include the following steps S210 to S260.

[0059] S210: Perform structured parsing on the job execution log to obtain timestamps and structured data associated with those timestamps.

[0060] The semi-structured streaming logs are cleaned, merged, and structured to extract data objects containing timestamps, monitoring metrics, etc., and generate corresponding structured data.

[0061] S220: Convert the format of the timestamp to obtain a normalized timestamp.

[0062] The format of different types of dates and times is converted to a unified floating-point timestamp format in seconds. The unified time is then normalized using the following formula:

[0063]

[0064] In the formula, t is the normalized timestamp, t' is the timestamp after format conversion, and t s t e These are the start and end timestamps of the log collection window, respectively.

[0065] The collection period of the monitoring log is represented by Δt, which is used in this application to characterize the window scale, i.e. the size of the time window.

[0066] S230: Normalize the structured data to obtain monitoring data associated with the normalized timestamp.

[0067] The parsed structured data is normalized, converting percentage metrics such as memory usage and process occupancy into decimals. Metrics such as network traffic and the number of running nodes are normalized to their maximum values ​​according to technical standards.

[0068] The processed monitoring data should conform to the following format:

[0069] <t,d> N (N = 1, 2, ..., N) max )

[0070] In the formula, t is the timestamp after processing in step S220, d is the data after normalization, and N represents the monitoring data index within the time window, with a maximum storage capacity of N. max indivual.

[0071] S240: The monitoring data is grouped according to multiple time windows with different window scales to obtain multiple sets of window data corresponding to multiple time windows.

[0072] S250: Perform feature mapping processing on multiple sets of window data to obtain prediction data corresponding to the time window to be predicted.

[0073] S260: When the number of predicted data exceeding the threshold range exceeds the quantity threshold, an early warning message for abnormal operation is triggered based on the predicted data.

[0074] The implementation details of steps S240 to S260 can be found in steps S120 to S140 of the above embodiments, and will not be repeated here.

[0075] In the job operation early warning method provided in this application embodiment, by performing structured parsing, timestamp conversion, and data normalization on the original log data, semi-structured streaming logs can be converted into structured data with the same format and time scale, thereby improving the early warning efficiency and accuracy.

[0076] Figure 3 A flowchart illustrating the steps of a job execution early warning method based on a long short-term memory network and a residual network in one embodiment of this application is shown. Figure 3 As shown, the operation early warning method may include the following steps S310 to S370.

[0077] S310: Parse and process the job operation log to obtain monitoring data associated with timestamps.

[0078] S320: The monitoring data is grouped according to multiple time windows with different window scales to obtain multiple sets of window data corresponding to multiple time windows.

[0079] The implementation details of steps S310 to S320 can be referred to S110 to S120 in the above embodiments, or S210 to S240 in the above embodiments, and will not be repeated here.

[0080] S330: Obtain a long short-term memory network with the same window size as the time window to be predicted.

[0081] S340: Extract data sequences with the same window scale as the time window to be predicted from multiple sets of window data.

[0082] S350: The data sequence is processed by feature mapping through a Long Short-Term Memory (LSTM) network to obtain the hidden layer vector output by the LSM network.

[0083] The time window to be predicted is Δt1, Δt2, ..., Δt n (n≥1). The minimum number of windows is 1, and the window size increases sequentially with the index n. Each time window to be predicted corresponds to a pre-trained Long Short-Term Memory network.

[0084] Long Short-Term Memory (LSTM) networks, as a variant of Recurrent Neural Networks (RNNs), can address the problem that RNNs cannot handle long sequences of information. When an input sequence is fed into an LSTM network, the hidden layer vector output by the LSTM cell state at time t is ht. The calculation formula is as follows:

[0085] zt=σ([t-1,xt]+b Z )

[0086] rt=σ([t-1,xt]+b r )

[0087]

[0088]

[0089] Among them, z t It acts as an input gate to control the amount of information left over from the past and the amount of new information; t The forget gate in the GRU structure is used to control how much information is written to the current candidate set from the previous state, if r t If the value is 0, then all previous states will be forgotten.

[0090] In one embodiment of this application, a method for obtaining a Long Short-Term Memory (LSTM) network with the same window size as the time window to be predicted may include: obtaining sample data with the same window size as the time window to be predicted; obtaining the ratio of the time window to be predicted to the minimum time window, and obtaining a regularization parameter positively correlated with the ratio; training an initial network based on the regularization parameter and the sample data to obtain an LTM network with the same window size as the time window to be predicted. The regularization parameter is a parameter in the loss function used to train the LTM network.

[0091] For example, the window ratio is represented by r. i The calculation formula is as follows:

[0092]

[0093] In the formula, i represents the window index, and Δt1 represents the smallest window.

[0094] Based on the ratios of each window, a regularization adjustment parameter λ is constructed. λ and r should satisfy the following equation:

[0095]

[0096] In the formula, λ increases as the window size increases, which can not only improve the prediction accuracy of the model, but also improve the ability of the medium- and long-term window to suppress abnormally frequent short-term noise fluctuations.

[0097] S360: The hidden layer vector is nonlinearly transformed by a residual network composed of multiple residual blocks to obtain the prediction data corresponding to the time window to be predicted.

[0098] In one embodiment of this application, a method for performing nonlinear transformation on a hidden layer vector using a residual network composed of multiple residual blocks may include: performing nonlinear transformation on the input data using each residual block in the residual network to obtain the residual value output by the residual block; wherein the input data of the first residual block is the hidden layer vector, and the input data of the next residual block is the residual value output by the previous residual block; and determining the prediction data corresponding to the time window to be predicted based on the residual values ​​output by each residual block.

[0099] For example, h t The hidden layer vectors are input into the residual network for nonlinear transformation. The residual network then performs dimensionality reduction on the hidden vectors. The depth of the residual connections is represented by D. i The residual network is represented by (i = 1, 2, ..., n), and consists of multiple interconnected residual blocks. Each residual block is represented as follows:

[0100] y=F(x,{W i}+x)

[0101] In the formula: x is the input vector, y represents the residual block, and also represents the output of the last layer, F(x,{W i}) represents the activation function.

[0102] For h, the output of the LSTM network at time t t After residual calculation, the new output state can be obtained, as shown in the following formula:

[0103] h′ t =f ω(h t) (h t )+h t

[0104] The data output by the residual network is the predicted data corresponding to the time window to be predicted.

[0105] In one embodiment of this application, a method for determining the prediction data corresponding to the time window to be predicted based on the residual values ​​output by each residual block may include: weighting the residual values ​​output by each residual block according to the attention weights obtained through pre-training to obtain the prediction data corresponding to the time window to be predicted.

[0106] By introducing an attention mechanism, the depth of the residual network can be effectively adjusted, allowing for a better assessment of the quality of feature fitting at different depths and avoiding overfitting.

[0107] S370: When the number of predicted data exceeding the threshold range exceeds the quantity threshold, an early warning message for abnormal operation is triggered based on the predicted data.

[0108] Based on the above embodiments, it can be seen that the technical solutions in some embodiments of this application have the following characteristics:

[0109] 1) Improve model algorithms and enhance model quality: Traditional machine learning requires different feature extraction methods for different types of job logs, and domain experts are needed to develop feature templates. In contrast, using deep learning for anomaly detection not only enhances the predictive ability of monitoring indicators but also achieves better fitting results. With appropriate parameters, the analysis and detection of job logs can be completed with almost no manual intervention in feature extraction.

[0110] 2) Flexible control of prediction timeliness: Apply time windows of different sizes to various monitoring indicators and run them in parallel with the training task.

[0111] 3) Possesses anti-overfitting ability and improves the accuracy of medium- and long-term window prediction: Add a regularization loss function with positive correlation of time window to the network model to reduce the impact of abnormally frequent short-term noise fluctuations on medium- and long-term windows, improve model accuracy, and prevent the model from overfitting.

[0112] The method for providing job operation early warning in an application scenario according to the embodiments of this application may include the following process.

[0113] S1: Monitoring log collection.

[0114] The system retrieves resource monitoring logs, process monitoring logs, node monitoring logs, and job monitoring logs from HDFS, Yarn, HBase, and Zookeeper, and sends the data to the S2 module for data preprocessing.

[0115] S2: Data preprocessing.

[0116] Preprocessing is performed on the job operation log data, including structured parsing of the raw log data, timestamp conversion, data normalization, and filtering out data that does not conform to the specifications.

[0117] Log structured parsing includes the following sub-steps:

[0118] S21: Log structure parsing.

[0119] Factors such as cluster storage, node operation status, CPU, memory, virtual resources, business job execution status, and resource usage are structured and converted into JSON format. Data objects containing timestamps and monitoring metrics are then extracted from these data objects.

[0120] S22: Timestamp conversion.

[0121] Taking the time "2022-06-20 17:15:57" as an example, it can be converted to "1655716557", with the unit being seconds. The normalization process for the unified time is calculated using the following formula:

[0122]

[0123] In the formula, t is the normalized timestamp, t' is the timestamp after format conversion, and t s t e These are the start and end timestamps of the log collection window, respectively.

[0124] The collection period of the monitoring log is represented by Δt, which is used in this embodiment to characterize the size of the time window.

[0125] In the formula, let Δt = 4800s, and let the end timestamp be t. e =1655716557, and the monitoring data timestamp is t′=1655716557. Therefore, the start timestamp is t. s =1655711757, t=1 after normalization.

[0126] S23: Data normalization processing.

[0127] The parsed structured data undergoes data normalization, converting percentage metrics such as memory usage and process occupancy into decimals. For example, a current process occupancy rate of 91% translates to 0.91 after decimalization.

[0128] The processed monitoring data should conform to the following format:

[0129] <t,d> N (N = 1, 2, ..., N) max )

[0130] In the formula, t is the timestamp after processing in sub-step S22, d is the data after normalization processing in sub-step S23, and N represents the monitoring data index within the time window, with a maximum storage capacity of N. max indivual.

[0131] S3: Model training.

[0132] In different time windows, a hierarchical residual connection LSTM model is constructed. The monitoring log data processed in step S2 is input into the model for iterative training to obtain a fitting model that adapts to different window sizes, which is then used for anomaly detection in step S4.

[0133] The construction of the hierarchical residual connection LSTM model described above includes the following sub-steps:

[0134] S31: Determine the size of the training time window.

[0135] The window values ​​are Δt1 = 4800s, Δt2 = 57600s, and Δt3 = 115200s.

[0136] S32: Construct ResNet-LSTM fusion models under different time windows.

[0137] Figure 4 The model structure of the ResNet-LSTM fusion model in one embodiment of this application is shown, as follows: Figure 4 As shown,

[0138] The network includes the following characteristics:

[0139] (1) It contains multiple residual block structures, each of which contains at least two softmax fully connected layers.

[0140] (2) It contains a Long Short-Term Memory (LSTM) network.

[0141] (3) The Dynamic ReLU activation function is used.

[0142] (4) Introduce an attention mechanism module to adjust the weights.

[0143] S321: Construct a hierarchical residual connection LSTM model structure under different time windows, and use the residual network (ResNet) as the backbone network for extracting high-level features (specific steps are as shown in S3211 and S3212).

[0144] S3211: Constructing an LSTM network: As a variant of the Recurrent Neural Network (RNN), the LSTM network can solve the problem that RNNs cannot handle long sequence information. It takes the input sequence x = (x1, x2, ..., x...) as input. n The input is fed into a hierarchical residual connection LSTM network. At time t, the LSTM cell state output hidden layer vector is h. t The calculation formula is as follows:

[0145] zt=σ(Wz[t-1,xt]+b z )

[0146] rt=σ(Wr[t-1,xt]+b r )

[0147]

[0148]

[0149] In the formula, z tIt acts as an input gate to control the amount of information left over from the past and the amount of new information; t The forget gate in the GRU structure is used to control how much information is written to the current candidate set from the previous state, if r t If the value is 0, then all previous states will be forgotten.

[0150] S3212: Constructing the ResNet network: h t The hidden layer vectors are input into the residual network for nonlinear transformation. The residual network then performs dimensionality reduction on the hidden vectors. The depth of the residual connections is represented by D. i The residual network is represented by (i = 1, 2, 3, 4, 5), and consists of multiple residual blocks connected together.

[0151] Figure 5 The residual block structure in one embodiment of this application is shown, as follows: Figure 5 As shown, each residual block is represented as follows:

[0152] y=F(x,{W i}+x)

[0153] In the formula: x is the input vector, y represents the residual block, and also represents the output of the last layer, F(x,{W i}) represents the activation function.

[0154] S322: Global encoding is performed using a dynamic ReLU activation function. A suitable activation function is obtained through dynamic selection. The dimension-reduced vector obtained in step S3212 is then augmented to restore the original input dimension. The specific calculation steps are as follows:

[0155] S3221: Dimensionality Reduction: Pooling layers are used to compress feature vectors, thereby reducing parameters and network complexity. The formula is as follows:

[0156] x′=(x1,x2,...x m (m=9)

[0157] S3222: Normalization: The feature vector after dimensionality reduction is input into the ReLU activation function g(x) and normalized to filter out unimportant feature information, ensuring that the vector value is between [-1, 1]. The formula is as follows:

[0158]

[0159] S3223: Dimensional Increase Processing: Add the normalized vector values ​​from step S3222 to the original input dimension. The calculation formula is as follows:

[0160]

[0161]

[0162]

[0163]

[0164] h′ t =f(Wh t +Wx t +b)

[0165] In the formula, the hyperparameter α i and β i They are respectively and The initial value, λ, is used to determine the vector coefficient matrix. a With λ b As a scalar for controlling the residual range, hyperparameter From the coefficient vector matrix The optimal values ​​are calculated and selected as the coefficients of the activation function to determine the activation function, where c is the number of channels and i is the number of functions. For the definition of a dynamic activation function, the input vector The value of can determine the hyperparameter coefficient.

[0166] At this point, for the h output by the LSTM network at time t t After residual calculation, the new output state can be obtained, as shown in the following formula:

[0167] h′ t =f ω(ht) (h t )+h t

[0168] S323: Define the attention mechanism module: By introducing an attention mechanism, the depth of the residual network can be effectively adjusted, better judging the quality of feature fitting at different depths and avoiding overfitting. The output vectors H′=h′ of LSTM cells at different depths are then used. kt(k>＝0) The input is fed into the attention mechanism, and the calculation steps are as follows:

[0169]

[0170] In the formula, softmax(·) is used for normalization to obtain the weight parameters Z. k The depth with the largest weight is selected as the connection depth of the residual layer, and the output of the current maximum depth is used as the vector h of the LSTM cell output layer. t .

[0171] S33: Calculate the window ratio, using r i The calculation formula is as follows:

[0172]

[0173] In the formula, i represents the window index, and Δt1 represents the smallest window.

[0174] S331: Based on the ratios of each window in S331, construct the regularization adjustment parameter λ. λ and r should satisfy the following equation:

[0175]

[0176] In the formula, λ increases as the window size increases. This not only improves the prediction accuracy of the model, but also enhances the ability of the medium- and long-term window to suppress abnormally frequent short-term noise fluctuations.

[0177] S34: Model training.

[0178] The LSTM model is trained using hierarchical residual connections with time windows of different sizes. The time and data after data preprocessing in step S2 are used as the training set and input into the model. After the model stops training, the prediction model under the i-th window is obtained, and step S4 is entered for anomaly prediction.

[0179] S4: Anomaly prediction.

[0180] Predicted data from different windows are fed into the model trained in S3 to calculate the predicted value at the current moment. The prediction results are then saved to a distributed database, stored with the timestamp as the primary key. This database can be used as a data source to send to S5 for alarm judgment.

[0181] The sub-steps for calculating the predicted value are:

[0182] S41: Determine the size of the prediction time window.

[0183] Define the data to be predicted, D. i The window size is Δt i ′, The first j points of the data to be predicted are used as the time window and input into the prediction model. Taking i=5 and j=4, the time window should satisfy the following formula:

[0184] Δt5′=(D1,D2,D3,D4)

[0185] S42: Real-time predictive data processing.

[0186] Forecast data often fluctuates. A single moving average method is used to smooth the forecast data, taking m sets of data for calculation. The formula is as follows:

[0187]

[0188] In the formula, each new forecast value is a correction to the previous moving average forecast value.

[0189] S5: Alarm determination.

[0190] The threshold range is determined by setting a maximum and a minimum threshold. If the predicted value D i If the threshold is within the specified range, no alarm is triggered, and the process returns to step S1. If the threshold is exceeded, and the frequency of exceeding the threshold is f within a certain time period, step S6 is triggered.

[0191] S6: Send service alarm.

[0192] Once an alarm is triggered, the warning log is converted into text according to a preset template and notified to maintenance personnel via instant messaging software communication groups, SMS, telephone, etc., so that effective handling can be carried out as soon as possible.

[0193] Based on the above application scenarios, it can be seen that compared with current threshold-based alarm software and similar related technologies, the embodiments of this application have the following advantages:

[0194] 1) Improve the model algorithm to enhance model quality.

[0195] In the related technologies of this application, the proposed business operation monitoring model and individual parameter values ​​often result in mismatches between predicted and actual values. This application's embodiment uses a hierarchical residual connection LSTM model for anomaly detection. By stacking the LSTM network depth, the nonlinear fitting capability of features is enhanced, avoiding gradient vanishing or gradient exploding phenomena. By introducing a dynamic activation function, not only can the number of parameters in the computation process be reduced, but the optimal activation function can also be obtained, further improving the model's fitting effect.

[0196] 2) Flexible control over forecast timeliness.

[0197] Existing technologies can only predict abnormal operations within a certain timeframe, and are difficult to effectively predict monitoring logs with different time windows of varying sizes. This application's embodiments apply different time windows of varying sizes to various monitoring indicators, running them in parallel with the training task.

[0198] 3) It has the ability to resist overfitting and improves the accuracy of medium- and long-term window prediction.

[0199] Traditional neural networks tend to overfit as their depth increases, leading to poor validation performance. This application incorporates a time-window-positive correlation attention mechanism into the network model. This not only dynamically adjusts the residual network depth but also reduces the impact of frequent short-term noise fluctuations on the medium- to long-term window and prevents overfitting.

[0200] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0201] The following describes an embodiment of the apparatus of this application, which can be used to execute the operation early warning method in the above embodiments of this application. Figure 6 A schematic block diagram of the operation early warning device provided in an embodiment of this application is shown. Figure 6 As shown, the operation early warning device 600 may include:

[0202] The parsing module 610 is configured to parse and process the job execution log to obtain monitoring data associated with timestamps;

[0203] The grouping module 620 is configured to group the monitoring data according to multiple time windows with different window scales to obtain multiple sets of window data corresponding to the multiple time windows;

[0204] The mapping module 630 is configured to perform feature mapping processing on the multiple sets of window data to obtain prediction data corresponding to the time window to be predicted.

[0205] The early warning module 640 is configured to trigger an early warning message for abnormal operation based on the predicted data when the number of predicted data exceeding the threshold range exceeds the quantity threshold.

[0206] In one embodiment of this application, based on the above embodiments, the parsing module 610 can be further configured to: perform structured parsing on the job operation log to obtain a timestamp and structured data associated with the timestamp; perform format conversion on the timestamp to obtain a normalized timestamp; and perform normalization processing on the structured data to obtain monitoring data associated with the normalized timestamp.

[0207] In one embodiment of this application, based on the above embodiments, the grouping module 620 may be further configured to: acquire a window sequence including multiple time windows with different window scales, wherein the window scale of each time window in the window sequence increases sequentially; and group the monitoring data according to each time window in the window sequence to obtain multiple sets of window data corresponding to the multiple time windows.

[0208] In one embodiment of this application, based on the above embodiments, the mapping module 630 may further include:

[0209] The network acquisition module is configured to acquire a long short-term memory network with the same window scale as the time window to be predicted;

[0210] The sequence extraction module is configured to extract data sequences from the multiple sets of window data that have the same window scale as the time window to be predicted.

[0211] The feature mapping module is configured to perform feature mapping processing on the data sequence through the long short-term memory network to obtain the hidden layer vector output by the long short-term memory network;

[0212] The nonlinear transformation module is configured to perform a nonlinear transformation on the hidden layer vector through a residual network composed of multiple residual blocks to obtain prediction data corresponding to the time window to be predicted.

[0213] In one embodiment of this application, based on the above embodiments, the nonlinear transformation module can be further configured to: perform nonlinear transformation on the input data through each residual block in the residual network to obtain the residual value output by the residual block; wherein, the input data of the first residual block is the hidden layer vector, and the input data of the next residual block is the residual value output by the previous residual block; and determine the prediction data corresponding to the time window to be predicted based on the residual values ​​output by each residual block.

[0214] In one embodiment of this application, based on the above embodiments, the nonlinear transformation module can be further configured to: weight the residual values ​​output by each residual block according to the attention weights obtained through pre-training, to obtain the prediction data corresponding to the time window to be predicted.

[0215] In one embodiment of this application, based on the above embodiments, the network acquisition module may be further configured to: acquire sample data with the same window size as the time window to be predicted; acquire the ratio of the time window to be predicted to the minimum time window, and acquire a regularization adjustment parameter positively correlated with the ratio; train the initial network according to the regularization adjustment parameter and the sample data to obtain a long short-term memory network with the same window size as the time window to be predicted.

[0216] The specific details of the operation early warning device provided in the various embodiments of this application have been described in detail in the corresponding method embodiments, and will not be repeated here.

[0217] Figure 7 A schematic block diagram of a computer system architecture for implementing an electronic device according to embodiments of the present application is shown.

[0218] It should be noted that, Figure 7The computer system 700 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0219] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for system operation. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output interface 705 (I / O interface) is also connected to the bus 704.

[0220] The following components are connected to the input / output interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a local area network card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0221] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit 701, it performs various functions defined in the system of this application.

[0222] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0223] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0224] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0225] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0226] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0227] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for early warning of operational activities, characterized in that, include: The job execution logs are parsed and processed to obtain monitoring data associated with timestamps; The monitoring data is grouped according to multiple time windows with different window scales to obtain multiple sets of window data corresponding to the multiple time windows; The multiple sets of window data are subjected to feature mapping processing to obtain the prediction data corresponding to the time window to be predicted; When the number of predicted data exceeding the threshold range exceeds the quantity threshold, an early warning message for abnormal operation is triggered based on the predicted data. The process of performing feature mapping on the multiple sets of window data to obtain prediction data corresponding to the time window to be predicted includes: acquiring a Long Short-Term Memory (LSTM) network with the same window scale as the time window to be predicted; extracting a data sequence with the same window scale as the time window to be predicted from the multiple sets of window data; performing feature mapping on the data sequence through the LTM network to obtain the hidden layer vector output by the LTM network; and performing a nonlinear transformation on the hidden layer vector through a residual network composed of multiple residual blocks to obtain the prediction data corresponding to the time window to be predicted.

2. The operation early warning method according to claim 1, characterized in that, The job execution logs are parsed to obtain monitoring data associated with timestamps, including: The job execution log is parsed in a structured manner to obtain timestamps and structured data associated with those timestamps; The timestamp is converted to a normalized timestamp. The structured data is normalized to obtain monitoring data associated with the normalized timestamp.

3. The operation early warning method according to claim 1, characterized in that, The monitoring data is grouped according to multiple time windows with different window scales to obtain multiple sets of window data corresponding to the multiple time windows, including: Obtain a window sequence comprising multiple time windows with different window sizes, wherein the window size of each time window in the window sequence increases sequentially; The monitoring data is grouped according to each time window in the window sequence to obtain multiple sets of window data corresponding to the multiple time windows.

4. The operation early warning method according to claim 1, characterized in that, The hidden layer vector is nonlinearly transformed by a residual network composed of multiple residual blocks to obtain prediction data corresponding to the time window to be predicted, including: The input data is nonlinearly transformed by each residual block in the residual network to obtain the residual value output by the residual block; wherein, the input data of the first residual block is the hidden layer vector, and the input data of the next residual block is the residual value output by the previous residual block. The prediction data corresponding to the time window to be predicted is determined based on the residual values ​​output by each residual block.

5. The operation early warning method according to claim 4, characterized in that, Determine the prediction data corresponding to the time window to be predicted based on the residual values ​​output by each residual block, including: The residual values ​​output by each residual block are weighted according to the attention weights obtained through pre-training to obtain the prediction data corresponding to the time window to be predicted.

6. The operation early warning method according to claim 1, characterized in that, Obtaining a long short-term memory network with the same window size as the time window to be predicted includes: Obtain sample data with the same window scale as the time window to be predicted; Obtain the ratio of the time window to be predicted to the minimum time window, and obtain the regularization adjustment parameter that is positively correlated with the ratio; The initial network is trained based on the regularization adjustment parameters and the sample data to obtain a long short-term memory network with the same window size as the time window to be predicted.

7. A work operation early warning device, characterized in that, include: The parsing module is configured to parse and process job execution logs to obtain monitoring data associated with timestamps; The grouping module is configured to group the monitoring data according to multiple time windows with different window scales, so as to obtain multiple sets of window data corresponding to the multiple time windows; The mapping module is configured to perform feature mapping processing on the multiple sets of window data to obtain prediction data corresponding to the time window to be predicted; The early warning module is configured to trigger an early warning message for abnormal operation based on the predicted data when the number of predicted data exceeding the threshold range exceeds the quantity threshold. The process of performing feature mapping on the multiple sets of window data to obtain prediction data corresponding to the time window to be predicted includes: acquiring a Long Short-Term Memory (LSTM) network with the same window scale as the time window to be predicted; extracting a data sequence with the same window scale as the time window to be predicted from the multiple sets of window data; performing feature mapping on the data sequence through the LTM network to obtain the hidden layer vector output by the LTM network; and performing a nonlinear transformation on the hidden layer vector through a residual network composed of multiple residual blocks to obtain the prediction data corresponding to the time window to be predicted.

8. A computer-readable medium, characterized in that, The computer-readable medium stores a computer program that, when executed by a processor, implements the job operation early warning method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions to implement the job operation early warning method as described in any one of claims 1 to 6.

Images