Method and device for realizing component performance bottleneck prediction processing based on capacity and call chain statistical data, processor and storage medium thereof

Abstract

The invention relates to a method for realizing prediction processing aiming at component performance bottleneck based on capacity and call chain statistical data. The method comprises the following steps: collecting and storing component call chain index data; the capacity system obtains and stores component pressure measurement index data; analyzing and storing the component pair call chain and the index information; constructing calling link information from the source component to the target component; determining a module TPS, time consumption and CPU utilization rate relation prediction model through neural network deep learning; and comparing the pressure measurement capacity baseline values of the components, and analyzing that the component reaching the capacity bottleneck first is a link bottleneck. According to the method, the device, the processor and the computer readable storage medium for predicting the component performance bottleneck based on the capacity and call chain statistical data, a deep learning neural network algorithm is introduced, and higher prediction accuracy is achieved. And based on the topological dependence of the call chain, analyzing and predicting the component which firstly reaches the performance bottleneck from the call chain so as to confirm that the component is the performance bottleneck point of the system.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to the fields of call links and deep learning, and specifically refers to a method, device, processor and computer-readable storage medium thereof for predicting component performance bottlenecks based on capacity and call chain statistics. Background Art

[0002] An important part of ensuring the stability of computer systems is to use call chain technology to monitor and alert the system. By applying probe point placement and data statistics technology to each application component of the system, it is possible to obtain the call relationship between components, access volume, request time, CPU, memory, disk, network and other indicator data, and regularly store the above data in a time series database. Based on the above time series data, analysis can be performed and a trend chart can be constructed.

[0003] In the field of computer deep learning, neural networks are a machine learning model based on artificial neural networks. They are a layered "neuron" network structure that learns from input data, captures the relationship between response variables and predictor variables, and makes predictions.

[0004] Based on the above call link indicator data, system administrators often judge whether a component has reached a performance bottleneck based on experience and by configuring alarm thresholds. The call chain system alarm module determines whether a performance bottleneck alarm should be triggered by calculating time series data in real time and comparing it with the alarm threshold.

[0005] At present, the constant threshold-based method for judging whether an application component has reached a performance bottleneck often becomes distorted as the application changes (new services, performance optimization, scaling, etc.), which requires manual observation of the application and adjustment of the threshold. The constant threshold method for judging application bottlenecks relies too much on human subjective judgment, and the accuracy of the threshold setting is not necessarily accurate. In addition, the above method lacks predictive ability. When the system really reaches a capacity bottleneck, it may have caused a production accident, affecting the user experience. In order to solve the above problems, the present invention provides a method and device for intelligently predicting component performance bottlenecks based on call chain indicator data. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and to provide a method, device, processor and computer-readable storage medium thereof that meet the requirements of high accuracy, low error and wide application range for predicting component performance bottlenecks based on capacity and call chain statistical data.

[0007] In order to achieve the above-mentioned purpose, the method, device, processor and computer-readable storage medium thereof for predicting component performance bottlenecks based on capacity and call chain statistics of the present invention are as follows:

[0008] The method for predicting and processing component performance bottlenecks based on capacity and call chain statistics is mainly characterized in that the method comprises the following steps:

[0009] (1) Collect and store component call chain indicator data;

[0010] (2) The capacity system obtains and stores component stress test indicator data;

[0011] (3) Parse and store component call chains and indicator information;

[0012] (4) Constructing the call link information from the source component to the destination component;

[0013] (5) Determine the component TPS, time consumption, and CPU utilization relationship prediction model through neural network deep learning;

[0014] (6) Compare the component stress test capacity baseline values ​​and analyze the component that first reaches the capacity bottleneck as the link bottleneck.

[0015] Preferably, the step (1) is specifically as follows:

[0016] The Agent collects component process operation indicators and writes the indicator data to the message queue Kafka.

[0017] Preferably, the step (2) specifically comprises the following steps:

[0018] (2.1) Obtain component performance stress test indicator data;

[0019] (2.2) Store component performance stress test indicators in the capacity system;

[0020] (2.3) Provide interface query to obtain performance indicator data of each component in the system dimension.

[0021] Preferably, the step (3) specifically comprises the following steps:

[0022] (3.1) Use Flink to perform real-time aggregation calculations on component call chains and monitoring indicator data stored in Kafka;

[0023] (3.2) Perform aggregate statistics to generate various indicator data for component pairs;

[0024] (3.3) The obtained indicator data is saved in the time series database InfluxDB.

[0025] Preferably, the step (4) specifically comprises the following steps:

[0026] (4.1) Query the time series database InfluxDB to obtain all stored component call link information data;

[0027] (4.2) Add the queried call link information data to the graph data structure to build a complete component call link topology graph;

[0028] (4.3) Traverse all nodes in the topology graph and define the source component and the destination component;

[0029] (4.4) Use the depth-first search or breadth-first search algorithm to traverse the topology graph, find all paths from the source component to the destination component, and record the complete link information from the source component to the destination component;

[0030] (4.5) Store the traversed link information into the specified storage medium.

[0031] Preferably, the step (5) specifically comprises the following steps:

[0032] (5.1) Perform data preprocessing;

[0033] (5.2) Establish a neural network model;

[0034] (5.3) Send the data information to the information layer for preprocessing, train and learn the features, and obtain the prediction information of CPU utilization;

[0035] (5.4) Set a number of neurons, set the hyperparameter gamma, and initialize the RBF layer center;

[0036] (5.5) Calculate the error between the model prediction result and the actual result through the loss function, and calculate the gradient of the loss function relative to each parameter;

[0037] (5.6) K-fold cross validation method is used for training.

[0038] Preferably, the step (5.1) specifically comprises the following steps:

[0039] (5.1.1) Obtain component CPU utilization, TPS data, request duration data, and link call relationships;

[0040] (5.1.2) Clean the data to remove outliers and missing values;

[0041] (5.1.3) Standardize or normalize the data.

[0042] Preferably, the step (5.6) specifically comprises the following steps:

[0043] (5.6.1) Divide the original data set into K equal parts;

[0044] (5.6.2) From the K divided parts, select the first part as the test set and the rest as the training set;

[0045] (5.6.3) Use the training set to train the model, use the test set to verify the model output, and calculate the accuracy of the model;

[0046] (5.6.4) Repeat (5.6.2) and (5.6.3) K times, using a different part as the test set each time;

[0047] (5.6.5) The average accuracy is taken as the model accuracy.

[0048] Preferably, the step (6) specifically comprises the following steps:

[0049] (6.1) Traverse each component in the link;

[0050] (6.2) Use pre-trained deep learning models to predict the performance of each component;

[0051] (6.3) Set the TPS amount to predict component CPU resources and request duration;

[0052] (6.4) Compare the predicted CPU resource usage and request processing time with the component's stress test capacity baseline value;

[0053] (6.5) Analyze the component that reaches the capacity bottleneck first.

[0054] The main feature of the device for predicting and processing component performance bottlenecks based on capacity and call chain statistical data is that the device comprises:

[0055] a processor configured to execute computer-executable instructions;

[0056] A memory stores one or more computer executable instructions, which, when executed by the processor, implement the various steps of the method for predicting and processing component performance bottlenecks based on capacity and call chain statistics.

[0057] The processor for implementing predictive processing for component performance bottlenecks based on capacity and call chain statistics has the main feature that the processor is configured to execute computer executable instructions, and when the computer executable instructions are executed by the processor, the various steps of the above-mentioned method for implementing predictive processing for component performance bottlenecks based on capacity and call chain statistics are implemented.

[0058] The main feature of the computer-readable storage medium is that a computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the above-mentioned method for predicting and processing component performance bottlenecks based on capacity and call chain statistics.

[0059] The method, device, processor and computer-readable storage medium of the present invention for predicting component performance bottlenecks based on capacity and call chain statistics are adopted. Production applications introduce agent mode, and the application does not need to modify the source code, so that call chain indicator data can be collected at low cost. The deep learning neural network algorithm is introduced, which has a higher prediction accuracy. Based on the call chain topology dependency, the component that first reaches the performance bottleneck is predicted from the call chain analysis, so as to confirm that it is the performance bottleneck point of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The present invention is a flowchart of a method for predicting and processing component performance bottlenecks based on capacity and call chain statistics.

[0061] Figure 2 A CPU utilization prediction model diagram of the method for predicting component performance bottlenecks based on capacity and call chain statistics of the present invention.

[0062] Figure 3 An LSTM network expansion diagram according to the time dimension for the method of predicting and processing component performance bottlenecks based on capacity and call chain statistics of the present invention.

[0063] Figure 4 The present invention is a flow chart of the RBF-BP neural network construction method for implementing a method for predicting and processing component performance bottlenecks based on capacity and call chain statistical data.

[0064] Figure 5 A diagram of a K-fold cross-validation method for implementing a method for predicting component performance bottlenecks based on capacity and call chain statistics of the present invention.

[0065] Figure 6 A diagram of a link performance bottleneck component determination method for implementing a method for predicting component performance bottlenecks based on capacity and call chain statistical data of the present invention. DETAILED DESCRIPTION

[0066] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

[0067] The method of the present invention for predicting and processing component performance bottlenecks based on capacity and call chain statistics includes the following steps:

[0068] (1) Collect and store component call chain indicator data;

[0069] (2) The capacity system obtains and stores component stress test indicator data;

[0070] (3) Parse and store component call chains and indicator information;

[0071] (4) Constructing the call link information from the source component to the destination component;

[0072] (5) Determine the component TPS, time consumption, and CPU utilization relationship prediction model through neural network deep learning;

[0073] (6) Compare the component stress test capacity baseline values ​​and analyze the component that first reaches the capacity bottleneck as the link bottleneck.

[0074] As a preferred embodiment of the present invention, the step (1) is specifically:

[0075] The Agent collects component process operation indicators and writes the indicator data to the message queue Kafka.

[0076] As a preferred embodiment of the present invention, the step (2) specifically comprises the following steps:

[0077] (2.1) Obtain component performance stress test indicator data;

[0078] (2.2) Store component performance stress test indicators in the capacity system;

[0079] (2.3) Provide interface query to obtain performance indicator data of each component in the system dimension.

[0080] As a preferred embodiment of the present invention, the step (3) specifically comprises the following steps:

[0081] (3.1) Use Flink to perform real-time aggregation calculations on component call chains and monitoring indicator data stored in Kafka;

[0082] (3.2) Perform aggregate statistics to generate various indicator data for component pairs;

[0083] (3.3) The obtained indicator data is saved in the time series database InfluxDB.

[0084] As a preferred embodiment of the present invention, the step (4) specifically comprises the following steps:

[0085] (4.1) Query the time series database InfluxDB to obtain all stored component call link information data;

[0086] (4.2) Add the queried call link information data to the graph data structure to build a complete component call link topology graph;

[0087] (4.3) Traverse all nodes in the topology graph and define the source component and the destination component;

[0088] (4.4) Use the depth-first search or breadth-first search algorithm to traverse the topology graph, find all paths from the source component to the destination component, and record the complete link information from the source component to the destination component;

[0089] (4.5) Store the traversed link information into the specified storage medium.

[0090] As a preferred embodiment of the present invention, the step (5) specifically comprises the following steps:

[0091] (5.1) Perform data preprocessing;

[0092] (5.2) Establish a neural network model;

[0093] (5.3) Send the data information to the information layer for preprocessing, train and learn the features, and obtain the prediction information of CPU utilization;

[0094] (5.4) Set a number of neurons, set the hyperparameter gamma, and initialize the RBF layer center;

[0095] (5.5) Calculate the error between the model prediction result and the actual result through the loss function, and calculate the gradient of the loss function relative to each parameter;

[0096] (5.6) K-fold cross validation method is used for training.

[0097] As a preferred embodiment of the present invention, the step (5.1) specifically comprises the following steps:

[0098] (5.1.1) Obtain component CPU utilization, TPS data, request duration data, and link call relationships;

[0099] (5.1.2) Clean the data to remove outliers and missing values;

[0100] (5.1.3) Standardize or normalize the data.

[0101] As a preferred embodiment of the present invention, the step (5.6) specifically comprises the following steps:

[0102] (5.6.1) Divide the original data set into K equal parts;

[0103] (5.6.2) From the K divided parts, select the first part as the test set and the rest as the training set;

[0104] (5.6.3) Use the training set to train the model, use the test set to verify the model output, and calculate the accuracy of the model;

[0105] (5.6.4) Repeat (5.6.2) and (5.6.3) K times, using a different part as the test set each time;

[0106] (5.6.5) The average accuracy is taken as the model accuracy.

[0107] As a preferred embodiment of the present invention, the step (6) specifically comprises the following steps:

[0108] (6.1) Traverse each component in the link;

[0109] (6.2) Use pre-trained deep learning models to predict the performance of each component;

[0110] (6.3) Set the TPS amount to predict component CPU resources and request duration;

[0111] (6.4) Compare the predicted CPU resource usage and request processing time with the component's stress test capacity baseline value;

[0112] (6.5) Analyze the component that reaches the capacity bottleneck first.

[0113] The device for predicting and processing component performance bottlenecks based on capacity and call chain statistics of the present invention comprises:

[0114] a processor configured to execute computer executable instructions;

[0115] A memory stores one or more computer executable instructions, which, when executed by the processor, implement the various steps of the method for predicting and processing component performance bottlenecks based on capacity and call chain statistics.

[0116] The present invention provides a processor for implementing predictive processing of component performance bottlenecks based on capacity and call chain statistics, wherein the processor is configured to execute computer executable instructions. When the computer executable instructions are executed by the processor, the various steps of the above-mentioned method for implementing predictive processing of component performance bottlenecks based on capacity and call chain statistics are implemented.

[0117] The computer-readable storage medium of the present invention stores a computer program thereon, and the computer program can be executed by a processor to implement the various steps of the above-mentioned method for predicting and processing component performance bottlenecks based on capacity and call chain statistics.

[0118] In a specific implementation of the present invention, starting from the perspective of the call link, the link relationship between the components is analyzed, and machine learning training is performed based on the collected component monitoring indicator data to accurately predict the component capacity, effectively detect and discover the performance bottleneck components in each link of the system, and discover the risk points in the system in advance.

[0119] The method of intelligently predicting component performance bottlenecks based on capacity and call chain indicator data of the present invention has the following main features:

[0120] (1) Collect and store component call chain indicator data;

[0121] (2) The capacity system obtains and stores component stress test indicator data;

[0122] (3) Parse and store component call chains and indicator information;

[0123] (4) Constructing the call link information from the source component to the destination component;

[0124] (5) Determine the component TPS, time consumption, and CPU utilization relationship prediction model through neural network deep learning;

[0125] (6) Predict the component in the link that will reach the performance bottleneck first.

[0126] The step (1) collects component process running indicators through Agent and writes the indicator data into the message queue kafka;

[0127] The Agent has different implementations according to the programming language. Taking the Java agent as an example, it uses the characteristics of JavaInstrumentation to dynamically modify the bytecode after the process is started to inject the monitoring logic code to achieve monitoring and business indicator collection;

[0128] The collected information includes the source of the call, destination data and indicator data. The indicator data includes the number of requests per second, CPU usage, memory usage, and interface request time. The collected indicators are spit into the message queue Kafka through the Kafka Producer client.

[0129] Furthermore, the component performance stress test indicators in step (2) are stored in the capacity system, and the performance indicator data of each component in the system dimension is obtained through interface query;

[0130] The component performance indicator data includes the component's TPS, memory usage, CPU usage, and request duration information under performance stress testing;

[0131] Further, the step (3) aggregates and calculates the component pair call chain and monitoring indicator data stored in the Kafka consumption call chain through the Flink real-time computing engine;

[0132] The aggregate calculation is performed by aggregating statistics by component pairs, time, and call links to obtain indicator data of component pairs including TPS, CPU utilization per second, memory utilization per second, and request time per second, and save them in the time series database Influxdb;

[0133] Further, the step (4) constructs a call link topology graph between components by querying the system component call link information data stored in the time series database, and then traverses the graph topology structure to obtain the link information from the source component to the destination component;

[0134] The source component is a component that has no called traffic, and the destination component is a component that does not call other components;

[0135] Further, the step (5) pre-processes the component CPU data, TPS data, and request time data, and then constructs a component capacity prediction model through data training using a neural network, and proposes a component performance bottleneck prediction method based on RBF-BP (Radial Basis Function and Back Propagation) and LSTM (Long Short Time Memory) deep learning models;

[0136] The data training is performed based on pre-processed component CPU data, TPS data, request time consumption data and preset training end conditions;

[0137] The preset training end conditions include: the global error tends to a minimum value and / or the number of iterations reaches a preset value;

[0138] Further, the step (6) traverses the components of the call link, and based on the deep learning prediction model, predicts the component CPU resources and request time by setting the TPS amount, and then compares the component stress test capacity baseline value, and analyzes that the component that first reaches the capacity bottleneck is the link bottleneck;

[0139] The set TPS amount is to determine the prediction model input amount TPS data through the call amount relationship ratio of the link components at the same time.

[0140] The present invention intelligently predicts component performance bottlenecks based on call chain indicator data, and in a specific embodiment includes the following steps:

[0141] (1) Collect and store component call chain indicator data. This step collects the running indicators of component processes through Agent and writes these indicator data into the message queue Kafka. The specific implementation steps are as follows:

[0142] (1.1) Agent design and implementation: The implementation of Agent will be different according to different programming languages. Taking Java Agent as an example, by using the Java Instrumentation feature, the monitoring logic code is injected by dynamically modifying the bytecode after the process is started, thereby realizing the monitoring of the process and the collection of business indicators.

[0143] (1.2) Indicator data collection: The running indicators of the component process are collected through the Agent. The collected data information includes the source data and destination data of the call chain and various performance indicator data, such as the number of transactions per second (TPS) processed by the server, CPU usage, memory usage, interface request time, etc. After starting the process, the Java Agent dynamically modifies the bytecode of the target process and injects the monitoring logic code. At this time, the monitoring logic will collect the above performance indicator data in real time.

[0144] (1.3) Storage of indicator data: The collected data is sent to the Kafka message queue through the Kafka Producer client to ensure efficient data transmission and storage.

[0145] (2) The capacity system obtains and stores component stress test indicator data. The capacity system is responsible for obtaining the indicator data of each component under performance stress testing. These data are stored in the capacity system, and the performance indicator data of each component in the system dimension can be obtained through interface query. The specific implementation steps are as follows:

[0146] (2.1) Obtaining component performance stress test indicator data: Performance stress testing refers to testing the performance of a system or component under simulated high-load conditions to evaluate its performance in actual operation. The capacity system is responsible for obtaining indicator data of each component under performance stress testing. The indicators include TPS, memory usage, CPU usage, request duration information, etc.

[0147] (2.2) Component performance stress testing indicators are stored in the capacity system: Various indicator data generated during the performance stress testing process are collected and stored in the capacity system in real time or periodically to ensure data integrity and consistency.

[0148] (2.3) Provide an interface to query and obtain the performance indicator data of each component in the system dimension: The capacity system provides an API interface, allowing users or other systems to query the performance indicator data of each component through programming.

[0149] (3) Parse and store component call chains and indicator information. This step aggregates and calculates the component call chains and monitoring indicator data stored in Kafka through the Flink real-time computing engine, and finally saves the aggregated indicator data to the time series database InfluxDB. The specific implementation steps are as follows:

[0150] (3.1) Flink real-time computing engine aggregation calculation: The high-performance, low-latency Flink real-time computing engine can be used to process and analyze streaming data in real time. Flink can be used to perform real-time aggregation calculations on component call chains and monitoring indicator data stored in Kafka, providing high throughput and low-latency computing capabilities.

[0151] (3.2) Aggregate statistics by component, time, and call link: The aggregate calculation process aggregates statistics by component pair, time, and call link to generate various indicator data for component pairs. These indicator data include TPS, memory usage, CPU usage, request time information, etc.

[0152] (3.3) Data is stored in the time series database InfluxDB: The indicator data obtained by aggregation calculation is finally saved in the time series database InfluxDB, which is specially used to store and query time series data.

[0153] (4) Construct the call link information from the source component to the destination component. The specific implementation steps are as follows:

[0154] (4.1) Data query and extraction: Query the time series database InfluxDB to obtain all stored component call link information data. Extract the call link information data, including the call source component, target component, and call time.

[0155] (4.2) Construct component call link topology graph: Use graph data structure (such as directed graph) to represent component call link. Each node represents a component, and the edge represents the relationship between one component calling another component. Add the queried call link information data to the graph data structure to construct a complete component call link topology graph.

[0156] (4.3) Define source components and destination components: traverse all nodes in the topology graph, find nodes without incoming edges, and mark these nodes as source components; traverse all nodes in the topology graph, find nodes without outgoing edges, and mark these nodes as destination components.

[0157] (4.4) Traversing the topology graph: Starting from each source component, use the depth-first search (DFS) or breadth-first search (BFS) algorithm to traverse the topology graph and find all paths from the source component to the destination component. For each path, record the information of all nodes (components) on the path, that is, the complete link information from the source component to the destination component.

[0158] (4.5) Result storage and analysis: The link information obtained by traversal is stored in the specified storage medium for subsequent query and analysis. Use visualization tools (such as Graphviz) to visualize the component call link topology and link information to help operation and maintenance personnel intuitively understand the call relationship of system components.

[0159] (5) Determine the component TPS, time consumption and CPU utilization relationship prediction model through neural network deep learning. This step is the key content of the present invention. The specific implementation steps are as follows:

[0160] (5.1) First, data preprocessing is required.

[0161] (5.1.1) Collect data: Obtain data about the component’s CPU utilization, TPS data, request duration, and link call relationships.

[0162] (5.1.2) Clean data: remove outliers and missing values ​​to ensure data accuracy and completeness.

[0163] (5.1.3) Standardized / normalized data: In order to improve the training effect, the data is standardized or normalized.

[0164] (5.2) Next, we need to build a neural network model. In view of the time-varying and nonlinear characteristics of the indicator data, we use a deep learning model based on RBF-BP and LSTM.

[0165] (5.3) After the four sets of data information are sent to the information layer for preprocessing, the features are first trained and learned based on the advantages of LSTM, and then the features enter the RBF-BP hybrid neural network for adaptive learning to finally obtain the prediction information of CPU utilization.

[0166] (5.3.1) LSTM is a special type of recurrent neural network (RNN) designed specifically for processing and predicting sequence data. The memory cells in LSTM can retain information for a long time, allowing the network to retain and utilize information at any position in the sequence, and LSTM is particularly suitable for processing time series data or sequence data because it can model temporal dependencies. Figure 4 Expand the graph of the LSTM network along the time dimension.

[0167] In the present invention, CPU data, TPS data, and request time data are represented as F(t1), F(t2), and F(t3) respectively. The underlying logic of the LSTM neural network algorithm is to set three "gates", namely the forget gate, input gate, and output gate. The "gate" is composed of a Sigmoid function and a dot multiplication logic, and its function is to determine whether the data can pass effectively. The output of the Sigmoid function is [0, 1]. If it takes 0, the input data will be completely forgotten. If it takes 1, the input data will be completely saved.

[0168] The main component of the forget gate is the Sigmoid function, whose input is the output data G(t-1) of the previous unit and the input data F(t) at this time. It determines the degree of forgetting of the state information of the H(t-1) unit by generating a (0, 1) value. The calculation expression is:

[0169] f t =μ(W f [G(t-1), F(t)]+b f )……(1.1)

[0170] Among them, W f represents the weight matrix of the forget gate, b f Represents the bias term of the forget gate, and μ is the Sigmoid function.

[0171] The input gate controls the input of data through the Tanh activation function. The Tanh function can be calculated to obtain the variable H(t) * , the input gate will generate a value of (0, 1) and assign it to H(t) * , control the new input data, combined with the output f of the forget gate t And the output of the input gate in t , we get the cell state information H(t) of the new memory cell. The calculation expression of the input gate is

[0172] in t =μ(W i [G(t-1), F(t)]+b i )……(1.2)

[0173] H(t) * =Tanh(W h [G(t-1), F(t)]+b h )……(1.3)

[0174] H(t)=f t ·H(t-1)+in t ·H(t) * ……(1.4)

[0175] Among them, Wi represents the weight matrix of the input gate, b i is the bias term of the input gate, W h is the weight matrix of the unit state, b h is the bias term of the cell state.

[0176] The purpose of the output gate is to control the amount of information loss in the current cell state. It generates a value between 0 and 1 and applies it to each element of the cell state, thereby controlling the degree of filtering of the cell state. In other words, the purpose of the output gate is to reduce unnecessary information loss through this mechanism. The expression of the output gate is

[0177] Out t =μ(W out [G(t-1), F(t)]+b out )……(1.5)

[0178] G(t)=Out t tanh(H(t))…(1.6)

[0179] Among them, W out is the weight matrix of the output gate, b out is the bias term of the output gate.

[0180] (5.3.2) In order to improve the accuracy and efficiency of CPU utilization prediction, the present invention adopts a method of combining BP neural network with RBF neural network to make full use of the advantages of both. Specifically, this method uses BP neural network and RBF neural network as two independent sub-networks. Figure 5 This is the RBF-BP neural network model diagram. The structure of the BP neural network is set to have one hidden layer and use the Sigmoid function as the activation function; while the RBF neural network has only one hidden layer and the activation function is the Gaussian function. By using the output of the BP neural network as the input of the RBF neural network, the two networks are effectively fused to form a new hybrid model. This model can improve the accuracy and efficiency of prediction while maintaining high training accuracy.

[0181] After the data is processed by the LSTM neural network, it is input into the RBF-BP neural network. The algorithm of the RBF-BP neural network mainly includes two parts: forward propagation and error back propagation. During the forward propagation process, the input value of each neuron is multiplied by the weight matrix, and the bias term is added, and then the output of the neuron is calculated by the activation function. The forward propagation expression is

[0182] pre i =μ(W i ·F(i)+bi )……(1.7)

[0183] Among them, W i is the weight matrix of the input gate, b i is the bias term of the input gate, and F(i) is the input.

[0184] Error back propagation is a process of dynamically adjusting weights and biases. By setting the loss function and using the optimizer to continuously reduce the loss function value. In this process, the optimizer uses the gradient descent method, and the loss function uses the mean square error function. Its expression is as follows:

[0185]

[0186] Among them, M is the number of data, y tar is the true value, y pre is the predicted value.

[0187] (5.4) The RBF layer defined in the present invention sets a number of neurons and sets the hyperparameter gamma. Initialize the RBF layer center: First, use the KMeans clustering algorithm to cluster the input feature variables; after the clustering is completed, the center points of the RBF layer are set to the cluster center points found by the KMeans algorithm.

[0188] (5.5) The compilation model of the present invention uses the Adam optimizer, and the loss function is the mean square error (MSE). The error between the model prediction result and the actual result is calculated through the loss function (MSE); the gradient of the loss function relative to each parameter is calculated. According to these gradients, the Adam optimizer will update the weights and biases of the model to gradually reduce the loss function value.

[0189] (5.6) The training of the present invention adopts the K-fold cross-validation method, such as Figure 5 As shown in the figure, K-fold cross validation uses a non-repeated sampling technique, which is characterized by the fact that each sample point has only one chance to be included in the training set or test set during each iteration, so the conclusions obtained are relatively stable. The specific steps are as follows:

[0190] (5.6.1) Divide the original data set (including input and output) into K equal parts (“folds”);

[0191] (5.6.2) From the K divided parts, select the first part as the test set and the rest as the training set;

[0192] (5.6.3) Train the model with the training set, and then verify the model output with the test set, and calculate the accuracy of the model, namely the mean square error (MSE) and the mean error (MEA);

[0193] (5.6.4) Repeat steps 2 and 3K times, using a different part as the test set each time;

[0194] (5.6.5) The average accuracy is taken as the final model accuracy.

[0195] (6) Predict the component in the link that will first reach the performance bottleneck. The specific implementation steps are as follows:

[0196] (6.1) Traverse each component of the call link: traverse each component in the link, that is, analyze each component on the link one by one.

[0197] (6.2) Based on deep learning prediction model: Use pre-trained deep learning model to predict the performance of each component. This model has been trained to predict the CPU resource usage of the component based on the input data.

[0198] (6.3) Setting TPS to predict component CPU resources and request time: The TPS is set based on the ratio of the call volumes of each component in the link at the same time. The set TPS is input into the deep learning model to predict the CPU resource usage and request processing time of each component under the current TPS.

[0199] (6.4) Compare component stress test capacity baseline values: Compare the predicted CPU resource usage and request processing time with the component stress test capacity baseline value. The stress test capacity baseline value refers to the maximum tolerable performance index of the component determined during the stress test.

[0200] (6.5) Analyze the component that reaches the capacity bottleneck first: The loop starts from the source node component and traverses the component link. Its predicted input TPS is the component Max TPS plus the k-th round multiplication factor *MaxTPS. Then a prediction is made. If the current component reaches the capacity bottleneck, this component is the weakest link in the entire link.

[0201] The technical solution of the present invention constructs a prediction model by constructing the relationship between the request volume of application components and key resources, and does not include machine learning training of capacity prediction and link relationship, and automatically predicts the component that first encounters performance bottleneck in the link. The technical solution of the present invention involves predicting the capacity and bottleneck components of components by using machine learning models by analyzing the link relationship between components.

[0202] The technical solution of the present invention constructs a prediction model for the request volume and key resource utilization rate of a component, and combines it with capacity stress testing data to analyze and predict the component that first reaches the performance bottleneck in the call link. It introduces a method for predicting the performance bottleneck of a machine learning training component, emphasizing the analysis of the relationship between capacity prediction and the link, and ultimately identifying the component that first reaches the performance bottleneck in the link.

[0203] An important innovation of the technical solution of the present invention is that by introducing capacity stress testing data, a prediction model for the request volume and key resource utilization of application components is constructed, and by analyzing the link relationship between components, the capacity of the link component is predicted, and based on this, the performance bottleneck components in the system link are detected.

[0204] The specific implementation scheme of this embodiment can refer to the relevant description in the above embodiment, which will not be repeated here.

[0205] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0206] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0207] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0208] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0209] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0210] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0211] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0212] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0213] The method, device, processor and computer-readable storage medium of the present invention for predicting component performance bottlenecks based on capacity and call chain statistics are adopted. Production applications introduce agent mode, and the application does not need to modify the source code, so that call chain indicator data can be collected at low cost. The deep learning neural network algorithm is introduced, which has a higher prediction accuracy. Based on the call chain topology dependency, the component that first reaches the performance bottleneck is predicted from the call chain analysis, so as to confirm that it is the performance bottleneck point of the system.

[0214] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method for predicting component performance bottlenecks based on capacity and call chain statistics, characterized in that: The method comprises the following steps: (1) Collect and store component call chain indicator data; (2) The capacity system obtains and stores component stress test indicator data; (3) Parse and store component call chains and indicator information; (4) Constructing the call link information from the source component to the destination component; (5) Determine the component TPS, time consumption, and CPU utilization relationship prediction model through neural network deep learning; (6) Compare the component stress test capacity baseline values ​​and analyze the component that first reaches the capacity bottleneck as the link bottleneck.

2. The method for predicting component performance bottlenecks based on capacity and call chain statistics according to claim 1, characterized in that: The step (1) is specifically as follows: The Agent collects component process operation indicators and writes the indicator data to the message queue Kafka.

3. The method for predicting component performance bottlenecks based on capacity and call chain statistics according to claim 1, characterized in that: The step (2) specifically comprises the following steps: (2.1) Obtain component performance stress test indicator data; (2.2) Store component performance stress test indicators in the capacity system; (2.3) Provide interface query to obtain performance indicator data of each component in the system dimension.

4. The method for predicting component performance bottlenecks based on capacity and call chain statistics according to claim 1, characterized in that: The step (3) specifically comprises the following steps: (3.1) Use Flink to perform real-time aggregation calculations on component call chains and monitoring indicator data stored in Kafka; (3.2) Perform aggregate statistics to generate various indicator data for component pairs; (3.3) The obtained indicator data is saved in the time series database InfluxDB.

5. The method for predicting component performance bottlenecks based on capacity and call chain statistics according to claim 1, characterized in that: The step (4) specifically comprises the following steps: (4.1) Query the time series database InfluxDB to obtain all stored component call link information data; (4.2) Add the queried call link information data to the graph data structure to build a complete component call link topology graph; (4.3) Traverse all nodes in the topology graph and define the source component and the destination component; (4.4) Use the depth-first search or breadth-first search algorithm to traverse the topology graph, find all paths from the source component to the destination component, and record the complete link information from the source component to the destination component; (4.5) Store the traversed link information into the specified storage medium.

6. The method for predicting component performance bottlenecks based on capacity and call chain statistics according to claim 1, characterized in that: The step (5) specifically comprises the following steps: (5.1) Perform data preprocessing; (5.2) Establish a neural network model; (5.3) Send the data information to the information layer for preprocessing, train and learn the features, and obtain the prediction information of CPU utilization; (5.4) Set a number of neurons, set the hyperparameter gamma, and initialize the RBF layer center; (5.5) Calculate the error between the model prediction result and the actual result through the loss function, and calculate the gradient of the loss function relative to each parameter; (5.6) K-fold cross validation method is used for training.

7. The method for predicting component performance bottlenecks based on capacity and call chain statistics according to claim 6, characterized in that: The step (5.1) specifically comprises the following steps: (5.1.1) Obtain component CPU utilization, TPS data, request duration data, and link call relationships; (5.1.2) Clean the data to remove outliers and missing values; (5.1.3) Standardize or normalize the data.

8. The method for predicting component performance bottlenecks based on capacity and call chain statistics according to claim 6, characterized in that: The step (5.6) specifically comprises the following steps: (5.6.1) Divide the original data set into K equal parts; (5.6.2) From the K divided parts, select the first part as the test set and the rest as the training set; (5.6.3) Use the training set to train the model, use the test set to verify the model output, and calculate the accuracy of the model; (5.6.4) Repeat (5.6.2) and (5.6.3) K times, using a different part as the test set each time; (5.6.5) The average accuracy is taken as the model accuracy.

9. The method for predicting component performance bottlenecks based on capacity and call chain statistics according to claim 1, characterized in that: The step (6) specifically comprises the following steps: (6.1) Traverse each component in the link; (6.2) Use pre-trained deep learning models to predict the performance of each component; (6.3) Set the TPS amount to predict component CPU resources and request duration; (6.4) Compare the predicted CPU resource usage and request processing time with the component's stress test capacity baseline value; (6.5) Analyze the component that reaches the capacity bottleneck first.

10. A device for predicting component performance bottlenecks based on capacity and call chain statistics, characterized in that: The device comprises: a processor configured to execute computer-executable instructions; A memory storing one or more computer executable instructions, wherein when the computer executable instructions are executed by the processor, the steps of the method for predicting and processing component performance bottlenecks based on capacity and call chain statistics as described in any one of claims 1 to 9 are implemented.

11. A processor for implementing predictive processing of component performance bottlenecks based on capacity and call chain statistics, characterized in that: The processor is configured to execute computer executable instructions. When the computer executable instructions are executed by the processor, the various steps of the method for predicting and processing component performance bottlenecks based on capacity and call chain statistics described in any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the method for predicting and processing component performance bottlenecks based on capacity and call chain statistics as described in any one of claims 1 to 9.

Images