A choke point detection and prediction method based on stream computing

Abstract

The application relates to the technical field of computer software, and discloses a jam detection and prediction method based on stream computing, which comprises the following steps: through a snapshot mechanism, a snapshot request is initiated to nodes in a network in sequence from a snapshot starting point according to data flow to obtain system snapshot information of the network and send the system snapshot information to a server; after the server receives the system snapshot information, whether each node in the network is congested is detected according to the system snapshot information, and whether each node is about to be congested is predicted; if it is detected that a node is congested, the congestion type of the node is judged, and corresponding solving measures are formulated according to the congestion type of the node; if it is predicted that a node is about to be congested, corresponding solving measures are formulated. The application improves the stability and efficiency of stream computing.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and specifically to a method for detecting and predicting congestion points based on streaming computing. Background Technology

[0002] In existing streaming computing technologies, the detection and prediction of node congestion are mainly achieved through the following methods: First, real-time data processing, which involves collecting and processing data from node sensors to monitor the current node status; second, analysis of historical data, recording congestion patterns collected by the sensors, establishing corresponding relationships, and determining the probability of current node congestion; and third, monitoring through performance indicators, analyzing key metrics of node data flow (such as latency, throughput, system availability, and resource utilization) to determine the current node's operating status and thus its congestion level.

[0003] Existing implementation methods have several drawbacks. First, data latency exists. In some cases, stream computing may not achieve zero latency, especially with large datasets or complex processing logic. This can lead to untimely real-time analysis results, impacting decision-making. Second, state management and fault tolerance are required. The aforementioned stream computing implementation methods often need to manage intermediate states and properly maintain state consistency and recover from failures. Third, system overhead is significant, requiring more resources to maintain data, resulting in higher hardware costs and energy consumption. Fourth, data skew is a problem. In distributed computing environments, uneven data distribution and processing steps can overload some nodes while lightly loading others, affecting the overall system performance and stability.

[0004] Therefore, there is an urgent need for a bottleneck detection and prediction method to address the problems of data latency, state management, system overhead, and data skew caused by existing technologies in streaming computing. Summary of the Invention

[0005] To address the issues of data latency, state management, system overhead, and data skew in streaming computing caused by existing technologies, this invention provides a congestion detection and prediction method based on streaming computing to improve the efficiency and reliability of streaming computing. First, it proposes a congestion type judgment method to determine the status of nodes in a chain reaction, identifying whether congestion is caused by a node reaching a bottleneck itself or by the influence of subsequent nodes. Second, it proposes a snapshot mechanism to take snapshots of the system at fixed or variable time intervals. For systems with large short-term fluctuations and high real-time requirements, it proposes a dynamic adjustment mechanism for the monitoring time window to further address the adaptability to different streaming systems, balance the contradiction between congestion detection and additional resource overhead, and flexibly detect congestion. Third, it proposes two prediction methods: one based on the rising rate of the data processing queue and the other based on a deep learning model using historical data, thereby achieving prediction of the node itself and performance monitoring of the entire system, respectively.

[0006] This invention is achieved through the following technical solution:

[0007] A blockage detection and prediction method based on streaming computation includes:

[0008] Through the snapshot mechanism, snapshot requests are initiated sequentially to nodes in the network from the snapshot start point according to the data flow to obtain the system snapshot information of the network and send it to the server.

[0009] After receiving the system snapshot information, the server detects whether each node in the network is congested based on the system snapshot information, and predicts whether each node is about to become congested.

[0010] If a node is detected to be congested, determine the type of congestion for that node and formulate corresponding solutions based on that type of congestion.

[0011] If congestion is predicted at a node, corresponding solutions will be developed.

[0012] As an optimization, the system snapshot information includes node snapshot information for each node in the network. The specific process of obtaining the system snapshot information of the network by initiating snapshot requests to nodes in the network sequentially according to the data flow from the snapshot start point is as follows:

[0013] A1. Initiate a snapshot request for the snapshot start point according to the snapshot command, and obtain the node snapshot information of the snapshot start point;

[0014] A2. Send the snapshot information of the node at the snapshot start point and the snapshot command to the intermediate node located downstream of the data according to the data flow direction;

[0015] A3. Initiate a snapshot request for the intermediate node according to the snapshot command to obtain the node snapshot information of the intermediate node;

[0016] A4. Send the node snapshot information of all nodes upstream of the intermediate node's data, as well as the node snapshot information and snapshot command of the intermediate node, to the adjacent intermediate node downstream of the intermediate node's data according to the data flow direction.

[0017] A5. Repeat A3-A4 until a snapshot request is made to the last node in the network along the data flow direction, and obtain the node snapshot information of the last node;

[0018] A6. Send the snapshot information of all nodes involved in this data flow direction as system snapshot information to the server.

[0019] As an optimization, the node snapshot information includes the data stream and sensor information of each node, and the data stream includes the data queue length information of that node.

[0020] As an optimization, the congestion types of the nodes include three categories:

[0021] The first type: congestion caused by node failure;

[0022] The second type: congestion caused by the node's computing power being less than the amount of data received;

[0023] The third type: Congestion at a node is caused by congestion at a downstream node.

[0024] As an optimization, let the already congested nodes be designated as bottlenecks. For a given bottleneck, determine its congestion type and formulate corresponding solutions based on that type. The specific process is as follows:

[0025] B1. Determine whether a current congestion point has collapsed through the heartbeat mechanism. If yes, proceed to B2; otherwise, proceed to B3.

[0026] B2. If the current congestion point is determined to be of type 1, restart the current congestion point or use a backup node;

[0027] B3. Determine whether the sensor information of the current congestion point is not lower than the first threshold. If yes, jump to B4; otherwise, jump to B5.

[0028] B4. Determine that the current congestion type of the congestion point is the second type, and expand the capacity of the current congestion point;

[0029] B5. Determine that the congestion type of the current congestion point is the third type, and find the congestion points downstream of all data of the current congestion point;

[0030] B6. Set the downstream bottleneck of the current bottleneck as the new current bottleneck, and then return to B2 to determine and resolve the type of the new current bottleneck.

[0031] As an optimization, the snapshot request interval is inversely proportional to the frequency of network congestion and inversely proportional to the network load fluctuation.

[0032] As an optimization, the length of the data processing queue at each node is used to predict whether congestion is imminent. The specific process is as follows:

[0033] C1. Obtain the data queue length information of a node obtained from a snapshot request;

[0034] C2. Determine if the length of the data queue exceeds the length threshold. If yes, jump to C3; otherwise, return to C1.

[0035] C3. Perform differential operations on the data queue length information of consecutive time points, including the data queue length information and the data queue length information obtained from the N adjacent snapshot requests adjacent to the data queue length information, according to the time order, to obtain the data processing queue length increment of adjacent time points;

[0036] C4. Obtain the rate of increase of the data processing queue length of the node based on the data processing queue length increment;

[0037] C5. Determine whether the rate of increase exceeds the second threshold. If so, issue a warning indicating that the node is about to become congested. Otherwise, return to C1.

[0038] As an optimization, in C4, the average of the data processing queue length increments at adjacent time points is used as the rate of increase of the data processing queue length for that node.

[0039] As an optimization, in C4, the slope of the increase in the data processing queue length at adjacent time points is used as the rate of increase of the data processing queue length at that node.

[0040] As an optimization, a deep computing model based on historical data is used to predict whether congestion is about to occur at each node. The specific process of establishing the deep computing model is as follows:

[0041] D1. Collect historical data of the network, including key network indicators;

[0042] D2. Convert the first intermediate data into a format suitable for input to a deep learning model to obtain the first intermediate data;

[0043] D3. Select the Long Short-Term Memory Network (LSTM) deep learning model as the deep learning model. Based on the characteristics of the historical data and the prediction target, define the input layer, output layer, hidden layer, activation function, optimization algorithm, and loss function of the deep learning model.

[0044] D4. Divide the historical data into a training set and a validation set. Use the training set to train the deep learning model. Continuously optimize the model parameters of the deep learning model through the backpropagation algorithm so that the deep learning model can better fit the historical data.

[0045] D5. Evaluate the performance of the deep learning model using the validation set, and adjust the hyperparameters of the deep learning model based on the performance of the validation set to improve the generalization ability of the deep learning model.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] This invention primarily proposes techniques for congestion detection and prediction in distributed streaming computing systems. It presents three different congestion types and a multi-strategy detection method (a heartbeat mechanism, sensor information detection, and a snapshot mechanism used in combination). The heartbeat mechanism can directly detect node failures, sensor information detection often only detects the second type of congestion—congestion caused by the node's own capacity limitations—while snapshots can detect congestion caused by subsequent nodes, directly identifying the source of congestion and thus formulating corresponding solutions. This helps to more accurately understand and handle congestion problems, find the source node, and solve the congestion problem at its source, thus resolving system congestion more quickly. The snapshot mechanism allows for a one-time acquisition of the entire system status, and the snapshot time interval can be dynamically adjusted as needed to adapt to different system loads and congestion conditions. This flexibility improves system efficiency and stability and is highly helpful for quickly determining the congestion situation and type.

[0048] Two congestion prediction methods are provided: one based on data from a single node, and the other based on historical data and a deep learning model. These methods can help identify potential congestion problems in advance, allowing for timely measures to prevent congestion and improving the stability and efficiency of streaming computing. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0050] Figure 1 A block diagram illustrating the use of a snapshot mechanism to detect node congestion;

[0051] Figure 2 A block diagram illustrating the use of a heartbeat mechanism to detect node congestion;

[0052] Figure 3 A block diagram illustrating the use of sensor information to detect node congestion.

[0053] Figure 4 Flowchart for bottleneck type detection;

[0054] Figure 5 This is a flowchart of sensor-based congestion prediction.

[0055] Figure 6 This is a flowchart of a deep learning-based congestion prediction process. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0057] This embodiment 1 provides a blockage detection and prediction method based on streaming computation, including:

[0058] Through the snapshot mechanism, snapshot requests are initiated sequentially to nodes in the network from the snapshot start point according to the data flow to obtain the system snapshot information of the network and send it to the server.

[0059] Let a certain node be the starting point of the snapshot, such as Figure 1 As shown, in this embodiment, the node that generates the data stream is taken as the starting point of the snapshot, and the nodes in the network through which the data stream passes are taken as intermediate nodes.

[0060] In this embodiment, the system snapshot information includes node snapshot information for each node in the network. The node snapshot information includes the data stream and sensor information for each node, and the data stream includes the data queue length information for that node. That is, the data in a node's data queue includes not only the data information it generates itself, but also the length information of that data queue.

[0061] The specific process of obtaining system snapshot information of the network by initiating snapshot requests to nodes in the network sequentially according to the data flow from the snapshot start point is as follows:

[0062] A1. Initiate a snapshot request for the snapshot start point according to the snapshot command, and obtain the node snapshot information of the snapshot start point;

[0063] A2. Send the snapshot information of the node at the snapshot start point and the snapshot command to the intermediate node located downstream of the data according to the data flow direction;

[0064] A3. Initiate a snapshot request for the intermediate node according to the snapshot command to obtain the node snapshot information of the intermediate node;

[0065] A4. Send the node snapshot information of all nodes upstream of the intermediate node's data, as well as the node snapshot information and snapshot command of the intermediate node, to the adjacent intermediate node downstream of the intermediate node's data according to the data flow direction.

[0066] A5. Repeat A3-A4 until a snapshot request is made to the last node in the network along the data flow direction, and obtain the node snapshot information of the last node;

[0067] A6. Send the snapshot information of all nodes involved in this data flow direction as system snapshot information to the server.

[0068] After a snapshot request is initiated for a certain node, the snapshot information of that node is obtained. Each node located downstream of the data in the network receives the snapshot information and snapshot command from the upstream node of the data, and transmits the snapshot information from the upstream node and the snapshot information of that node to the downstream node of the data. The node information of all nodes through which the data stream flows from the last node in the data stream direction is the system snapshot information.

[0069] After receiving the system snapshot information, the server detects whether each node in the network is congested based on the system snapshot information, and predicts whether each node is about to become congested.

[0070] If a node is detected to be congested, determine the type of congestion for that node and formulate corresponding solutions based on that type of congestion.

[0071] If congestion is predicted at a node, corresponding solutions will be developed.

[0072] The snapshot mechanism detects node congestion primarily by checking the data queue length information of the nodes to determine their congestion status.

[0073] In the classification of congestion types, there are three main types of congestion.

[0074] The first type: congestion caused by node failure;

[0075] The most basic way to determine this type of congestion is through the platform (server) itself, such as... Figure 2 As shown, each node dynamically interacts with nodes that have scheduling or synchronization functions. Figure 2The system dynamically interacts with the server in the middle, and each node sends heartbeat information to it (the node with scheduling or synchronization function) at a fixed frequency to ensure that the platform can detect the failure of the node itself as soon as possible and make corresponding responses.

[0076] The second type: congestion caused by the node's computing power being less than the amount of data received;

[0077] like Figure 3 As shown, congestion is caused by the limitation of the node's own computing power. The node receives a large amount of data in a short period of time and cannot process it in time, resulting in congestion. This type of congestion is mainly detected by setting a threshold. The sensor collects node information in real time, i.e., sensor information (including output / input, memory occupancy, throughput, computing resource occupancy, etc.). When the first threshold is reached, an alarm is triggered, thereby judging the congestion situation.

[0078] The third type: Congestion at a node is caused by congestion at a downstream node.

[0079] The third type is congestion at the current node caused by congestion at the downstream node (the node downstream of the data). Downstream node congestion also has three causes: node crash, node computing capacity bottleneck, and congestion at the downstream node. The type of congestion needs to be determined based on the current node's basic information (input / output status, memory occupancy, throughput, etc.), primarily whether its output buffer throughput has reached a threshold, and the status of the downstream nodes.

[0080] a) If the current node's computing power has not reached its bottleneck, the throughput of the output buffer pool (the data before it is processed and output is stored in the output buffer pool) reaches the threshold, and the successor node crashes (i.e., the successor node belongs to the first type of congestion), then the current node is judged to be in the third type of congestion.

[0081] b) If the current node's computing power has not reached its bottleneck, the output buffer pool throughput has reached its threshold, and the subsequent node's computing resources are limited (i.e., the subsequent node belongs to the second type of congestion), then the current node is judged to be in the third type of congestion.

[0082] c) If the current node's computing power has not reached its bottleneck, the output buffer pool throughput has reached its threshold, and the successor node is in the third type of congestion (i.e., the congestion of the successor node is due to the congestion of the downstream node of the data of the successor node (i.e., the successor node of the successor node)), then the current node is determined to be in the third type of congestion.

[0083] The first and second types of congestion points can be treated as the original congestion points and restored. The third type requires tracing back to the nodes of the first or second type of congestion that were initially congested (i.e., finding the node that was initially congested along the upstream direction of the data flow and belonged to the first or second type of congestion). The method to resolve the congestion also starts from the original congestion point.

[0084] Therefore, there are three main types of methods for detecting bottlenecks. The first type is to send a signal to a fixed central node or scheduling node (…). Figure 2 This method involves sending heartbeat messages to the server in the middle, which can quickly determine if a node has crashed or is down, and can quickly detect the first type of congestion. The first method is highly dependent on the central node or scheduling node, and also has relatively high requirements for network conditions. Failure of the central node or network can easily lead to incorrect judgments; therefore, the use of this method has high limitations.

[0085] The second method involves deploying sensors. This method involves placing a sensor on each node to collect information about the current node and determine whether the node is congested based on a predefined threshold. However, this second method cannot identify the first and third types of congestion, nor can it determine whether congestion is caused by limited resources based on information from other nodes. Furthermore, if a node crashes, it cannot collect congestion information immediately. Therefore, this method has significant limitations.

[0086] The third method is to use a snapshot mechanism, starting from a certain node as the snapshot starting point, and initiating snapshot requests sequentially according to the computing structure. Finally, the snapshot information of the entire system is collected, thereby obtaining the status of the entire system (network) at once, which can quickly determine the congestion situation and congestion type.

[0087] The third method can obtain global resource utilization and congestion information, and with appropriate strategies, can achieve rapid recovery and congestion resolution. The snapshot time interval can be dynamically adjusted as needed; that is, the snapshot request interval is inversely proportional to the frequency of network congestion and inversely proportional to network load fluctuations. If congestion occurs frequently or system load fluctuates significantly, the time interval can be shortened to detect congestion more quickly; if congestion occurs infrequently or system load is stable, the time interval can be lengthened to reduce resource overhead.

[0088] Specifically, the Poisson distribution from the probability model in statistical methods can be used for prediction. The general steps are as follows:

[0089] 1. Define the event: First, define congestion as the event to be monitored;

[0090] 2. Set initial time interval: Set an initial time interval;

[0091] 3. Data collection: Collect data on system congestion within the initial time interval (specifically, the number of times congestion occurred within a time interval);

[0092] 4. Calculate the Poisson parameter: Calculate the parameter λ of the Poisson distribution using the collected data. λ represents the average number of events occurring per unit time.

[0093] Let's define a fixed interval t. We then count the number of times congestion occurs within that fixed interval t. Assuming the number of congestion occurrences within that fixed interval t is k, the probability of congestion is: α = k / t. Since the unit time is a fixed interval t, λ = α * 1. Using λ as the constant of the Poisson distribution, ideally, the number of congestion occurrences x per t should follow a Poisson distribution P(x; λ). This allows us to list the probabilities of x = 0, 1, 2, 3... The specific formula is:

[0094]

[0095] 5. Predicting future events: Using the Poisson distribution to predict the probability of an event occurring within a future time interval;

[0096] Set a confidence interval, for example, X=5, with a probability of 95%. If the probability exceeds the set threshold of 90%, then it is determined that 5 congestion events will occur within the next fixed interval t.

[0097] 6. Adjust the time interval: Adjust the time interval based on the predicted probability. If the predicted probability is high, it means that congestion occurs frequently, and the time interval should be shortened to detect congestion more quickly; conversely, if the predicted probability is low, it means that congestion occurs less frequently, and the monitoring interval can be appropriately extended to reduce resource consumption.

[0098] 7. Feedback loop: The actual congestion situation is compared with the predicted results, and the Poisson parameter λ is further adjusted based on the comparison results in order to more accurately predict future congestion.

[0099] The number of congestion events occurring within a fixed interval t is counted and compared with the actual predicted number. The specific number needs to be adjusted according to the actual situation. The following are some main approaches to consider:

[0100] 1. Recalculate the Poisson parameter λ based on the actual number of congestion events;

[0101] 2. Re-divide the time interval t, recalculate λ and the corresponding predicted value, and compare whether it is more accurate;

[0102] 3. Are there other factors contributing to congestion, such as a significantly higher probability of congestion during a certain time period compared to other times? In such cases, it is necessary to divide the congestion into different time periods for separate calculations.

[0103] If the predicted number of congestion events is the same as the actual number of congestion events, then this time interval can be used as the time interval for the snapshot mechanism.

[0104] The time intervals for each period can be different.

[0105] After obtaining the congestion status of all nodes in the system through the snapshot mechanism, the process of determining the congestion type of the already congested nodes (blockage points) is as follows:

[0106] like Figure 4 As shown, let the congested nodes be called bottlenecks. For a given bottleneck, the process of determining the congestion type of the bottleneck and formulating corresponding solutions based on the congestion type is as follows:

[0107] B1. Determine if a current congestion point has collapsed. If yes, jump to B2; otherwise, jump to B3.

[0108] B2. If the current congestion point is determined to be of type 1, restart the current congestion point or use a backup node;

[0109] B3. Determine whether the sensor information of the current congestion point is not lower than the first threshold. If yes, jump to B4; otherwise, jump to B5.

[0110] B4. Determine that the current congestion type of the congestion point is the second type, and expand the capacity of the current congestion point;

[0111] B5. Determine that the congestion type of the current congestion point is the third type, and find the congestion points downstream of all data of the current congestion point;

[0112] B6. Set the downstream bottleneck of the current bottleneck as the new current bottleneck, and then return to B2 to determine and resolve the type of the new current bottleneck.

[0113] like Figure 1-3 As shown, red nodes represent congestion points detected by the snapshot mechanism, green nodes represent non-congested nodes, and yellow nodes represent nodes about to become congested. Nodes C and D are congestion points. Regarding the congestion type of node C, we first determine if node C belongs to the first or second category. If it doesn't, then node C belongs to the third category. In this case, we need to examine the downstream nodes of node C (nodes B and D). Node B is not congested, but node D is. Therefore, we need to examine the congestion type of node D. The downstream node of node D (i.e., the successor node of node D) is node E. E is not congested. Therefore, the congestion type of node D can only belong to either the first or second category. Thus, resolving the congestion problem of node D will solve the congestion problem of node C.

[0114] Congestion Prediction:

[0115] There are two main types of predictions for traffic congestion.

[0116] This invention employs a method for predicting congestion based on data from a single node. During congestion detection, sensors are deployed to monitor node resource status (including output / input, memory occupancy, throughput, and computational resource utilization) at fixed time intervals or according to actual needs. The system records the data processing queue length at the current time point and stores it in the data queue. If the data queue length exceeds a preset threshold, the next step of rate of increase detection is triggered. Based on real-time data collection, the invention performs differential operations on the data queue lengths (data queue length information) at consecutive time points to obtain the increments between adjacent time points. The average or slope of these increments is used as the rate of increase in the queue length. If the rate of increase exceeds a preset threshold, an early warning is issued, indicating that system congestion may be imminent. A timely request is sent to the scheduling node / central node to expand the corresponding computational resources and prevent congestion.

[0117] That is, predicting whether each node will soon experience congestion by using the length of the node's data processing queue, such as... Figure 5 As shown, the specific process is as follows:

[0118] C1. Obtain the data queue length information of a node obtained from a snapshot request;

[0119] C2. Determine if the length of the data queue exceeds the length threshold. If yes, jump to C3; otherwise, return to C1.

[0120] C3. Perform differential operations on the data queue length information of consecutive time points, including the data queue length information and the data queue length information obtained from the N adjacent snapshot requests adjacent to the data queue length information, according to the time order, to obtain the data processing queue length increment of adjacent time points;

[0121] Let A be the length of the data queue of a node obtained by a snapshot request (the m-th snapshot request). m Find the data queue length information A obtained from the previous N adjacent snapshot requests. m-1 A m-2 A m-N Let m be the number of snapshot requests, and m ≥ N. Then, performing the difference operation sequentially yields:

[0122] ΔL1=A m -A m-1 ΔL2=A m-1 -A m-2 ΔL3=A m-2 -Am-3 ,···、ΔL n =A m-(N-1) -A m-N ; ΔL1, ΔL2, ..., ΔL n This sequence represents the increment of the data processing queue length at adjacent time points.

[0123] C4. Obtain the rate of increase of the data processing queue length of this node based on the data processing queue length increment; here, the rate of increase is... Figure 5 The rate of length growth in the middle.

[0124] Specifically, the rate of increase of the data processing queue length at adjacent time points is calculated as the slope of the increase in the data processing queue length at that node.

[0125] The specific formula for the slope k of the data processing queue length increment between adjacent time points is:

[0126] Let the time of a snapshot request (the m-th snapshot request) be T. m The times of the first N adjacent snapshot requests are T. m-1 T m-2 、···、T m-N Let m be the number of snapshot requests, and m ≥ N. Then, performing the difference operation sequentially yields:

[0127] ΔT1=T m -T m-1 ΔT2=T m-1 -T m-2 ΔT3=T m-2 -T m-3 ,···、ΔT n =T m-(N-1) -T m-N ; ΔT1, ΔT2, ..., ΔT n This sequence represents the time difference between adjacent snapshots.

[0128] Then the data growth slope for each snapshot can be calculated:

[0129] k1, k2, k3, ..., k n Let be the slope of this sequence.

[0130] By calculating the average increase in slope over a period of time, the rate of increase of the slope can be determined. The specific formula is as follows:

[0131]

[0132] If the slope is used as the rate of ascent, then... If the threshold is exceeded, it is determined that there may be a risk of congestion.

[0133] C5. Determine whether the rate of increase exceeds the second threshold. If so, it indicates that the node is about to become congested. Predict the congestion time and issue a scheduling request. Otherwise, return to C1.

[0134] That is when When the value exceeds the set average threshold, it indicates that the node is about to become congested; or when k exceeds the set slope threshold, it indicates that the node is about to become congested.

[0135] The specific calculation process for the predicted congestion time is as follows:

[0136] Using an iterative method, estimate the increase in slope for the next time period:

[0137]

[0138] Similarly, the queue length after the next time interval can be estimated:

[0139] A m+1 =A m +k0*ΔT.

[0140] Where ΔT is the preset snapshot interval, the two iterative methods described above are used repeatedly until node congestion occurs, and the number of iterations is recorded as r. Then, the difference between the estimated time of node congestion and the time of the m-th snapshot request is:

[0141] T r =r*ΔT.

[0142] The second method is to establish a corresponding deep computing model based on historical data to determine the resource usage of the entire system and whether any nodes are about to enter a congested or idle state, and to make timely adjustments to the strategy accordingly.

[0143] like Figure 2 As shown, a deep computing model based on historical data is used to predict whether congestion is about to occur at each node. The specific process of establishing the deep computing model is as follows:

[0144] D1. Collect historical data of the network, including key network indicators;

[0145] D2. Convert the historical data into a format suitable for input to a deep learning model to obtain the first intermediate data;

[0146] D3. Select the Long Short-Term Memory Network (LSTM) deep learning model as the deep learning model. Based on the characteristics of the historical data and the prediction target, define the input layer, output layer, hidden layer, activation function, optimization algorithm, and loss function of the deep learning model.

[0147] The characteristics of historical data taken into consideration typically include: model traffic (data throughput at different time periods), network conditions (including stability and congestion), node congestion, congestion caused by historical tasks, node idleness, and the proportion of computing tasks undertaken (the proportion of larger tasks to lighter tasks), etc.

[0148] The prediction objective is to identify which nodes are about to experience congestion, as well as the probability and timing of such congestion.

[0149] The input layer, output layer, hidden layer, activation function, optimization algorithm, and loss function of the deep learning model are defined as follows:

[0150] 1. Input layer: Contains all the features used for prediction and determines the number of input neurons.

[0151] 2. Hidden layer: Because it contains multiple prediction targets, the hidden layer is usually chosen by dividing the sum of the input layer and the output layer by a certain constant and taking the integer part.

[0152] 3. Activation function: A linear activation function can be selected for prediction of regression problems (node ​​traffic).

[0153] 4. Output layer: Depends on the target to be predicted, including traffic, node congestion, etc.

[0154] 5. Loss function: Mean squared error or root mean squared error can be used.

[0155] 6. Optimizers: Gradient descent, Adam, etc.

[0156] 7. Evaluation metrics: accuracy, F1 score, etc.

[0157] D4. Divide the first intermediate data into a training set and a validation set. Use the training set to train the deep learning model. Continuously optimize the model parameters of the deep learning model through the backpropagation algorithm so that the deep learning model can better fit the historical data.

[0158] D5. Evaluate the performance of the deep learning model using the validation set, and adjust the hyperparameters of the deep learning model based on the performance on the validation set to improve the generalization ability of the deep learning model. This is an existing technique, and adjustments can be made using tools and libraries such as Hyperopt, Optuna, and AutoML.

[0159] In the implementation of the second method, node congestion prediction is mainly achieved through the following steps:

[0160] 1. Data Preparation. Collect historical data, including key system metrics such as node load, communication status, and data processing queue length. Convert the historical data into a format suitable for deep learning model input, such as time series data or multi-dimensional feature data.

[0161] 2. Construct a deep learning model. Select an appropriate deep learning model, a Long Short-Term Memory (LSTM) network, to predict the changing trend of system performance. Based on the characteristics of historical data and the prediction objective, define the model's input and output layers, hidden layer structure, activation functions, optimization algorithms, and loss functions.

[0162] 3. Model Training. The transformed historical data is divided into training and validation sets. The deep learning model is trained using the training set, and the model parameters are continuously optimized through backpropagation to enable the model to better fit the historical data. The model's performance is evaluated using the validation set, and the model's hyperparameters are adjusted based on the validation set performance to improve the model's generalization ability.

[0163] 4. Model Prediction. A trained deep learning model is used to predict future system performance. The latest system state data is input into the model to obtain prediction results, i.e., the future trend of system performance. Based on the model prediction results, it is determined whether the system is about to experience congestion or performance degradation, and an early warning is issued.

[0164] 5. Resource Allocation Adjustment. Based on the model's predictions, corresponding resource allocation adjustment measures are taken. Congestion can be alleviated by reallocating load, adjusting communication routes, and increasing node resources.

[0165] 6. Monitoring and Feedback. Continuously monitor system performance and resource utilization, collect model prediction results and actual congestion conditions. Adjust resources accordingly based on the actual situation, and continuously optimize the deep learning model to improve the accuracy and timeliness of predictions.

[0166] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting and predicting congestion points based on streaming computation, characterized in that, include: Through the snapshot mechanism, snapshot requests are initiated sequentially to nodes in the network from the snapshot start point according to the data flow to obtain the system snapshot information of the network and send it to the server. After receiving the system snapshot information, the server detects whether each node in the network is congested based on the system snapshot information, and predicts whether each node is about to become congested. If a node is detected to be congested, determine the type of congestion for that node and formulate corresponding solutions based on that type of congestion. If it is predicted that a node will become congested and the congestion time is predicted, then corresponding solutions will be developed. The specific process for predicting the congestion time of nodes is as follows: E1, Let the data queue length information of the node obtained from the m-th snapshot request be... Find the data queue length information obtained from the previous N adjacent snapshot requests. , ... , m is the number of snapshot requests, and Then, performing the difference operation sequentially results in: , , , ···、 ; , ... This sequence represents the increment of the data processing queue length at adjacent time points; E2, Let the time of the m-th snapshot request be... The times of the first N adjacent snapshot requests are respectively , ... , m is the number of snapshot requests, and Then, performing the difference operation sequentially results in: , , , ···、 ; , ... This sequence represents the time difference between adjacent snapshots; E3. Calculate the data growth slope for each snapshot: , , , ···, ; ... This is the slope of the sequence; E4. By calculating the average increase in slope over a period of time, the rate of increase of the slope can be determined. The specific formula is as follows: ； E5, if If the slope exceeds the preset threshold, it is determined that there is a risk of congestion and the process jumps to E6; otherwise, it returns to E1. E6. Let r=1; E7. Using an iterative method, estimate the increase in the slope of the next time period after the m-th snapshot request. : ； E8. Estimate the queue length after the next time interval. : ,in The preset snapshot interval; E8. Determine if the node will be congested in the next time interval. If so, let r = r + 1. , Then return E7; otherwise, estimate the time when the node becomes congested. The difference between the time when the node becomes congested and the time of the m-th snapshot request is: .

2. The method for detecting and predicting congestion points based on streaming computation according to claim 1, characterized in that, The system snapshot information includes node snapshot information for each node in the network. The specific process of obtaining the system snapshot information of the network by initiating snapshot requests to the nodes in the network sequentially according to the data flow from the snapshot start point is as follows: A1. Initiate a snapshot request for the snapshot start point according to the snapshot command, and obtain the node snapshot information of the snapshot start point; A2. Send the snapshot information of the node at the snapshot start point and the snapshot command to the intermediate node located downstream of the data according to the data flow direction; A3. Initiate a snapshot request for the intermediate node according to the snapshot command to obtain the node snapshot information of the intermediate node; A4. Send the node snapshot information of all nodes upstream of the intermediate node's data, as well as the node snapshot information and snapshot command of the intermediate node, to the adjacent intermediate node downstream of the intermediate node's data according to the data flow direction. A5. Repeat A3-A4 until a snapshot request is made to the last node in the network along the data flow direction, and obtain the node snapshot information of the last node; A6. Send the snapshot information of all nodes involved in this data flow direction as system snapshot information to the server.

3. The method for detecting and predicting congestion points based on streaming computation according to claim 2, characterized in that, The node snapshot information includes the data stream and sensor information of each node, and the data stream includes the data queue length information of that node.

4. The blockage detection and prediction method based on streaming computation according to claim 3, characterized in that, The congestion types of the nodes include three categories: The first type: congestion caused by node failure; The second type: congestion caused by the node's computing power being less than the amount of data received; The third type: Congestion at a node is caused by congestion at a downstream node.

5. The method for detecting and predicting congestion points based on streaming computation according to claim 4, characterized in that, Let the congested nodes be designated as bottlenecks. For a given bottleneck, determine its congestion type and formulate corresponding solutions based on that type. The specific process is as follows: B1. Determine whether a current congestion point has collapsed through the heartbeat mechanism. If yes, proceed to B2; otherwise, proceed to B3. B2. If the current congestion point is determined to be of type 1, restart the current congestion point or use a backup node; B3. Determine whether the sensor information of the current congestion point is not lower than the first threshold. If yes, jump to B4; otherwise, jump to B5. B4. Determine that the current congestion type of the congestion point is the second type, and expand the capacity of the current congestion point; B5. Determine that the congestion type of the current congestion point is the third type, and find the congestion points downstream of all data of the current congestion point; B6. Set the downstream bottleneck of the current bottleneck as the new current bottleneck, and then return to B2 to determine and resolve the type of the new current bottleneck.

6. The method for detecting and predicting congestion points based on streaming computation according to claim 1, characterized in that, The snapshot request interval is inversely proportional to the frequency of network congestion and inversely proportional to the network load fluctuation.

7. The method for detecting and predicting congestion points based on streaming computation according to claim 3, characterized in that, Predicting whether a node is about to experience congestion by using the length of the node's data processing queue, the specific process is as follows: C1. Obtain the data queue length information of a node obtained from a snapshot request; C2. Determine if the length of the data queue exceeds the length threshold. If yes, jump to C3; otherwise, return to C1. C3. Perform differential operations on the data queue length information of consecutive time points, including the data queue length information and the data queue length information obtained from the N adjacent snapshot requests adjacent to the data queue length information, according to the time order, to obtain the data processing queue length increment of adjacent time points; C4. Obtain the rate of increase of the data processing queue length of the node based on the data processing queue length increment; C5. Determine whether the rate of increase exceeds the second threshold. If so, it indicates that the node is about to become congested. Predict the congestion time and issue a scheduling request. Otherwise, return to C1.

8. The method for detecting and predicting congestion points based on streaming computation according to claim 7, characterized in that, In C4, the rate of increase of the data processing queue length at adjacent time points is calculated as the slope of the increase in the data processing queue length at that node.

9. The blockage detection and prediction method based on streaming computation according to claim 3, characterized in that, A deep computing model based on historical data is used to predict whether congestion is about to occur at each node. The specific process of building the deep computing model is as follows: D1. Collect historical data of the network, including key network indicators; D2. Convert the historical data into a format suitable for input to a deep learning model to obtain the first intermediate data; D3. Select the Long Short-Term Memory Network (LSTM) deep learning model as the deep learning model. Based on the characteristics of the historical data and the prediction target, define the input layer, output layer, hidden layer, activation function, optimization algorithm, and loss function of the deep learning model. D4. Divide the first intermediate data into a training set and a validation set. Use the training set to train the deep learning model. Continuously optimize the model parameters of the deep learning model through the backpropagation algorithm so that the deep learning model can better fit the historical data. D5. Evaluate the performance of the deep learning model using the validation set, and adjust the hyperparameters of the deep learning model based on the performance of the validation set to improve the generalization ability of the deep learning model.

Images