Resource adjustment method and device and electronic equipment

Abstract

The invention discloses a resource adjustment method and device and electronic equipment, relates to the technical field of data processing, and is used for improving the resource adjustment accuracy when a back pressure problem occurs, and the method comprises the steps: firstly, obtaining a network resource data set detected by a network monitoring system in real time; then, on the basis of the network resource data set, back pressure prediction is carried out by adopting the target back pressure prediction model, and a real-time back pressure prediction value is obtained; and finally, in response to the fact that the real-time back pressure prediction value is greater than a preset back pressure threshold value, calculating a target parallelism degree and a target buffer area size based on the real-time back pressure prediction value, and performing resource adjustment based on the target parallelism degree and the target buffer area size. Through the method, whether the system has a back pressure problem or not can be predicted in advance, resource adjustment is performed in advance, the back pressure problem is avoided so as to ensure the performance and stability of the system, and the to-be-adjusted resource parameters can be accurately calculated based on the back pressure prediction value without wasting resources.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a resource adjustment method, device, and electronic device. Background Art

[0002] Flink is a distributed stream processing framework widely used in various fields, including real-time data analysis and event-driven applications. However, Flink's streaming computing often faces backpressure during execution.

[0003] Specifically, the back pressure problem is a dynamic rate-limiting mechanism in Flink's streaming computing caused by the data processing rate of downstream task nodes being lower than the data sending rate of upstream task nodes. This in turn causes data that cannot be processed in time to accumulate in downstream task nodes, increasing data processing latency and seriously affecting the processing performance of Flink's streaming computing.

[0004] In related technologies, when solving back pressure problems, operations and maintenance personnel can usually adjust the task parameters of Flink streaming computing to achieve a balance in the amount of data processed by upstream and downstream task nodes. However, this method often relies on the experience of operations and maintenance personnel, is time-consuming and labor-intensive, and cannot be adjusted accurately. In addition, related technologies can also use static configuration to configure parameters such as the data processing rate of each task node in Flink streaming computing. However, in actual application scenarios, data traffic usually shows a dynamic trend. The static configuration method cannot be dynamically adjusted according to changes in data traffic, and back pressure problems will still occur, or there will be problems of wasted resources. Summary of the Invention

[0005] Embodiments of the present application provide a resource adjustment method, apparatus, and electronic device to address the back pressure problem caused by the inability to accurately configure task parameters in Flink tasks in related technologies. The present application can accurately predict back pressure problems and adjust system resources in a timely manner before back pressure problems occur, thereby effectively avoiding the impact of back pressure problems on Flink tasks.

[0006] In a first aspect, the present application provides a resource adjustment method, the method comprising:

[0007] Obtain a network resource data set detected in real time by a network monitoring system; the network resource data set includes at least: CPU usage, memory usage, network bandwidth usage, buffer occupancy, network latency, and data throughput;

[0008] Based on the network resource dataset, the target back pressure prediction model is used to predict back pressure and obtain real-time back pressure prediction values.

[0009] In response to the real-time back pressure prediction value being greater than the preset back pressure threshold, the target parallelism and the target buffer size are calculated based on the real-time back pressure prediction value, and resources are adjusted based on the target parallelism and the target buffer size; wherein, the preset back pressure threshold is used to determine whether a back pressure problem currently occurs, the target parallelism represents the number of parallel instances that the same operator is split into to process data simultaneously, and the target buffer size represents the capacity used to temporarily store data sent by the upstream task node.

[0010] Through the above method, the future load pressure can be quantified through the back pressure prediction value, so that resource adjustment can be transformed from passive to active adaptation. Based on the back pressure prediction value, the severity of the back pressure problem in the future can be reflected. Therefore, the calculation of the target parallelism and target buffer size based on the back pressure prediction value can adapt to the current dynamic changes in the system's traffic, and can adaptively adjust resources according to the severity of the back pressure problem, so as to avoid the back pressure problem in advance and avoid excessive waste of pre-allocated resources.

[0011] In an optional implementation, calculating the target parallelism based on the real-time back pressure prediction value includes:

[0012] Get the current real-time parallelism and real-time back pressure prediction values;

[0013] The target parallelism is calculated based on the real-time parallelism, the real-time back pressure prediction value, the preset parallelism adjustment coefficient, and the preset parallelism adjustment threshold.

[0014] In an optional implementation, calculating the target buffer size based on the real-time back pressure prediction value includes:

[0015] Get the current real-time buffer size and real-time back pressure prediction value;

[0016] The target buffer size is calculated based on the real-time buffer size, the real-time back pressure prediction value, the preset buffer adjustment coefficient, and the preset buffer adjustment threshold.

[0017] In an optional embodiment, when calculating the target parallelism based on the real-time parallelism, the real-time back pressure prediction value, the preset parallelism adjustment coefficient, and the preset parallelism adjustment threshold, the following calculation formula is used:

[0018] P(t+1)=P(t)×(1+α×(B prob (t)-T thresh ))

[0019] Among them, P(t+1) represents the target parallelism;

[0020] P(t) represents the real-time parallelism;

[0021] α represents the parallelism adjustment coefficient;

[0022] B prob (t) represents the real-time back pressure prediction value;

[0023] T thresh Indicates the preset parallelism adjustment threshold.

[0024] In an optional embodiment, when calculating the target buffer size based on the real-time buffer size, the real-time back pressure prediction value, the preset buffer adjustment coefficient, and the preset buffer adjustment threshold, the following calculation formula is used:

[0025] B(t+1)=B(t)(1+β×(B prob (t)-L thresh ))

[0026] Among them, B(t+1) represents the target buffer size;

[0027] B(t) represents the real-time buffer size;

[0028] β represents the preset buffer adjustment coefficient;

[0029] B prob (t) represents the real-time back pressure prediction value;

[0030] L thresh Indicates the preset buffer adjustment threshold.

[0031] In an optional implementation, the target back pressure prediction model is trained in the following manner:

[0032] Acquire historical network resource data and form a historical network resource data set;

[0033] Input the historical network resource dataset into the back pressure prediction model to be trained to obtain the historical back pressure prediction value;

[0034] Based on the difference between the historical back pressure prediction value and the historical back pressure probability, the parameters of the back pressure prediction model to be trained are optimized through the adaptive moment estimation method to obtain the target back pressure prediction model.

[0035] In a second aspect, the present application provides a resource adjustment device, the device comprising:

[0036] An acquisition module is used to acquire a network resource data set detected in real time by a network monitoring system; the network resource data set includes at least: CPU usage, memory usage, network bandwidth usage, buffer occupancy, network delay, and data throughput;

[0037] A prediction module is used to perform back pressure prediction based on a network resource dataset using a target back pressure prediction model to obtain a real-time back pressure prediction value.

[0038] A processing module is used to calculate the target parallelism and target buffer size based on the back pressure threshold in response to a real-time back pressure prediction value being greater than a preset back pressure threshold, and to adjust resources based on the target parallelism and target buffer size; wherein the preset back pressure threshold is used to determine whether a back pressure problem currently occurs, the target parallelism represents the number of parallel instances of the same operator being split into to process data simultaneously, and the target buffer size represents the capacity for temporarily storing data sent by upstream task nodes.

[0039] In an optional implementation, when calculating the target parallelism based on the real-time back pressure prediction value, the processing module is specifically configured to:

[0040] Get the current real-time parallelism and real-time back pressure prediction values;

[0041] The target parallelism is calculated based on the real-time parallelism, the real-time back pressure prediction value, the preset parallelism adjustment coefficient, and the preset parallelism adjustment threshold.

[0042] In an optional implementation, when calculating the target buffer size based on the back pressure prediction value, the processing module is specifically configured to:

[0043] Get the current real-time buffer size and real-time back pressure prediction value;

[0044] The target buffer size is calculated based on the real-time buffer size, the real-time back pressure prediction value, the preset buffer adjustment coefficient, and the preset buffer adjustment threshold.

[0045] In an optional embodiment, when calculating the target parallelism based on the real-time parallelism, the real-time back pressure prediction value, the preset parallelism adjustment coefficient, and the preset parallelism adjustment threshold, the processing module specifically uses the following calculation formula to calculate the target parallelism:

[0046] P(t+1)=P(t)×(1+α×(B prob (t)-T thresh ))

[0047] Among them, P(t+1) represents the target parallelism;

[0048] P(t) represents the real-time parallelism;

[0049] α represents the parallelism adjustment coefficient;

[0050] B prob (t) represents the real-time back pressure prediction value;

[0051] Tthresh Indicates the preset parallelism adjustment threshold.

[0052] In an optional embodiment, when the target buffer size is calculated based on the real-time buffer size, the real-time back pressure prediction value, the preset buffer adjustment coefficient, and the preset buffer adjustment threshold, the processing module specifically uses the following calculation formula to calculate the target buffer size:

[0053] B(t+1)=B(t)(1+β×(B prob (t)-L thresh ))

[0054] Among them, B(t+1) represents the target buffer size;

[0055] B(t) represents the real-time buffer size;

[0056] β represents the preset buffer adjustment coefficient;

[0057] B prob (t) represents the real-time back pressure prediction value;

[0058] L thresh Indicates the preset buffer adjustment threshold.

[0059] In an optional embodiment, the device further includes a training module, wherein the training module is specifically configured to:

[0060] Acquire historical network resource data and form a historical network resource data set;

[0061] Input the historical network resource dataset into the back pressure prediction model to be trained to obtain the historical back pressure prediction value;

[0062] Based on the difference between the historical back pressure prediction value and the historical back pressure probability, the back pressure prediction model to be trained is optimized by using an adaptive moment estimation method to obtain the target back pressure prediction model.

[0063] In a third aspect, the present application provides an electronic device comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the resource adjustment method described in the first aspect above.

[0064] In a fourth aspect, the present application provides a computer-readable storage medium comprising a program code. When the program code is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the resource adjustment method described in the first aspect above.

[0065] In a fifth aspect, the present application provides a computer program product, which, when called by a computer, enables the computer to execute the steps of the resource adjustment method as described in the first aspect.

[0066] In addition, other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or may be understood by practicing the present application. The objectives and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0068] Figure 1 A schematic diagram of an application scenario of a resource adjustment method provided in an embodiment of the present application;

[0069] Figure 2 A schematic diagram of an implementation flow of a resource adjustment method provided in an embodiment of the present application;

[0070] Figure 3 This is a schematic diagram of the structure of a target back pressure prediction model adopted in an embodiment of the present application;

[0071] Figure 4 A schematic diagram of the structure of a resource adjustment device provided in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0074] It should be noted that in the description of this application, "multiple" is understood to mean "at least two." "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. A and B are connected, which can mean: A and B are directly connected, and A and B are connected through C. In addition, in the description of this application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0075] In addition, the collection, dissemination, and use of data in the technical solution of this application comply with relevant national laws and regulations.

[0076] The following is a brief introduction to the design concept of the embodiment of this application:

[0077] Flink is a distributed stream processing framework widely used in various fields, including real-time data analysis and event-driven applications. However, Flink's streaming computing often faces backpressure during execution.

[0078] Specifically, the back pressure problem is a dynamic rate-limiting mechanism in Flink's streaming computing caused by the data processing rate of downstream task nodes being lower than the data sending rate of upstream task nodes. This in turn causes data that cannot be processed in time to accumulate in downstream task nodes, increasing data processing latency and seriously affecting the processing performance of Flink's streaming computing.

[0079] In related technologies, when solving back pressure problems, operations and maintenance personnel can usually adjust the task parameters of Flink streaming computing to achieve a balance in the amount of data processed by upstream and downstream task nodes. However, this method often relies on the experience of operations and maintenance personnel, is time-consuming and labor-intensive, and cannot be adjusted accurately. In addition, related technologies can also use static configuration to configure parameters such as the data processing rate of each task node in Flink streaming computing. However, in actual application scenarios, data traffic usually shows a dynamic trend. The static configuration method cannot be dynamically adjusted according to changes in data traffic, and back pressure problems will still occur, or there will be problems of wasted resources.

[0080] In view of this, the present application provides a resource adjustment method that can dynamically adjust the resource situation in Flink streaming computing, avoid back pressure problems and avoid resource waste problems. The method includes: first, obtaining a network resource data set detected in real time by a network monitoring system; then, based on the network resource data set, using a target back pressure prediction model to perform back pressure prediction and obtain a real-time back pressure prediction value; finally, in response to the real-time back pressure prediction value being greater than a preset back pressure threshold, the target parallelism and target buffer size are calculated based on the real-time back pressure prediction value, and resources are adjusted based on the target parallelism and target buffer size. Through the above method, the future load pressure of the system can be quantified through the back pressure prediction value, so that resource adjustment is converted from passive to active adaptation. Based on the back pressure prediction value, the severity of the back pressure problem in the future can be reflected. Therefore, the calculation of the target parallelism and target buffer size based on the back pressure prediction value can adapt to the current dynamic changes in the system's traffic, and can adaptively adjust resources according to the severity of the back pressure problem, thereby avoiding the back pressure problem in advance and not excessively wasting pre-allocated resources.

[0081] For further information, see Figure 1 As shown, it is a schematic diagram of an application scenario provided by an embodiment of the present application. In this application scenario, it includes a network monitoring system 1, a server 2 and each task node. Among them, the network monitoring system 1, the server and each task node can be connected through cellular mobile communication technology, wherein the cellular mobile communication technology, for example, includes the fifth generation mobile communication (5th Generation Mobile Networks, 5G) technology; it can also be connected through a short-range wireless communication method, wherein the short-range wireless communication method, for example, includes wireless fidelity (Wireless Fidelity, Wi-Fi), and of course, communication connection can also be carried out in other ways. In an embodiment of the present application, the network monitoring system 1 can detect and obtain a network resource data set in real time. A target back pressure prediction model can be deployed in the server 2 to realize back pressure prediction and obtain a back pressure prediction value. The server 2 can adjust the resources corresponding to each task node (direction and buffer size) according to the back pressure prediction value to avoid back pressure problems and improve the stability of the system.

[0082] The resource adjustment method provided by the exemplary embodiments of the present application is described below with reference to the accompanying drawings.

[0083] See Figure 2 FIG. 1 is a schematic diagram of an implementation process of a resource adjustment method provided in an embodiment of the present application. The specific implementation process of the method is as follows:

[0084] S1: Obtain the network resource dataset detected in real time by the network monitoring system.

[0085] In an embodiment of the present application, the network monitoring system can obtain the network resource data set in Flink streaming computing in real time.

[0086] It should be noted that, in the embodiment of the present application, the network resource data set includes at least: CPU usage, memory usage, network bandwidth usage, buffer occupancy, network delay and data throughput.

[0087] Specifically, in the embodiment of the present application, the CPU usage can be calculated using the following formula:

[0088]

[0089] Among them, C cpu Indicates the current CPU usage;

[0090] T total (t) represents the total CPU usage time;

[0091] T idle (t) represents the idle CPU time.

[0092] The memory usage can be calculated using the following formula:

[0093]

[0094] Among them, C mem Indicates the current memory usage;

[0095] M used (t) indicates the size of the used memory;

[0096] M total (t) represents the total memory size.

[0097] The network bandwidth usage can be calculated using the following formula:

[0098]

[0099] Among them, C net Indicates the current network bandwidth usage;

[0100] N used (t) represents the amount of bandwidth used;

[0101] N cap Indicates the maximum bandwidth capacity.

[0102] The buffer occupancy rate can be calculated using the following formula:

[0103]

[0104] Among them, C buf Indicates the current buffer occupancy;

[0105] B size (t) indicates the current buffer size;

[0106] B cap (t) represents the maximum capacity of the buffer.

[0107] The network delay can be calculated using the following formula:

[0108] L task =O ts (t)-I ts (t)

[0109] Among them, L task Indicates the current network delay;

[0110] O ts (t) represents the timestamp of the output;

[0111] I ts (t) represents the timestamp of the input.

[0112] The data throughput can be calculated using the following formula:

[0113]

[0114] Among them, T data Indicates the current data throughput;

[0115] ΔR(t) represents the number of data processed;

[0116] Δt represents the time interval.

[0117] When the network monitoring system obtains the network resource data set in real time, it can determine the CPU utilization, memory utilization, network bandwidth utilization, buffer occupancy, network delay and data throughput data in the embodiment of the present application through the above-mentioned calculation formulas, so as to facilitate the subsequent prediction of back pressure problems.

[0118] S2: Based on the network resource dataset, the target back pressure prediction model is used to perform back pressure prediction and obtain real-time back pressure prediction values.

[0119] In the embodiment of the present application, back pressure prediction can be performed through the target back pressure prediction model, so that performance changes in Flink streaming computing can be perceived in real time, and system resources can be adjusted in time before back pressure problems occur.

[0120] Specifically, in an embodiment of the present application, a long short-term memory network (LSTM) can be used to train the target back pressure prediction model. When processing time series data, LSTM can capture long-term dependencies, so it is suitable for predicting the trend of back pressure problems. By training the model with a historical network resource data set, it is possible to predict back pressure problems at future time points.

[0121] See Figure 3 As shown, the back pressure prediction model to be trained in the embodiment of the present application may include an input layer, a hidden layer, and an output layer, and each layer has corresponding neurons connected to the next layer. The input layer can be used to receive the network resource data set in the embodiment of the present application and perform feature extraction; the hidden layer includes a forget gate, an input gate, a candidate memory unit update layer, a memory unit state update layer, and an output gate, wherein the forget gate is used to determine which information in the memory unit needs to be forgotten, the input gate is used to determine the degree of update of the memory unit by the current input information, the candidate memory unit state layer is used to calculate the new memory unit state, the memory unit update layer is used to combine the output of the forget gate and the input gate, update the memory unit state, and the output gate is used to control the output information of the memory unit; the output layer is used to connect to the hidden layer and output the final result, that is, the output layer is used to output the final back pressure prediction probability.

[0122] Furthermore, in an optional embodiment, when training the back pressure prediction model to be trained, historical network resource data is first obtained to form a historical network resource dataset. This historical network resource dataset also includes data obtained during the historical time period, such as CPU usage, memory usage, network bandwidth usage, buffer occupancy, network latency, and data throughput. For example, the network detection system obtains network resource data for the historical time period of 12:00 PM to 12:00 PM on January 5, 2024, to form the historical network resource dataset.

[0123] Then, the data in the historical network resource dataset is standardized to ensure that each network resource data is at the same scale.

[0124] For example, in the embodiment of the present application, the following formula can be used for normalization:

[0125]

[0126] Among them, F norm represents the standardized eigenvalue;

[0127] F represents the original eigenvalue;

[0128] min(F) represents the minimum value corresponding to the eigenvalue;

[0129] max(F) represents the maximum value corresponding to the eigenvalue.

[0130] After standardizing the data in the historical network resource dataset, all data can be made at the same scale.

[0131] Furthermore, the back pressure prediction model to be trained is iteratively trained using the standardized historical network resource dataset. Before the first iterative training, the parameters of the back pressure prediction model to be trained need to be initialized. For example, the weights and bias parameters of the back pressure prediction model to be trained are randomly initialized, and the number of iterations (e.g., 100) and the learning rate are set.

[0132] Thus, after the historical network resource data set is input into the back pressure prediction model to be trained, forward propagation is performed to calculate the output and loss value of each time step, thereby obtaining the historical back pressure prediction value. For example, based on the back pressure prediction model to be trained, the historical back pressure prediction value of the historical time period of 12:00-14:00 on January 5, 2024 is obtained. Then, for the historical network resource data set, manual labeling can also be used to label the historical back pressure probability. Furthermore, based on the historical back pressure prediction value and the historical back pressure probability, the difference between the historical back pressure prediction value and the historical back pressure probability can be obtained. Specifically, in an embodiment of the present application, a preset loss function can be used to calculate the difference between the historical back pressure prediction value and the historical back pressure probability. For example, in an embodiment of the present application, a cross entropy loss function can be used to calculate the difference between the historical back pressure prediction value and the historical back pressure probability.

[0133] After obtaining the difference between the historical back pressure prediction value and the historical back pressure probability, the time-based backpropagation algorithm (BPTT) can be used to calculate the gradient. The accuracy of the back pressure prediction model to be trained is evaluated using evaluation indicators such as cross-validation, F1 value, precision, and recall rate. The model parameters of the back pressure prediction model to be trained are updated using the adaptive moment estimation optimizer (Ada).

[0134] When optimizing model parameters using the Ada algorithm, including but not limited to the number of hidden layer neurons, learning rate, and number of training times, the trained target back pressure prediction model is made to converge to the global optimum while achieving high prediction accuracy.

[0135] After the target back pressure prediction model is trained in the above manner, real-time back pressure prediction can be performed to obtain a real-time back pressure prediction value. When the predicted real-time back pressure prediction value is greater than the preset back pressure threshold, resource adjustment measures are taken in a timely manner to avoid back pressure problems and optimize system performance.

[0136] In an embodiment of the present application, a network monitoring system detects and obtains a network resource data set in real time. After the obtained network resource data set is standardized, the target back pressure prediction model can be used to perform back pressure prediction, thereby obtaining a real-time back pressure prediction value. For example, a network resource data set within the time period of 14:00-16:00 on February 5, 2024 is obtained, and the network resource data set is input into the target back pressure prediction model to obtain a real-time back pressure prediction value for the time period of 16:00-18:00 on February 5, 2024. For example, the real-time back pressure prediction value is 0.8. At the same time, in an embodiment of the present application, a preset back pressure threshold is also set, and the preset back pressure threshold can be used to determine whether a back pressure problem currently occurs. For example, the preset back pressure threshold is set to 0.6. Obviously, the real-time back pressure prediction value obtained by predicting the target back pressure prediction model is greater than the preset back pressure threshold. In an embodiment of the present application, the preset back pressure threshold can be set according to the application scenario, and the embodiment of the present application does not make specific limitations on this.

[0137] S3: In response to the real-time back pressure prediction value being greater than the preset back pressure threshold, a target parallelism and a target buffer size are calculated based on the real-time back pressure prediction value, and resources are adjusted based on the target parallelism and the target buffer size.

[0138] It should be noted that in the embodiments of this application, the back pressure prediction value is used to indicate the probability and severity of back pressure problems in the future time period. The higher the back pressure prediction value, the greater the probability of back pressure problems and the greater the severity of back pressure problems; the opposite is true. A larger back pressure prediction value indicates that system resources need to be further increased, which in this embodiment means increasing the degree of parallelism and buffer size.

[0139] In an embodiment of the present application, when the real-time back pressure prediction value is greater than the preset back pressure threshold, it indicates that there is a high probability that a back pressure problem will occur in the system. In order to avoid the back pressure problem from affecting the overall stability and efficiency of the system, it is necessary to adjust the system resources in a timely manner.

[0140] Specifically, after obtaining the real-time back pressure prediction value, the target parallelism and target buffer size that need to be adjusted by the system can be calculated based on the real-time back pressure prediction value.

[0141] It should be noted that, in the embodiment of the present application, the target parallelism represents the number of parallel instances into which the same operator is split to process data simultaneously, and the target buffer size represents the capacity for temporarily storing data sent by the upstream task node.

[0142] In an optional embodiment, after obtaining the current real-time parallelism and real-time back pressure prediction value, the target parallelism can be calculated based on the real-time parallelism, real-time back pressure prediction value, preset parallelism adjustment coefficient and preset parallelism adjustment threshold.

[0143] In the embodiment of the present application, the network monitoring system can directly obtain the current real-time parallelism, and the preset parallelism adjustment system and the preset parallelism adjustment threshold can be specifically set according to the application scenario.

[0144] In an optional implementation, the target parallelism can be calculated using the following formula:

[0145] P(t+1)=P(t)×(1+α×(B prob (t)-T thresh ))

[0146] Among them, P(t+1) represents the target parallelism;

[0147] P(t) represents the real-time parallelism;

[0148] α represents the parallelism adjustment coefficient;

[0149] B prob (t) represents the real-time back pressure prediction value;

[0150] T thresh Indicates the preset parallelism adjustment threshold.

[0151] The above calculation formula can be used to calculate the target parallelism based on the real-time back pressure prediction value according to the real-time system situation.

[0152] In an optional embodiment, after obtaining the current real-time buffer size and real-time back pressure prediction value, the target buffer size can also be calculated based on the real-time buffer size, real-time back pressure prediction value, preset buffer adjustment coefficient and preset buffer adjustment threshold.

[0153] In the embodiment of the present application, the current real-time buffer size can also be directly obtained through the network monitoring system, and the preset buffer adjustment coefficient and the preset buffer adjustment threshold can be set according to the specific application scenario.

[0154] In an optional implementation, the target buffer size can be calculated using the following formula:

[0155] B(t+1)=B(t)(1+β×(B prob (t)-L thresh ))

[0156] Among them, B(t+1) represents the target buffer size;

[0157] B(t) represents the real-time buffer size;

[0158] β represents the preset buffer adjustment coefficient;

[0159] B prob (t) represents the real-time back pressure prediction value;

[0160] L thresh Indicates the preset buffer adjustment threshold.

[0161] The above calculation formula can be used to calculate the target buffer size based on the real-time back pressure prediction value according to the real-time system situation.

[0162] Furthermore, after obtaining the target parallelism and target buffer size, resource adjustment can be performed.

[0163] Specifically, when the real-time back pressure prediction value is greater than the preset back pressure threshold, there is a high probability that back pressure problems will occur in the downstream task nodes. In this case, the target parallelism and target buffer size can be calculated to reallocate resources. The current parallelism is configured as the target parallelism and the current buffer size is configured as the target buffer size. The task is restarted by calling the Flink client command. After the task is restarted, the target parallelism is reconfigured. After configuring the target parallelism, the target buffer size is also reconfigured to ensure that there are sufficient buffer resources available when the number of operator instances increases.

[0164] For example, if the current parallelism is 2 and the target parallelism is 4, then the operator resources can be configured when adjusting resources, and the corresponding parallelism can be configured to 4. For another example, if the current buffer size is 1000 bytes and the target buffer size is 3000 bytes, then the buffer size can be configured to 3000 bytes corresponding to the target buffer size when adjusting resources.

[0165] Through the resource adjustment method provided by the embodiment of the present application, the target back pressure prediction model can predict the probability of back pressure problems that may occur in the future based on the current situation of the system. The greater the back pressure prediction probability, the more serious the back pressure problem may be in the downstream task node in the future time period. Accordingly, it is necessary to increase the parallelism and buffer size to ensure that the system actively allocates resources before the back pressure problem occurs. Expanding the parallelism and buffer size in advance can avoid data loss or task blocking. At the same time, combining the parallelism adjustment coefficient and the preset parallelism adjustment threshold to adjust the parallelism and combining the buffer adjustment coefficient and the preset buffer adjustment threshold to adjust the parallelism and buffer size can ensure that the appropriate parallelism and buffer size are calculated before the back pressure problem occurs, avoiding excessive pre-allocation and resulting in resource waste.

[0166] Furthermore, based on the same technical concept, the embodiment of the present application provides a resource adjustment device, which is used to implement the above method flow of the embodiment of the present application. Figure 4 As shown, the device includes: an acquisition module 401, a prediction module 402 and a processing module 403, wherein:

[0167] Acquisition module 401 is used to acquire a network resource data set detected in real time by the network monitoring system; the network resource data set includes at least: CPU usage, memory usage, network bandwidth usage, buffer occupancy, network delay and data throughput;

[0168] Prediction module 402, configured to perform back pressure prediction based on a network resource dataset using a target back pressure prediction model to obtain a real-time back pressure prediction value;

[0169] Processing module 403 is used to calculate the target parallelism and target buffer size based on the back pressure threshold in response to the real-time back pressure prediction value being greater than the preset back pressure threshold, and to adjust resources based on the target parallelism and target buffer size; wherein, the preset back pressure threshold is used to determine whether a back pressure problem currently occurs, the target parallelism represents the number of parallel instances that the same operator is split into to process data simultaneously, and the target buffer size represents the capacity used to temporarily store data sent by the upstream task node.

[0170] In an optional implementation, when calculating the target parallelism based on the real-time back pressure prediction value, the processing module 403 is specifically configured to:

[0171] Get the current real-time parallelism and real-time back pressure prediction values;

[0172] The target parallelism is calculated based on the real-time parallelism, the real-time back pressure prediction value, the preset parallelism adjustment coefficient, and the preset parallelism adjustment threshold.

[0173] In an optional implementation, when calculating the target buffer size based on the real-time back pressure prediction value, the processing module 403 is specifically configured to:

[0174] Get the current real-time buffer size and real-time back pressure prediction value;

[0175] The target buffer size is calculated based on the real-time buffer size, the real-time back pressure prediction value, the preset buffer adjustment coefficient, and the preset buffer adjustment threshold.

[0176] In an optional embodiment, when calculating the target parallelism based on the real-time parallelism, the real-time back pressure prediction value, the preset parallelism adjustment coefficient, and the preset parallelism adjustment threshold, the processing module 403 specifically uses the following calculation formula to calculate the target parallelism:

[0177] P(t+1)=P(t)×(1+α×(B prob (t)-T thresh ))

[0178] Among them, P(t+1) represents the target parallelism;

[0179] P(t) represents the real-time parallelism;

[0180] α represents the parallelism adjustment coefficient;

[0181] B prob (t) represents the real-time back pressure prediction value;

[0182] T thresh Indicates the preset parallelism adjustment threshold.

[0183] In an optional embodiment, when the target buffer size is calculated based on the real-time buffer size, the real-time back pressure prediction value, the preset buffer adjustment coefficient, and the preset buffer adjustment threshold, the processing module 403 specifically uses the following calculation formula to calculate the target buffer size:

[0184] B(t+1)=B(t)(1+β×(B prob (t)-L thresh ))

[0185] Among them, B(t+1) represents the target buffer size;

[0186] B(t) represents the real-time buffer size;

[0187] β represents the preset buffer adjustment coefficient;

[0188] B prob (t) represents the real-time back pressure prediction value;

[0189] L thresh Indicates the preset buffer adjustment threshold.

[0190] In an optional embodiment, the apparatus further includes a training module 404, wherein the training module 404 is specifically configured to:

[0191] Acquire historical network resource data and form a historical network resource data set;

[0192] Input the historical network resource dataset into the back pressure prediction model to be trained to obtain the historical back pressure prediction value;

[0193] Based on the difference between the historical back pressure prediction value and the historical back pressure probability, the back pressure prediction model to be trained is optimized by using an adaptive moment estimation method to obtain the target back pressure prediction model.

[0194] Based on the same technical concept, the embodiment of the present application also provides an electronic device, which can implement the resource adjustment method provided in the above embodiment of the present application. In one embodiment, the electronic device can be a server, or a terminal device or other electronic device. Figure 5 As shown, the electronic device may include:

[0195] At least one processor 501, and a memory 502 connected to the at least one processor 501. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 In the example, the processor 501 and the memory 502 are connected via a bus 500. Figure 5 The bus 500 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The diagram is represented by only one thick line, but this does not mean that there is only one bus or one type of bus. Alternatively, the processor 501 may also be referred to as a controller, without limitation to the name.

[0196] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can execute a resource adjustment method discussed above by executing the instructions stored in the memory 502. The processor 501 can implement Figure 4 The functions of each module in the device shown.

[0197] Among them, the processor 501 is the control center of the device, which can use various interfaces and lines to connect the various parts of the entire control device, and monitor the device as a whole by running or executing instructions stored in the memory 502 and calling data stored in the memory 502, the various functions of the device and processing data.

[0198] In one possible design, processor 501 may include one or more processing units. Processor 501 may integrate an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily processes wireless communications. It is understood that the modem processor may not be integrated into processor 501. In some embodiments, processor 501 and memory 502 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0199] The processor 501 can be a general-purpose processor, such as a CPU, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of a resource adjustment method disclosed in the embodiments of this application can be directly implemented as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0200] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0201] By designing and programming the processor 501, the code corresponding to the resource adjustment method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 3 The steps of a resource adjustment method in the embodiment shown are as follows: How to design and program the processor 501 is a technique well known to those skilled in the art and will not be described in detail here.

[0202] Based on the same inventive concept, an embodiment of the present application further provides a storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer executes a resource adjustment method discussed above.

[0203] In some possible implementations, the present application also provides various aspects of a resource adjustment method that can also be implemented in the form of a program product, which includes program code. When the program product is run on the device, the program code is used to enable the control device to execute the steps of a resource adjustment method according to various exemplary implementations of the present application described above in this specification.

[0204] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0205] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0206] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0207] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a server, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0208] The program code used to perform the operations of the present application may be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0209] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0210] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A resource adjustment method, characterized in that: The method comprises: Obtaining a network resource data set detected in real time by a network monitoring system; the network resource data set includes at least: CPU usage, memory usage, network bandwidth usage, buffer occupancy, network latency, and data throughput; Based on the network resource data set, a target back pressure prediction model is used to perform back pressure prediction to obtain a real-time back pressure prediction value; In response to the real-time back pressure prediction value being greater than a preset back pressure threshold, the target parallelism and the target buffer size are calculated based on the real-time back pressure prediction value, and resources are adjusted based on the target parallelism and the target buffer size; wherein, the preset back pressure threshold is used to determine whether a back pressure problem currently occurs, the target parallelism represents the number of parallel instances that the same operator is split into to process data simultaneously, and the target buffer size represents the capacity for temporarily storing data sent by the upstream task node.

2. The method according to claim 1, wherein The calculating the target parallelism based on the real-time back pressure prediction value includes: Obtaining the current real-time parallelism and the real-time back pressure prediction value; The target parallelism is calculated based on the real-time parallelism, the real-time back pressure prediction value, a preset parallelism adjustment coefficient, and a preset parallelism adjustment threshold.

3. The method according to claim 1, wherein The calculating the target buffer size based on the real-time back pressure prediction value includes: Obtaining the current real-time buffer size and the real-time back pressure prediction value; The target buffer size is calculated based on the real-time buffer size, the real-time back pressure prediction value, a preset buffer adjustment coefficient, and a preset buffer adjustment threshold.

4. The method according to claim 2, wherein When calculating the target parallelism based on the real-time parallelism, the real-time back pressure prediction value, the preset parallelism adjustment coefficient, and the preset parallelism adjustment threshold, the following calculation formula is used: P(t+1)=P(t)×(1+α×(B prob (t)-T thresh )) Wherein, P(t+1) represents the target parallelism; P(t) represents the real-time parallelism; α represents the parallelism adjustment coefficient; B prob (t) represents the real-time back pressure prediction value; T thresh Indicates the preset parallelism adjustment threshold.

5. The method according to claim 3, wherein When the target buffer size is calculated based on the real-time buffer size, the real-time back pressure prediction value, the preset buffer adjustment coefficient, and the preset buffer adjustment threshold, the following calculation formula is used: B(t+1)=B(t)(1+β×(B prob (t)-L thresh )) Wherein, B(t+1) represents the target buffer size; B(t) represents the real-time buffer size; β represents the preset buffer adjustment coefficient; B prob (t) represents the real-time back pressure prediction value; L thresh Indicates the preset buffer adjustment threshold.

6. The method according to claim 1, wherein The target back pressure prediction model is trained in the following way: Acquire historical network resource data and form a historical network resource data set; Inputting the historical network resource data set into the back pressure prediction model to be trained to obtain a historical back pressure prediction value; Based on the difference between the historical back pressure prediction value and the historical back pressure probability, the back pressure prediction model to be trained is subjected to parameter optimization processing by an adaptive moment estimation method to obtain the target back pressure prediction model.

7. A resource adjustment device, characterized in that: The device comprises: An acquisition module is used to acquire a network resource data set detected in real time by a network monitoring system; the network resource data set includes at least: CPU usage, memory usage, network bandwidth usage, buffer occupancy, network delay, and data throughput; A prediction module, configured to perform back pressure prediction based on the network resource dataset using a target back pressure prediction model to obtain a real-time back pressure prediction value; The processing module is configured to calculate a target parallelism and a target buffer size based on the back pressure threshold in response to the real-time back pressure prediction value being greater than a preset back pressure threshold, and perform resource adjustment based on the target parallelism and the target buffer size.

8. The device according to claim 7, wherein When calculating the target parallelism based on the real-time back pressure prediction value, the processing module is specifically configured to: Obtaining the current real-time parallelism and the real-time back pressure prediction value; The target parallelism is calculated based on the real-time parallelism, the real-time back pressure prediction value, a preset parallelism adjustment coefficient, and a preset parallelism adjustment threshold.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images