Window control method, related device and computer-readable storage medium

Abstract

The present application provides a window control method and device. The method includes: allocating stream executors in the multi-node distributed system to multiple data streams; the input rates of at least two data streams in the multiple data streams are different; each node in the multi-node distributed system includes one or more stream executors; the processing rates of at least two stream executors in the multi-node distributed system are different; according to the correlation between the first stream executor and one or more second stream executors, determining whether to adjust the window size of the first stream executor; the first stream executor and the second stream executor are the allocated stream executors in the multi-node distributed system, and the first stream executor and the second stream executor are in the same node or process the same data stream. Through the embodiments of the present application, the resource utilization rate of the system can be improved, the processing capacity of the system can be fully utilized, and the throughput of the system can be further increased.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a window control method, related devices and equipment, and a computer-readable storage medium. Background Art

[0002] Big data analysis refers to the analysis of massive amounts of data. With the development and application of the internet, 5G, and the Internet of Things (IoT), massive amounts of data are generated daily. In some application scenarios, data volumes are skyrocketing. Big data analysis platforms that process and analyze massive data streams (also known as streaming data or simply streams) are becoming increasingly important. Generally speaking, big data analysis platforms monitor the operating status of each module and continuously optimize various parameters to achieve optimal performance within limited resources.

[0003] A key component of a big data analytics platform is the scheduler. The scheduler's primary responsibility is to allocate tasks to the various compute nodes in a distributed system using the optimal strategy and configuration, subject to effective resource and time constraints. The scheduler often needs to continuously monitor various metrics of the distributed system environment, such as execution time, execution rate, and system load, and then optimize scheduling strategies and parameters based on these indicators. Big data analytics platforms also include stream operators, which perform processing and analysis related to stream data. Stream operators typically occupy one or more nodes in the system for distributed execution.

[0004] For scenarios involving processing streaming data, an important optimization method for the scheduler is to adjust the window size (i.e., window size) of real-time streaming tasks to achieve optimal resource utilization and performance. To allow the system to adapt to changes in the input rate of the data stream, current technology uses algorithms to detect the difference between the amount of input data and the amount of data processed by the stream executor. If the difference exceeds a threshold, the batch size of the data stream is appropriately adjusted. However, in the process of adjusting the batch size of the data stream, it is necessary to ensure that the processing rate of all stream executors is higher than the input rate to avoid task backlogs or system failures. As a result, the system's resource utilization is low, and the system's processing capacity cannot be fully utilized to further increase the system's throughput. Summary of the Invention

[0005] The present application discloses a window processing method, a window processing apparatus, a window processing device and a computer-readable storage medium, which can improve the low resource utilization rate of the system, give full play to the processing capability of the system, and further increase the throughput of the system.

[0006] In a first aspect, an embodiment of the present application provides a window control method, which is applied to a multi-node distributed system. The method includes:

[0007] Assigning stream executors in the multi-node distributed system to multiple data streams; at least two of the multiple data streams have different input rates; each node in the multi-node distributed system includes one or more stream executors; at least two of the stream executors in the multi-node distributed system have different processing rates;

[0008] Determine whether to adjust the window size of the first stream executor based on the correlation between the first stream executor and one or more second stream executors; the above-mentioned first stream executor and the above-mentioned second stream executor are stream executors allocated in the above-mentioned multi-node distributed system, and the above-mentioned first stream executor and the above-mentioned second stream executor are on the same node or process the same data stream.

[0009] The embodiments of the present application are directed to a multi-node distributed system, and determine whether to adjust the window size of a stream executor based on the correlation between stream executors on the same node or processing the same data stream. That is, when adjusting the window size of a stream executor, the system takes into account the fact that multiple input data streams in the system may have different input rates and that the same data stream may be processed at different rates by multiple stream executors. Furthermore, the system determines whether to adjust the window size of the stream executor based on the mutual influence between different stream executors on the same node and / or the mutual influence between stream executors processing the same data stream. This prevents adjusting the window size of a stream executor from affecting the normal processing of data streams by other stream executors, thereby effectively resolving the problem of system failure caused by a mismatch between the input rate of a data stream and the processing rate of a stream executor. While avoiding task backlogs or system failures, it also effectively avoids the problem of some nodes being overloaded and others being underloaded, as well as the problem of slow task execution when the data stream input rate is low and system overload when the data stream input rate is increased. This improves resource utilization and load balancing, fully utilizes the system's processing capabilities, and further increases system throughput.

[0010] In one possible implementation, the above method can be executed periodically to dynamically control and optimize the window size according to changes in the processing capabilities of the stream executors in the multi-node distributed system, further enabling the system to achieve optimal performance with limited resources.

[0011] In one possible implementation, the first stream executor is any assigned stream executor in the multi-node distributed system. In embodiments of the present application, all assigned stream executors in the multi-node distributed system can perform window control sequentially or simultaneously in parallel, referring to the window control method for the first stream executor, thereby further enabling the system to achieve optimal performance with limited resources.

[0012] In one possible implementation, determining whether to adjust the window size of the first stream executor based on the correlation between the first stream executor and one or more second stream executors includes:

[0013] According to the correlation between the first stream executor and one or more second stream executors, it is analyzed whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of the data stream by the second stream executor.

[0014] Based on the correlation between stream executors on the same node or processing the same data stream, the embodiment of the present application analyzes whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of the data stream by the second stream executor, then the current window size of the first stream executor is not updated, that is, the window size of the first stream executor is not adjusted. If it does not affect the processing of the data stream by the second stream executor, then the current window size of the first stream executor is updated based on the current window adjustment step size. This effectively implements the determination of whether to adjust the window size of a stream executor based on the mutual influence between different stream executors on the same node and / or the mutual influence between stream executors processing the same data stream, thereby improving resource utilization and load balancing, fully utilizing the processing power of the system, and further increasing the system throughput.

[0015] In one possible implementation, analyzing, based on the correlation between the first stream executor and the one or more second stream executors, whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of the data stream by the second stream executors includes:

[0016] Determining a first adjustment coefficient based on a covariance between the first stream executor and the second stream executor, and a first window size; the first window size includes a window size determined based on a current window size of the first stream executor and an adjustment step size of the current window;

[0017] determining a second window size according to a first correlation coefficient between the first stream executor and the second stream executor, and the first adjustment coefficient;

[0018] If the first correlation coefficient is not 0, and the second window size exceeds the window value interval of the second stream executor, adjusting the current window size of the first stream executor based on the current window adjustment step size to affect the processing of the data stream by the second stream executor;

[0019] If the first correlation coefficient is 0, or the second window size does not exceed the window value range of the second stream executor, adjusting the current window size of the first stream executor based on the current window adjustment step does not affect the processing of the data stream by the second stream executor.

[0020] The present embodiment determines an adjustment coefficient based on the covariance between stream executors on the same node or processing the same data flow, and determines a second window size based on the correlation coefficient and adjustment coefficient between stream executors on the same node or processing the same data flow. If the second window size exceeds the window value range of the second stream executor and the correlation coefficient is not zero, then adjusting the current window size of the first stream executor based on the current window adjustment step size will affect the normal processing of the data flow by the second stream executor, potentially causing the second stream executor to process the data flow outside the window safety range, resulting in a mismatch between the data flow input rate and the processing rate of the second stream executor and system failure. If the correlation coefficient is zero or the second window size does not exceed the window value range of the second stream executor, then the current window size of the first stream executor is updated based on the current window adjustment step size. This allows determining whether to adjust the window size of a stream executor based on the mutual influence between different stream executors on the same node and / or the mutual influence between stream executors processing the same data flow, thereby improving resource utilization and load balancing, fully utilizing the processing capacity of the system, and further increasing system throughput.

[0021] In one possible implementation, the embodiment of the present application may determine the correlation coefficient coeff between two stream executors in the following manner:

[0022] If -0.2 <Coeff(SO i ,SO j )<0.2, then coeff=0;

[0023] If not, then coeff=Coeff(SO i ,SO j )

[0024] Among them, SO i Finger flow actuator i, SO j Refers to the flow actuator j.

[0025] In a possible implementation, determining the second window size according to the first correlation coefficient between the first stream executor and the second stream executor and the first adjustment coefficient includes:

[0026] determining a direction of window adjustment according to a first correlation coefficient between the first stream actuator and the second stream actuator;

[0027] The second window size is determined according to the first adjustment coefficient and the direction of the window adjustment.

[0028] The embodiment of the present application determines the direction of window adjustment based on the first correlation coefficient between the above-mentioned first stream executor and the above-mentioned second stream executor, and can accurately determine the second window size, thereby accurately determining whether to adjust the window size of the stream executor, so as to reasonably improve resource utilization and load balancing, give full play to the system's processing capabilities and increase the system's throughput.

[0029] In one possible implementation, the determining whether to adjust the window size of the first stream executor based on the correlation between the first stream executor and one or more second stream executors further includes:

[0030] Updating the impact factor according to a first window size; the first window size includes a window size determined according to a current window size of the first stream executor and a current window adjustment step size;

[0031] Determine whether the updated impact factor is within a confidence factor threshold range of the first stream executor.

[0032] The embodiment of the present application determines whether the updated impact factor is within the confidence factor threshold range of the first flow executor, and adds a judgment condition of the confidence factor, so as to more accurately determine whether to adjust the window size of the flow executor, thereby ensuring that the adjusted window size can more effectively improve resource utilization and load balancing, give full play to the system's processing capabilities and increase the system's throughput.

[0033] In a possible implementation, the impact factor includes at least one of a local impact factor and a global impact factor;

[0034] The local impact factor is determined according to the window value interval and the current window size of the first stream executor;

[0035] The global impact factor is determined according to the global window manager cost and the local window manager cost (ie, local cost) of the first stream executor.

[0036] In the embodiment of the present application, the influencing factor includes at least one of a local influencing factor and a global influencing factor, which makes the judgment of the confidence factor more meaningful and can more accurately determine whether to adjust the window size of the stream executor, ensuring that the adjusted window size can more effectively improve resource utilization and load balancing, give full play to the system's processing capabilities and increase the system's throughput.

[0037] In a possible implementation, the local impact factor C1 of the flow actuator in the embodiment of the present application can be determined by the following formula 1:

[0038]

[0039] Among them, Tmax i is the maximum value of the window value interval of stream executor i, Tmin i is the minimum value of the window value interval of stream executor i, is the current window size of stream executor i.

[0040] In one possible implementation, the global impact factor C2 of the stream executor in the embodiment of the present application can be determined by the following formula 2:

[0041]

[0042] Among them, gBest t is the current optimal global cost, is the optimal local cost of stream executor i.

[0043] In one possible implementation, the determining whether to adjust the window size of the first stream executor based on the correlation between the first stream executor and one or more second stream executors further includes:

[0044] determining whether a first global window manager cost is less than a current global window manager cost, and whether the first global window manager cost is closer to 1 than the current global window manager cost;

[0045] The first global window manager cost includes a cost determined according to a first window size; the first window size includes a window size determined according to a current window size of the first stream executor and the current window adjustment step size.

[0046] When updating the window size of a stream executor, the embodiment of the present application determines whether the first global window manager cost is less than the current global window manager cost after the window size adjustment, and whether the first global window manager cost is closer to 1 than the current global window manager cost. If so, the current window size of the first stream executor is updated to the first window size, ensuring that the global cost reaches the global optimal cost. This further optimizes system performance and ensures system load balancing.

[0047] In a possible implementation, the method further includes:

[0048] If it is determined to adjust the window size of the first stream executor, the current window size of the first stream executor is updated according to the current window adjustment step size.

[0049] The embodiment of the present application determines the window size for adjusting the above-mentioned first stream executor based on the correlation between the stream executors on the same node or processing the same data stream, and then updates the current window size of the above-mentioned first stream executor according to the current window adjustment step, thereby well achieving the adjustment of the window size of the stream executor according to the mutual influence between different stream executors on the same node and / or the mutual influence between the stream executors processing the same data stream, thereby improving resource utilization and load balancing, giving full play to the processing capability of the system, and further increasing the throughput of the system.

[0050] In a possible implementation, after updating the current window size of the first stream executor, the method further includes:

[0051] If a change in the target global window manager cost relative to the current global window manager cost reaches a change threshold, updating the current window adjustment step size, and performing again the step of analyzing, based on the correlation between the first stream executor and the one or more second stream executors, whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of the data stream by the second stream executors;

[0052] If the change in the target global window manager cost relative to the current global window manager cost does not reach the change threshold, then the updated current window size is adopted;

[0053] The target global window manager cost includes a cost determined according to the current window size after the update; and the current global window manager cost includes a cost determined according to the current window size before the update.

[0054] In this embodiment, after each update of the stream executor's window size, the global window manager cost (i.e., global cost) is combined to determine whether the change in the target global window manager cost relative to the current global window manager cost reaches a change threshold, and the window size is iteratively adjusted step by step, so that the window size adjustment achieves the desired effect, further optimizing system performance and ensuring system load balancing.

[0055] In one possible implementation, the global window manager cost can be given by the following formula 3:

[0056]

[0057] Among them, G cost The global window manager cost. i SOPR is the rate of input data flow to stream executor i (i.e. input rate). i The rate at which stream executor i processes the data stream (i.e., the processing rate). n is the number of stream executors in a multi-node distributed system. Median is used to calculate the median of an array. Variance is used to calculate the variance of an array.

[0058] In one possible implementation, the above-mentioned change threshold includes a change threshold predicted by a machine learning model based on collected historical data.

[0059] The embodiment of the present application predicts the changes in the input data stream or the changing trends of certain system attributes through machine learning based on the collected historical data, and then predicts the change threshold of the global window manager cost, so that the adjustment of the window size can achieve the expected effect, thereby better solving the problem of system failure caused by the mismatch between the data stream input rate and the stream executor processing rate; while avoiding task backlogs or system failures, it can more effectively avoid the problem of some nodes being highly loaded and some nodes being underloaded, and the problem of slow task execution when the data stream input rate is low, and easy system overload when the data stream input rate is increased, further improving resource utilization and load balancing, giving full play to the system's processing capabilities, and further increasing the system's throughput.

[0060] In a possible implementation, the updated current window adjustment step size is determined according to the last window adjustment step size, the current window size, the global window manager cost, and the local window manager cost of the first stream executor.

[0061] The embodiment of the present application determines the current window adjustment step size through the last window adjustment step size, the above-mentioned current window size, the global window manager cost and the local window manager cost of the above-mentioned first-stream executor, so as to gradually adjust the window size. Moreover, since the adjustment step size is associated with the previous historical data, the adjustment of the window size is easier to achieve the expected effect, thereby more efficiently making the global cost reach the optimal global cost, making the system performance reach the optimal level, and thus more efficiently ensuring the load balancing of the system.

[0062] In a possible implementation, the window adjustment step size in the embodiment of the present application can be determined by the following formula 4:

[0063]

[0064] in, The window adjustment step size that you want to adjust currently. w is the inertia, which can be a value obtained based on the previous historical data. r1 and r2 are random coefficients greater than 0 and less than or equal to 1. gBest t is the current optimal global cost, is the optimal local cost of stream executor i. c1 is the local impact factor of stream executor i, and c2 is the global impact factor of stream executor i. The current window size. is the window adjustment step of the last adjustment window, that is, After adjusting the window size, get the current window size

[0065] In a second aspect, an embodiment of the present application provides a window control device, comprising:

[0066] an allocating unit, configured to allocate a plurality of data streams to stream executors in the multi-node distributed system; wherein at least two of the plurality of data streams have different input rates; each node in the multi-node distributed system includes one or more stream executors; and at least two of the stream executors in the multi-node distributed system have different processing rates;

[0067] A window determination unit is used to determine whether to adjust the window size of the first stream executor based on the correlation between the first stream executor and one or more second stream executors; the first stream executor and the second stream executor are stream executors allocated in the multi-node distributed system, and the first stream executor and the second stream executor are on the same node or process the same data stream.

[0068] In a possible implementation, the window determining unit includes:

[0069] An analyzing unit is configured to analyze, based on a correlation between the first stream executor and one or more second stream executors, whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of the data stream by the second stream executor.

[0070] In a possible implementation, the analyzing unit includes:

[0071] an adjustment coefficient determining unit, configured to determine a first adjustment coefficient based on a covariance between the first stream executor and the second stream executor, and a first window size; the first window size comprising a window size determined based on a current window size of the first stream executor and the current window adjustment step size;

[0072] a window size determining unit, configured to determine a second window size according to a first correlation coefficient between the first stream executor and the second stream executor, and the first adjustment coefficient;

[0073] a coefficient determination unit, configured to determine whether the first coefficient is 0 and whether the second window size exceeds a window value interval of the second stream executor;

[0074] If the coefficient determination unit determines that the first correlation coefficient is not 0, and the second window size exceeds the window value interval of the second stream executor, adjusting the current window size of the first stream executor based on the current window adjustment step size to affect the processing of the data stream by the second stream executor;

[0075] If the coefficient determination unit determines that the first correlation coefficient is 0, or the second window size does not exceed the window value interval of the second stream executor, then adjusting the current window size of the first stream executor based on the current window adjustment step does not affect the processing of the data stream by the second stream executor.

[0076] In a possible implementation manner, the window size determining unit is specifically configured to:

[0077] determining a direction of window adjustment according to a first correlation coefficient between the first stream executor and the second stream executor;

[0078] A second window size is determined according to the first adjustment coefficient and the direction of the window adjustment.

[0079] In a possible implementation manner, the window determining unit further includes:

[0080] a factor updating unit, configured to update an impact factor according to a first window size; the first window size comprising a window size determined according to a current window size of the first stream executor and the current window adjustment step size;

[0081] A factor determination unit is configured to determine whether the updated impact factor is within a confidence factor threshold range of the first flow executor.

[0082] In a possible implementation, the impact factor includes a local impact factor or a global impact factor;

[0083] The local impact factor is determined according to the window value interval and the current window size of the first stream executor;

[0084] The global impact factor is determined based on a global window manager cost and a local window manager cost of the first stream executor.

[0085] In a possible implementation manner, the window determining unit further includes:

[0086] a cost determining unit, configured to determine whether a first global window manager cost is less than a current global window manager cost, and whether the first global window manager cost is closer to 1 than the current global window manager cost;

[0087] The first global window manager cost includes a cost determined according to a first window size; the first window size includes a window size determined according to a current window size of the first stream executor and the current window adjustment step size.

[0088] In a possible implementation, the apparatus further includes:

[0089] A window size updating unit is configured to update the current window size of the first stream executor according to the current window adjustment step if the window determining unit determines to adjust the window size of the first stream executor.

[0090] In a third aspect, an embodiment of the present application provides a window control device, comprising a processor and a memory; the memory is used to store programs; the processor is used to execute the programs stored in the memory, and when the program is executed, the processor is used to execute the method provided by the first aspect of the embodiment of the present application or any one of the implementation methods of the first aspect.

[0091] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the method provided by the first aspect or any one of the implementations of the first aspect.

[0092] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method provided by the above-mentioned first aspect or any one of the implementations of the first aspect.

[0093] It is understood that the window control apparatus provided in the second aspect, the window control device provided in the third aspect, the computer-readable storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the window control method provided in the first aspect. Therefore, the beneficial effects achievable by these methods can be referenced to the beneficial effects of the window control method provided in the first aspect and will not be further elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The following is an introduction to the drawings used in the embodiments of this application.

[0095] Figure 1 This is a data processing diagram of the big data analysis platform provided in an embodiment of the present application;

[0096] Figure 2 This is an architectural distribution diagram of the window control method provided in an embodiment of the present application;

[0097] Figure 3 1 is a flow chart of a window control method provided in an embodiment of the present application;

[0098] Figure 4 is a distribution diagram of a scheduler and a flow executor according to another embodiment of the present application;

[0099] Figure 5 This is a schematic diagram of the principle of calculating the window manager cost provided by an embodiment of the present application;

[0100] Figure 6a This is a diagram showing the relationship between the window size and the processing capability of the stream executor provided in an embodiment of the present application;

[0101] Figure 6b is another relationship diagram between the window size and the processing capability of the stream executor provided in an embodiment of the present application;

[0102] Figure 7 Schematic diagram of the process of window size adjustment analysis provided by the embodiment of the present application;

[0103] Figure 8 is a flow chart of another embodiment of the window control method provided by the present application;

[0104] Figure 9 is a flow chart of another embodiment of the window control method provided by the present application;

[0105] Figure 10 This is a graph showing the relationship between the amount of input data and the processing capacity of the stream executor provided in an embodiment of the present application;

[0106] Figure 11 is another relationship diagram between the input data volume and the processing capacity of the stream executor provided in an embodiment of the present application;

[0107] Figure 12 is a structural diagram of a window control device provided in an embodiment of the present application;

[0108] Figure 13 is a structural diagram of a window determination unit provided in an embodiment of the present application;

[0109] Figure 14 This is a structural diagram of a window control device provided in an embodiment of the present application;

[0110] Figure 15 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0111] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0112] The terms "first," "second," "third," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0113] Combine first Figure 1 Introducing a data processing diagram of the big data analysis platform provided in an embodiment of the present application.

[0114] Data can be input based on different application scenarios, such as network element key performance indicators (KPIs), user visits, or data streams. Data streams are a common concept in data processing, indicating that data flows continuously.

[0115] The big data analysis platform may include components such as the Executor Cluster, Scheduler, and System Monitoring (Dashboard).

[0116] The scheduler is used to schedule and control tasks, such as assigning configurations to tasks, such as resource pools, and assigning tasks to specific nodes in the execution cluster. Schedulers include intelligent schedulers, which use specific algorithms or strategies to adapt scheduling and automatically optimize it to the optimal scheduler.

[0117] System monitoring collects or monitors the system's initial parameter values or operating status, including the input rate of data streams, the processing rate of stream executors, the window size of stream executors' input streams, task status, KPI indicators and their historical information, or node resource status. The window size controls the range of data processed by the stream executors at each time.

[0118] The stream operator is used to perform processing and analysis related to stream data. The stream operator can occupy one or more nodes in the cluster for distributed execution. Its analysis process is also called real-time stream processing or real-time stream analysis. The scheduler can adjust the window size using sliding window technology. The scheduler of the embodiment of the present application can dynamically and adaptively adjust the window size based on various collected parameter values or indicator values.

[0119] By executing real-time stream processing or real-time stream analysis on the computing nodes in the cluster, the resulting data is obtained and output. The output data can be used for subsequent data analysis or storage.

[0120] The following combination Figure 2 The architecture distribution diagram of the window control method provided by the embodiment of the present application is shown, which illustrates the distribution of the scheduler and the flow executor in the multi-node distributed system of the present application.

[0121] like Figure 2 As shown, the multi-node distributed system has n input data streams, data streams S1~S n There are K stream executors on the cluster, stream executors SO1~SO k This multi-node distributed system has m computing nodes, Node-1 to Node-m. Each stream executor can have multiple computing units distributed across different nodes, and each stream executor may process one or more input data streams.

[0122] In one possible implementation, each flow executor may represent the execution of one service, and K flow executors may represent the execution of K services.

[0123] The scheduler in the multi-node distributed system can include two parts, a basic module and a machine learning module.

[0124] The basic module may include a resource pool manager, a window manager, a task manager, etc. The window manager may be used to implement an internal window control algorithm to adjust the window size. In other words, the window control method provided in the embodiment of the present application can be said to be executed by the scheduler, but this application does not limit this.

[0125] The machine learning module can be used to predict input data or the changing trends of certain system properties, such as predicting the threshold of global window manager cost changes, etc.

[0126] in, Figure 2 Each stream executor has computing units distributed on each node, but Figure 2 The node resource distribution in is only an example and is not limited in this application. The number of flow executors included in each node in the multi-node distributed system of the embodiment of the present application can be different, and the distribution of each flow executor on the node can also be different; for example Figure 2 The computing units of the midstream executor SO1 may be distributed only on nodes N1 and N2, while the computing units of the midstream executor SO2 may be distributed only on nodes N3 and N4.

[0127] The window size is a major factor affecting the processing speed of the stream executor. The input rate of a data stream can be expressed as the number of input data streams per second. Changes in the window size, therefore, indicate changes in the amount of data the stream processor must process within the window. The window size can be expressed as the number of input data streams per unit time, or the duration of a unit of input data streams. If the data within each window cannot be processed promptly, the data for the next window will be accumulated. Therefore, adjusting the window size based on resource utilization is a very effective way to optimize performance.

[0128] The following combination Figure 3 The flowchart of the window control method provided by the embodiment of the present application is shown, and Figure 4 A distribution diagram of a scheduler and a stream executor according to another embodiment of the present application is shown to illustrate how the scheduler according to the present application performs window control, which may include the following steps:

[0129] Step S300: allocating stream executors in the multi-node distributed system to multiple data streams;

[0130] Specifically, the scheduler can allocate stream executors in a multi-node distributed system to multiple data streams to be input according to specific business conditions through the resource manager in the basic module.

[0131] For example, Figure 4As shown, there are three input data streams S1 to S3, each with a different input rate. The scheduler assigns processing to three nodes in the system based on the traffic situation. That is, the computer cluster processing these three data streams has three nodes.

[0132] Each node has different resources. Node N1 has the computing units of stream executors SO1 and SO2, node N2 has the computing units of stream executors SO1 and SO3, and node N3 has the computing unit of stream executor SO2. Thus, there are three stream executors, SO1, SO2, and SO3, for a total of five computing units, and each computing unit can have a different processing rate.

[0133] The processing rate corresponding to the computing unit SO1 in node N1 can be P1, and the processing rate corresponding to the computing unit SO2 in node N1 can be P2. The performance of node N1 can be P1 + P2. The processing rate corresponding to the computing unit SO1 in node N2 can be P3, and the processing rate corresponding to the computing unit SO3 in node N2 can be P4. The performance of node N2 can be P3 + P4. The processing rate corresponding to the computing unit SO2 in node N3 can be P5, and the performance of node N3 can be P5.

[0134] like Figure 4 In the example, based on business requirements, the stream executors assigned to the three input data streams S1 to S3 can be as follows: Stream executor SO1 calculates the average value of data stream S1. Stream executor SO2 performs a join operation on data streams S1 and S2. Stream executor SO3 performs a union operation on data streams S1 and S3.

[0135] In the embodiments of the present application, adjusting the window size specifically refers to adjusting the window size of the input data stream of the stream executor. For example, this includes adjusting the window size of the input data stream of stream executor SO1, or adjusting the window size of the input data stream of stream executor SO3. Adjusting the window size of the input data stream of stream executor SO1 may include adjusting the window size of the input data stream of the computing units distributed on each node of stream executor SO1. For example, if the window size of the input data stream of the computing unit of stream executor SO3 in node N3 is adjusted to input 100 data streams per second, this may mean that 100 data streams are input from both data streams S1 and S3 per second.

[0136] In one possible implementation, step S300 can dynamically adjust the allocated resources as the services change, such as when new services are added or new input data flows are added.

[0137] The following steps S302 to S322 may be executed periodically, or may be triggered to execute after the allocated resources are dynamically adjusted in step S300.

[0138] Step S302: collecting various system indicators through the system monitoring module;

[0139] Specifically, the indicators collected by the monitoring module may include but are not limited to the initial value or operating status of the system, input rate, processing capacity of the stream executor and its node distribution, etc. Figure 4 For example, the input rates of data streams S1 to S3, the input data stream rates IR1, IR2 and IR3 of the three stream executors SO1, SO2 and SO3, the processing capabilities of the three stream executors SO1, SO2 and SO3, etc. are collected.

[0140] Step S304: Determine the correlation between the flow executors on the same node or processing the same data flow;

[0141] Specifically, SRel i ={SO j ,coeff,cov} to represent the stream executor SO i and stream executor SO j Where coeff is the correlation coefficient between the stream executors and cov is the covariance between the stream executors.

[0142] In one implementation, the correlation coefficient coeff between two stream executors can be determined directly by Coeff(SO i ,SO j ) can also be determined by:

[0143] If -0.2 <Coeff(SO i ,SO j )<0.2, then coeff=0;

[0144] If not, then coeff=Coeff(SO i ,SO j ).

[0145] The embodiment of the present application does not limit the correlation coefficient function Coeff(SO i ,SO j ) is calculated using the specific formula.

[0146] In one implementation, the covariance cov between the stream executors can be determined by cov=|Cov(SO i ,SO j )| to determine. Among them, the sequence SO in the covariancei Can refer to the stream executor SO i Processing capabilities, such as stream executors SO i The sequence of processing rates; the sequence of covariances SO j Can refer to the stream executor SO j Processing capabilities, such as stream executors SO j The processing rate of the series.

[0147] Step S306: Obtain the processing rate of the current stream executor;

[0148] Specifically, get the current processing rate of all stream executors in the system. Figure 4 For example, the current processing rates SOPR1, SOPR2 and SOPR3 of the three stream executors SO1, SO2 and SO3 are obtained.

[0149] Step S308: Input the input rate and processing rate into the trained machine learning model and output the global cost change threshold;

[0150] Specifically, embodiments of the present application can predict data changes through machine learning. A machine learning model can use collected historical data as sample data, such as the historical input rate of a data stream and the processing rate of a stream executor. The sample data is input into the machine learning model for training, and the parameters of the machine learning model are continuously optimized until the global cost change threshold output by the machine learning model meets the requirements, thus completing the training.

[0151] by Figure 4 For example, in step S308, the input rates of data streams S1 to S3 and the current processing rates SOPR1, SOPR2 and SOPR3 of the three stream executors SO1, SO2 and SO3 can be input into the trained machine learning model to output the global cost change threshold.

[0152] In one implementation, the machine learning model in the embodiment of the present application can also predict the change thresholds of global cost and local cost.

[0153] In one implementation, the embodiment of the present application may not predict data changes through machine learning. That is, step S308 may be replaced by directly setting a global cost change threshold, for example, by a technician based on historical data values, experimental values, or empirical values.

[0154] The following steps S310 to S316 take the first stream executor in the system as an example, for example Figure 4The following describes how to determine whether to adjust the window size of the first stream executor based on the correlation between the first stream executor and the second stream executor, and how to adjust the window size after determining to adjust the window size. The window control method of the embodiment of the present application can be applied to multiple other stream executors or all stream executors in the system.

[0155] Among them, the embodiment of the present application does not limit the execution order of step S310, the combination of steps S312 and S314, and step S316, and the execution order can be adjusted as needed.

[0156] Step S310: Analyzing, based on the correlation between the first stream executor and one or more second stream executors, whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of data streams by the second stream executors;

[0157] Specifically, the embodiment of the present application can pre-set analysis rules or judgment rules that affect the data flow processed by the flow executor, that is, define conditions or standards to determine whether the data flow belongs to or does not affect the data flow processed by the flow executor.

[0158] For example, each stream executor can have its own window value interval, which represents the range of window values within which the stream executor can safely process data streams. If adjusting the window size of a first stream executor causes the window size of a second stream executor to exceed its window value interval, this indicates that the second stream executor cannot promptly process the data stream within the window, resulting in a data backlog and affecting the second stream executor's data stream processing. If the window value interval does not exceed its window value interval, this indicates that the second stream executor's data stream processing is not affected.

[0159] For example, each stream executor has a processing rate range based on its own capabilities. This processing rate range indicates that the stream executor is processing data normally and reasonably. If adjusting the window size of the first stream executor causes the processing rate of the second stream executor to exceed its own processing rate range, this will either cause inefficiency or overload, thus affecting the second stream executor's processing of the data flow. If it does not exceed its own processing rate range, it indicates that the second stream executor's processing of the data flow will not be affected.

[0160] If step S310 analyzes that the second stream executor does not process the data stream, step S312 is executed. If step S310 analyzes that the second stream executor processes the data stream, the adjustment of the window size of the first stream executor is terminated. Alternatively, if step S310 analyzes that the second stream executor processes the data stream, the window adjustment step size can be adjusted. Specifically, the number of adjustments can be set as required. If the number of adjustments has not been reached, the window adjustment step size is adjusted, and step S310 is re-executed based on the new window adjustment step size. If the number of adjustments has been reached, the adjustment of the window size of the first stream executor is terminated.

[0161] In an embodiment of the present application, when there are multiple second stream executors in the system that are on the same node as the first stream executor or process the same data stream, whether to adjust the window size of the first stream executor can be determined based on the correlation between the first stream executor and all the second stream executors, or whether to adjust the window size of the first stream executor can be determined based on the correlation between the first stream executor and some (one or several) of the second stream executors.

[0162] For example Figure 4 In the example, stream executor SO1 is located on the same node N1 as stream executor SO2, and stream executor SO1 is located on the same node N2 as stream executor SO3. Therefore, the second stream executor located on the same node as the first stream executor includes at least stream executor SO2 and stream executor SO3. Step S310 may analyze, based on the correlation between stream executors SO1 and SO2, whether adjusting the current window size of stream executor SO1 based on the current window adjustment step size affects data flow processing by stream executor SO2. Furthermore, based on the correlation between stream executor SO1 and SO3, it may analyze whether adjusting the current window size of stream executor SO1 based on the current window adjustment step size affects data flow processing by stream executor SO3.

[0163] In one implementation, the embodiment of the present application does not limit the setting method of the window adjustment step size. The setting method can be performed by a technician based on historical data values, experimental values, or empirical values, or calculated according to a set algorithm. The window adjustment step size can be adjusted gradually, for example, the initial value is a very small step size, and then gradually increases.

[0164] Step S312: updating the impact factor according to the first window size;

[0165] Specifically, the first window size includes a window size determined according to the current window size of the first stream executor and the current window adjustment step size. In other words, the first window size is the window value after the current window size is adjusted by the current window adjustment step size.

[0166] In the embodiment of the present application, the influencing factor may specifically be a confidence factor used when adjusting the window size, for example, a confidence factor used to generate or update the window adjustment step size, or a confidence factor used to determine whether the adjusted and updated window size is appropriate.

[0167] In one implementation, the impact factor may include at least one of a local impact factor and a global impact factor;

[0168] The local impact factor is determined according to the window value interval and the current window size of the first stream executor;

[0169] The global impact factor is determined according to a global window manager cost and a local window manager cost of the first stream executor.

[0170] Then, updating the local impact factor according to the first window size may specifically include using the first window size as the current window size and combining the window value interval of the first stream executor to determine the updated local impact factor. Updating the global impact factor according to the first window size may specifically include using the first window size as the current window size to determine the local window manager cost and the global window manager cost of the first stream executor, and then updating the global impact factor based on the global window manager cost and the local window manager cost of the first stream executor.

[0171] Step S314: determining whether the updated impact factor is within a confidence factor threshold range of the first stream executor;

[0172] Specifically, each stream executor can pre-set its own confidence factor threshold range. After updating the impact factor, if it is determined that the updated impact factor value is within the confidence factor threshold range of the first stream executor, step S316 can be executed. Otherwise, the window size adjustment of the first stream executor can be terminated, or the window adjustment step size can be adjusted. Specifically, the number of adjustments can be set as required. If the number of adjustments has not been reached, the window adjustment step size is adjusted, and step S310 is re-executed based on the new window adjustment step size. If the number of adjustments has been reached, the window size adjustment of the first stream executor can be terminated.

[0173] Step S316: determining whether the first global window manager cost is less than the current global window manager cost, and whether the first global window manager cost is closer to 1 than the current global window manager cost;

[0174] In one possible implementation, Figure 5A schematic diagram illustrating the principle of calculating the window manager cost provided by an embodiment of the present application is shown. When calculating the global window manager cost, the input rate of all data streams and the processing rate of all stream executors can be used as input. When calculating the local window manager cost, the cost can be calculated for a specific stream executor by inputting the rate of the data stream input by the stream executor and the processing rate of the stream executor.

[0175] The global window manager cost can be obtained by the following formula 3:

[0176]

[0177] Among them, G cost The global window manager cost. i SOPR is the rate of input data flow to stream executor i (i.e. input rate). i The rate at which stream executor i processes the data stream (i.e., the processing rate). n is the number of stream executors in a multi-node distributed system. Median is used to calculate the median of an array. Variance is used to calculate the variance of an array.

[0178] Specifically, the first global window manager cost in the embodiment of the present application includes a cost determined based on the first window size. That is, when calculating the first global window manager cost using Formula 1, the input rate and processing rate of stream executor i are based on the assumed window value after adjusting the window size of stream executor i. When calculating the current global window manager cost using Formula 1, the current input rate and processing rate of stream executor i are used.

[0179] If it is determined that the first global window manager cost is less than the current global window manager cost, and the above-mentioned first global window manager cost is closer to 1 relative to the above-mentioned current global window manager cost, then step S318 is executed; if it is determined that the first global window manager cost is not less than the current global window manager cost, or the above-mentioned first global window manager cost is not closer to 1 relative to the above-mentioned current global window manager cost, then the adjustment of the window size of the first stream executor can be ended, or the window adjustment step can be adjusted. Specifically, the number of adjustments can be set according to needs. If the number of adjustments has not been reached, after adjusting the window adjustment step, step S310 is re-executed according to the new window adjustment step; if the number of adjustments is reached, the adjustment of the window size of the first stream executor can be ended.

[0180] Step S318: adjusting the step size of the current window to update the current window size of the first stream executor;

[0181] Specifically, in step S318, the update can be completed by the following formula 5:

[0182]

[0183] in, is the updated window size of stream executor i, is the current window size of stream executor i, Adjust the step size for the current window.

[0184] In one implementation, after step S318, step S320 may not be performed, and step S322 may be performed directly, that is, the updated current window size is directly adopted. If step S320 is not performed, then the corresponding steps S306 and S308 may also not be performed.

[0185] Step S320: determining whether a change in the target global window manager cost relative to the current global window manager cost reaches a change threshold;

[0186] Specifically, the target global window manager cost includes a cost determined based on the updated current window size. That is, the target global window manager cost is determined based on the input rates of all data streams obtained based on the updated current window size and the processing rates of all stream executors, and is calculated using Formula 1. The current global window manager cost includes a cost determined based on the current window size before the update. That is, the current global window manager cost is the cost calculated based on the input rates of all data streams obtained based on the current window size before the update in step S318 and the processing rates of all stream executors, and is calculated using Formula 1.

[0187] If it is determined that the change amount reaches the change threshold in step S308, for example, is greater than or equal to the change threshold, indicating that the adjustment has not achieved the expected result, the current window adjustment step size is updated, and step S310 and subsequent steps are performed again. If it is determined that the change amount does not reach the change threshold in step S308, for example, is less than the change threshold, indicating that the adjustment has achieved the expected result, step S322 is performed.

[0188] That is, after the current window size is updated in step S318, the updated current window size is not immediately adopted. Instead, an iterative determination is performed in step S320. During the iterative process, if the result of the analysis in step S310 is negative, or the result of the combination of steps S312 to S314 is negative, or the result of the determination in step S316 is negative, step S322 is directly executed.

[0189] In a possible implementation, the update window adjustment step size or the adjustment window adjustment step size in step S310 may be determined by the following formula 4:

[0190]

[0191] in, is the updated or adjusted window adjustment step size. w is inertia, which can be a value obtained based on previous historical data. r1 and r2 are random coefficients greater than 0 and less than or equal to 1. Each time the window adjustment step size is determined or updated, the values of r1 and r2 are different. gBest t is the current optimal global cost, is the optimal local cost of stream executor i. c1 is the local impact factor of stream executor i, and c2 is the global impact factor of stream executor i. The current window size. is the window adjustment step of the last adjustment window, that is, After adjusting the window size, get the current window size

[0192] Step S322: adopt the updated current window size.

[0193] Specifically, for example Figure 4 After step S322, the window size adjustment of the stream executor SO1 is completed. Then the window size adjustment of other stream executors is processed, or the window size adjustment of the stream executor SO1 is completed. Figure 4 The window size adjustment of the three stream executors SO1, SO2 and SO3 is performed synchronously and in parallel from S310 to S322.

[0194] In one implementation, for example, stream executor SO1 in the system includes only the computing unit SO1 in node N1 and the computing unit SO1 in node N2. Before adjustment, the window size of stream executor SO1 is 200 data streams per second. Specifically, the window size of the computing unit SO1 in node N1 is 100 data streams per second, and the window size of the computing unit SO1 in node N2 is 100 data streams per second. After the update, the window size of stream executor SO1 is 280 data streams per second. However, this embodiment of the application does not specifically limit how the window sizes of the computing units SO1 in node N1 and SO1 in node N2 are adjusted. For example, the window size of the computing unit SO1 in node N1 can be adjusted to 150 data streams per second, and the window size of the computing unit SO1 in node N2 can be adjusted to 130 data streams per second. Alternatively, the window size of the computing unit SO1 in node N1 may be adjusted to input 140 data streams per second, and the window size of the computing unit SO1 in node N2 may be adjusted to input 140 data streams per second, and so on.

[0195] In one implementation, before step S310, the scheduler may pre-acquire the following data structure:

[0196]

[0197]

[0198]

[0199] Among them, the data structure map SOStreamRelation Represents the correlation data between stream executors that process the same data stream. Data structure map SONodeRelation Represents the correlation data between stream executors on the same node. Data structure map StreamDivision Indicates the processing time of each data stream. Data structure map SORange Indicates the window value interval for each stream executor.

[0200] Among them, the window value interval of each stream executor [T min ,T max ], please refer to Figure 6a and Figure 6b The relationship diagram shown is a relationship between the window size and the processing capability of the stream executor. The relationship diagram can be obtained based on historical data or experimental data actually processed.

[0201] Figure 6a The horizontal axis is the window size, and the vertical axis is the end-to-end delay (i.e., the time it takes for the stream executor to receive the task and complete the task). In the technical scenario of big data analysis, when the window of the stream executor is very small, the data that can be processed is also relatively small, and the delay will be relatively large. As the window increases to a certain point, the delay will reach a minimum value. At this point, the slope of the curve is 0, and this point (i.e., the lowest point of the curve) is marked as the first minimum window size T min1 As the window size increases, the latency increases as more data needs to be processed. The lowest point of the curve represents a balance between latency and the processing power of the stream executor.

[0202] Figure 6b The horizontal axis is the window size, and the vertical axis is the ratio of processing rate to input rate. P represents the processing rate of different stream executors. For the processing rate and input rate, the ratio between the two indicates the degree of data backlog. As the window size increases, the stream executor can process more data. As the resource gradually reaches the upper limit, the processing capacity will drop sharply. There are two points in the figure where the ratio is 1, which are the second smallest window size T min2 and the maximum window size T max The point where the ratio is 1 is the ideal scenario.

[0203] Then, in the window value interval [T min ,T max ], T min =max(T min1 ,T min2 ), T max Can be Figure 6b T in max .

[0204] In one implementation, the execution of step S310 can refer to Figure 7 The flowchart of the window size adjustment analysis provided by the embodiment of the present application is shown, which may specifically include the following steps:

[0205] Step S700: determining a first adjustment coefficient according to the covariance between the first stream executor and the second stream executor, and the first window size;

[0206] In one implementation, before step S700, it is also possible to first determine whether the above-mentioned first window size exceeds the window value interval of the first stream executor, and whether the input rate of the current data stream S1 is less than or equal to the processing time of the data stream S1. If the above-mentioned first window size does not exceed the window value interval of the first stream executor, and the input rate of the current data stream S1 is less than or equal to the processing time of the data stream S1, then step S700 is executed. Otherwise, the window adjustment step size can be adjusted, and the number of adjustments can be set according to the needs. If the number of adjustments has not been reached, after adjusting the window adjustment step size, the judgment step before the above-mentioned step S700 is re-executed based on the new window adjustment step size obtained; if the number of adjustments has been reached, the adjustment of the window size of the first stream executor can be terminated.

[0207] Specifically, the scheduler can obtain the SOStreamRelation and / or data structure map SONodeRelation The covariance between the first stream executor and the second stream executor is obtained. The first window size includes a window size determined based on the current window size of the first stream executor and the current window adjustment step size; for example, the first window size is determined using Formula 6. The scheduler then determines a first adjustment coefficient based on the covariance and the first window size, for example, by multiplying the covariance by the first window size to obtain the first adjustment coefficient.

[0208] Step S702: determining a second window size according to the first correlation coefficient between the first stream executor and the second stream executor, and the first adjustment coefficient;

[0209] Specifically, the scheduler can also get the data structure map SOStreamRelationand / or data structure map SONodeRelation A first correlation coefficient between the first flow executor and the second flow executor is obtained.

[0210] In one implementation, when determining the second window size based on the first correlation coefficient, the direction of window adjustment can be determined specifically based on the first correlation coefficient. For example, if the first correlation coefficient is greater than 0, the window size is increased, which is equivalent to adjusting the window size to the right; if the first correlation coefficient is not greater than 0, the window size is decreased, which is equivalent to adjusting the window size to the left. Alternatively, if the first correlation coefficient is greater than or equal to 0, the window size is increased; if the first correlation coefficient is less than 0, the window size is decreased.

[0211] Then, based on the direction of window adjustment and the first adjustment coefficient, the second window size can be determined. For example, if the window size is adjusted to the right (increasing the window size), the second window size is equal to the first window size plus the first correlation coefficient; if the window size is adjusted to the left (decreasing the window size), the second window size is equal to the first window size minus the first correlation coefficient.

[0212] Step S704: Analyze whether the first correlation coefficient is 0 and whether the second window size exceeds the window value range of the second stream executor.

[0213] Specifically, the scheduler can obtain the SORange Get the window value interval of the second stream executor.

[0214] If the first correlation coefficient is not 0, and the second window size exceeds the window value interval of the second stream executor, it indicates that adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of the data stream by the second stream executor;

[0215] If the first correlation coefficient is 0, or the second window size does not exceed the window value range of the second stream executor, it indicates that adjusting the current window size of the first stream executor based on the current window adjustment step does not affect the processing of the data stream by the second stream executor.

[0216] In a possible implementation, the determination and update of the impact factor in step S312 may be specifically as follows:

[0217] In this embodiment of the application, the local impact factor C1 of the flow actuator can be determined by the following formula 1:

[0218]

[0219] Among them, Tmax iis the maximum value of the window value interval of stream executor i, Tmin i is the minimum value of the window value interval of stream executor i, is the current window size of stream executor i.

[0220] In one possible implementation, the global impact factor C2 of the stream executor in the embodiment of the present application can be determined by the following formula 2:

[0221]

[0222] Among them, gBest t is the current optimal global cost, is the optimal local cost of stream executor i.

[0223] In one implementation, when determining whether to adjust the window size of the first stream executor, the embodiment of the present application may not need to perform the combination of steps S312 and S314, and step S316, or may only perform the combination of steps S312 and S314, or only perform step S316. Figures 8 and 9 , to illustrate:

[0224] if Figure 3 The embodiment only performs the combination of steps S312 and S314, so if Figure 8 A flowchart illustrating another embodiment of the window control method provided by the present application may include the following steps:

[0225] Step S800: allocating stream executors in the multi-node distributed system to multiple data streams;

[0226] Step S802: Collect various system indicators through the system monitoring module;

[0227] Step S804: Determine the correlation between the flow executors on the same node or processing the same data flow;

[0228] Step S806: Obtain the processing rate of the current stream executor;

[0229] Step S808: Input the input rate and processing rate into the trained machine learning model and output the global cost change threshold;

[0230] Step S810: Analyzing, based on the correlation between the first stream executor and the second stream executor, whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of the data stream by the second stream executor;

[0231] Step S812: updating the impact factor according to the first window size;

[0232] The specific implementation of steps S802 to S812 can refer to Figure 3 Steps S302 to S312 in the embodiment.

[0233] Step S814: determining whether the updated impact factor is within a confidence factor threshold range of the first stream executor;

[0234] Specifically, each stream executor can pre-set its own confidence factor threshold range. After updating the impact factor, if it is determined that the updated impact factor value is within the confidence factor threshold range of the first stream executor, step S816 can be executed. Otherwise, the window size adjustment of the first stream executor can be terminated, or the window adjustment step size can be adjusted. Specifically, the number of adjustments can be set as required. If the number of adjustments has not been reached, the window adjustment step size is adjusted, and step S810 is re-executed based on the new window adjustment step size. If the number of adjustments has been reached, the window size adjustment of the first stream executor can be terminated.

[0235] Step S816: adjusting the step size of the current window to update the current window size of the first stream executor;

[0236] Step S818: determining whether the change in the target global window manager cost relative to the current global window manager cost reaches a change threshold;

[0237] Specifically, the specific implementation of steps S816 to S818 can refer to Figure 3 Steps S318 to S320 in the embodiment.

[0238] If it is determined that the change amount reaches the change threshold in step S808, for example, is greater than or equal to the change threshold, indicating that the adjustment has not achieved the expected result, the current window adjustment step size is updated, and step S810 and subsequent steps are executed again. If it is determined that the change amount does not reach the change threshold in step S808, for example, is less than the change threshold, indicating that the adjustment has achieved the expected result, step S820 is executed.

[0239] Step S820: adopt the updated current window size.

[0240] Specifically, the specific implementation of step S820 can refer to Figure 3 Step S322 in the embodiment.

[0241] in, Figure 8 In the embodiment, the order of executing step S810 and the order of executing the combination of steps S812 and S814 may be swapped.

[0242] if Figure 3The embodiment only executes step S316, so if Figure 9 A flowchart illustrating another embodiment of the window control method provided by the present application may include the following steps:

[0243] Step S900: allocating stream executors in the multi-node distributed system to multiple data streams;

[0244] Step S902: Collect various system indicators through the system monitoring module;

[0245] Step S904: Determine the correlation between the flow executors on the same node or processing the same data flow;

[0246] Step S906: Obtain the processing rate of the current stream executor;

[0247] Step S908: Input the input rate and processing rate into the trained machine learning model and output the global cost change threshold;

[0248] Step S910: Analyzing, based on the correlation between the first stream executor and the second stream executor, whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the data stream processed by the second stream executor;

[0249] Specifically, the specific implementation of step S900 to step S910 can refer to Figure 3 Steps S300 to S310 in the embodiment.

[0250] If step S910 analyzes that the second stream executor does not process the data stream, step S912 is executed. If step S910 analyzes that the second stream executor processes the data stream, the adjustment of the window size of the first stream executor is terminated. Alternatively, if step S910 analyzes that the second stream executor processes the data stream, the window adjustment step size can be adjusted. Specifically, the number of adjustments can be set as required. If the number of adjustments has not been reached, the window adjustment step size is adjusted, and step S910 is re-executed based on the new window adjustment step size. If the number of adjustments has been reached, the adjustment of the window size of the first stream executor is terminated.

[0251] Step S912: determining whether the first global window manager cost is less than the current global window manager cost, and whether the first global window manager cost is closer to 1 than the current global window manager cost;

[0252] If it is determined that the first global window manager cost is less than the current global window manager cost, and the above-mentioned first global window manager cost is closer to 1 relative to the above-mentioned current global window manager cost, then step S914 is executed; if it is determined that the first global window manager cost is not less than the current global window manager cost, or the above-mentioned first global window manager cost is not closer to 1 relative to the above-mentioned current global window manager cost, then the adjustment of the window size of the first stream executor can be ended, or the window adjustment step can be adjusted. Specifically, the number of adjustments can be set according to needs. If the number of adjustments has not been reached, after adjusting the window adjustment step, step S910 is re-executed according to the new window adjustment step; if the number of adjustments is reached, the adjustment of the window size of the first stream executor can be ended.

[0253] Step S914: adjusting the step size of the current window to update the current window size of the first stream executor;

[0254] Step S916: determining whether the change in the target global window manager cost relative to the current global window manager cost reaches a change threshold;

[0255] Step S918: adopt the updated current window size.

[0256] Specifically, the specific implementation of steps S914 to S918 can refer to Figure 3 Steps S318 to S322 in the embodiment.

[0257] in, Figure 9 In the embodiment, the order of executing step S910 and the order of executing step S912 may be swapped.

[0258] by Figure 4 For example, if a fixed window size mode is used to execute the input data stream, such as Figure 10 The relationship diagram of the input data volume and the processing capacity of the stream executor provided in the embodiment of the present application is shown. It can be seen from the processing rate / input rate curve in the figure that if the window size is fixed, the stream executor is available when the processing rate / input rate is not less than 1. That is, when the value of the vertical coordinate is less than 1, the system is unavailable. Therefore, the processing rate must be higher than the input rate. When the window is fixed, the stream executor must increase resources (such as the number of CPU cores, memory, etc.) to ensure that the processing rate is higher than the input rate, but increasing resources is not feasible in many application scenarios.

[0259] If the window size is dynamically adjusted by the window control method of the embodiment of the present application, such as Figure 11Another relationship diagram between the amount of input data and the processing capacity of the stream executor provided by the embodiment of the present application is shown. The curve of processing rate / input rate in the figure shows that when the input increases, the same stream executor can process more input data by adjusting the window size according to the embodiment of the present application. If the processing rate / input rate is less than 1, the system will also become unusable. But from Figure 11 As can be seen from the above, when the value of the vertical coordinate of the dynamic adjustment window is lower than the critical point of 1, the maximum input data that the system can process is relative to the value in Figure 10 That is to say, by adopting the window control method of the embodiment of the present application to dynamically adjust the window size, the system has a greater processing capability, or a greater throughput.

[0260] The embodiment of the present application also provides a window control device, such as Figure 12 The schematic diagram of the structure of the window control device provided by the embodiment of the present application is shown. The window control device 12 may include an allocation unit 120 and a window determination unit 122, wherein:

[0261] The allocation unit 120 is configured to allocate a plurality of data streams to stream executors in the multi-node distributed system; at least two of the plurality of data streams have different input rates; each node in the multi-node distributed system includes one or more stream executors; and at least two of the stream executors in the multi-node distributed system have different processing rates.

[0262] The window determination unit 122 is used to determine whether to adjust the window size of the first stream executor based on the correlation between the first stream executor and one or more second stream executors; the above-mentioned first stream executor and the above-mentioned second stream executor are stream executors allocated in the above-mentioned multi-node distributed system, and the above-mentioned first stream executor and the above-mentioned second stream executor are on the same node or process the same data stream.

[0263] In one implementation, such as Figure 13 The schematic diagram of the structure of the window determination unit provided in an embodiment of the present application is shown. The window determination unit 122 may include an analysis unit 1220, a factor updating unit 1222, a factor determination unit 1224, and a cost determination unit 1226, wherein:

[0264] The analyzing unit 1220 is configured to analyze, based on the correlation between the first stream executor and one or more second stream executors, whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of data streams by the second stream executors.

[0265] The factor updating unit 1222 is configured to update the impact factor according to a first window size; the first window size includes a window size determined according to a current window size of the first stream executor and the current window adjustment step size;

[0266] The factor determination unit 1224 is configured to determine whether the updated impact factor is within a confidence factor threshold range of the first stream executor.

[0267] The cost determining unit 1226 is configured to determine whether the first global window manager cost is less than the current global window manager cost, and whether the first global window manager cost is closer to 1 than the current global window manager cost;

[0268] The first global window manager cost includes a cost determined according to a first window size; the first window size includes a window size determined according to a current window size of the first stream executor and the current window adjustment step size.

[0269] In one implementation, the impact factor includes a local impact factor or a global impact factor;

[0270] The local impact factor is determined according to the window value interval and the current window size of the first stream executor;

[0271] The global impact factor is determined according to a global window manager cost and a local window manager cost of the first stream executor.

[0272] In one implementation, the analyzing unit 1220 may specifically include:

[0273] an adjustment coefficient determining unit, configured to determine a first adjustment coefficient based on a covariance between the first stream executor and the second stream executor, and a first window size; the first window size comprising a window size determined based on a current window size of the first stream executor and an adjustment step size of the current window;

[0274] a window size determining unit, configured to determine a second window size according to the first correlation coefficient between the first stream executor and the second stream executor, and the first adjustment coefficient;

[0275] a coefficient determination unit, configured to determine whether the first coefficient is 0 and whether the second window size exceeds a window value interval of the second stream executor;

[0276] If the coefficient determination unit determines that the first correlation coefficient is not 0, and the second window size exceeds the window value interval of the second stream executor, adjusting the current window size of the first stream executor based on the current window adjustment step size to affect the processing of the data stream by the second stream executor;

[0277] If the coefficient determination unit determines that the first correlation coefficient is 0, or the second window size does not exceed the window value range of the second stream executor, adjusting the current window size of the first stream executor based on the current window adjustment step does not affect the processing of the data stream by the second stream executor.

[0278] The window size determining unit may be specifically configured to:

[0279] determining a direction of window adjustment according to a first correlation coefficient between the first stream actuator and the second stream actuator;

[0280] The second window size is determined according to the first adjustment coefficient and the direction of the window adjustment.

[0281] In one implementation, the window control device 12 may further include:

[0282] The window size updating unit is configured to update the current window size of the first stream executor according to the current window adjustment step if the window determining unit determines to adjust the window size of the first stream executor.

[0283] It is understandable that the description of each unit in the window control device 12 can also refer to the embodiment of the aforementioned window control method, and will not be described in detail here.

[0284] like Figure 14 As shown, Figure 14 1 is a schematic diagram of the structure of a window control device provided in an embodiment of the present application. The window control device 140 includes at least one processor 1401 and at least one communication interface 1403, and optionally, at least one memory 1402. In addition, the device may also include common components such as an antenna, which will not be described in detail here.

[0285] The processor 1401 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above program.

[0286] The window control device includes a communication interface 1403, and the communication interface is used to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), core network, Wireless Local Area Networks (WLAN), etc.

[0287] If the window control device 140 includes a memory 1402, the memory may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory may be independent and connected to the processor via a bus. The memory may also be integrated with the processor.

[0288] The memory 1402 is used to store application code for executing the above solution, and the execution is controlled by the processor 1401. The processor 1401 is used to execute the application code stored in the memory 1402.

[0289] The code stored in the memory 1402 can be used to execute the above Figure 3-Figure 11 The steps in the provided window control method can be specifically referred to the implementation of the above method embodiment, which will not be described in detail here.

[0290] See also Figure 15 The embodiment of the present application further provides a chip 1500, comprising one or more processors 1501 and an interface circuit 1502. Optionally, the chip 1500 may further comprise a bus 1503.

[0291] The processor 1501 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 1501 or instructions in the form of software. The above-mentioned processor 1501 can be a general-purpose processor, a digital communicator (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods and steps disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0292] The interface circuit 1502 can complete the sending or receiving of data, instructions or information. The processor 1501 can use the data, instructions or other information received by the interface circuit 1502 to process it, and can send the processing completion information through the interface circuit 1502.

[0293] Optionally, the chip further includes a memory, which may include a read-only memory and a random access memory, and provides operating instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0294] Optionally, the memory stores an executable software module or data structure, and the processor can execute the above by calling the operation instructions stored in the memory (the operation instructions can be stored in the operating system). Figure 3-Figure 11 Steps in the provided window control method.

[0295] Optionally, the chip can be used in an electronic device, network device, or multi-node distributed system according to an embodiment of the present application. Optionally, the interface circuit 1502 can be used to output the execution result of the processor 1501. For the window control method provided in one or more embodiments of the present application, reference can be made to the aforementioned embodiments and will not be repeated here.

[0296] It should be noted that the corresponding functions of the processor 1501 and the interface circuit 1502 can be implemented through hardware design, software design, or a combination of hardware and software, and there is no limitation here.

[0297] The present application also provides a computer-readable storage medium having instructions stored therein that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of any of the aforementioned methods. If the various components of the aforementioned signal processing device are implemented as software functional units and sold or used as independent products, they may be stored in the computer-readable storage medium.

[0298] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0299] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A window control method, characterized in that: Applied to a multi-node distributed system, the method includes: Allocating stream executors in the multi-node distributed system to multiple data streams; at least two of the multiple data streams have different input rates; each node in the multi-node distributed system includes one or more stream executors; at least two of the stream executors in the multi-node distributed system have different processing rates; Determine whether to adjust the window size of the first stream executor based on the correlation between the first stream executor and one or more second stream executors; the first stream executor and the second stream executor are stream executors allocated in the multi-node distributed system, and the first stream executor and the second stream executor are on the same node or process the same data stream.

2. The method according to claim 1, wherein The determining whether to adjust the window size of the first stream executor according to the correlation between the first stream executor and the one or more second stream executors includes: According to the correlation between the first stream executor and one or more second stream executors, it is analyzed whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of the data stream by the second stream executor.

3. The method according to claim 2, wherein The analyzing, based on the correlation between the first stream executor and the one or more second stream executors, whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of the data stream by the second stream executor includes: Determining a first adjustment coefficient based on a covariance between the first stream executor and the second stream executor, and a first window size; the first window size includes a window size determined based on a current window size of the first stream executor and the current window adjustment step size; determining a second window size according to a first correlation coefficient between the first stream executor and the second stream executor, and the first adjustment coefficient; determining whether the first correlation coefficient is 0 and whether the second window size exceeds a window value interval of the second stream executor; If the first correlation coefficient is not 0, and the second window size exceeds the window value interval of the second stream executor, adjusting the current window size of the first stream executor based on the current window adjustment step size to affect the processing of the data stream by the second stream executor; If the first correlation coefficient is 0, or the second window size does not exceed the window value interval of the second stream executor, adjusting the current window size of the first stream executor based on the current window adjustment step does not affect the processing of the data stream by the second stream executor.

4. The method according to claim 3, wherein The determining the second window size according to the first correlation coefficient between the first stream executor and the second stream executor and the first adjustment coefficient includes: determining a direction of window adjustment according to a first correlation coefficient between the first stream executor and the second stream executor; A second window size is determined according to the first adjustment coefficient and the direction of the window adjustment.

5. The method according to any one of claims 2 to 4, characterized in that The determining whether to adjust the window size of the first stream executor according to the correlation between the first stream executor and the one or more second stream executors further includes: Updating the impact factor according to a first window size; the first window size includes a window size determined according to a current window size of the first stream executor and the current window adjustment step size; It is determined whether the updated impact factor is within a confidence factor threshold range of the first flow executor.

6. The method according to claim 5, wherein The impact factor includes a local impact factor or a global impact factor; The local impact factor is determined according to the window value interval and the current window size of the first stream executor; The global impact factor is determined based on a global window manager cost and a local window manager cost of the first stream executor.

7. The method according to any one of claims 2 to 4 or 6, wherein: The determining whether to adjust the window size of the first stream executor according to the correlation between the first stream executor and the one or more second stream executors further includes: determining whether a first global window manager cost is less than a current global window manager cost and whether the first global window manager cost is closer to 1 than the current global window manager cost; The first global window manager cost includes a cost determined according to a first window size; the first window size includes a window size determined according to a current window size of the first stream executor and the current window adjustment step size.

8. The method according to any one of claims 1 to 4 and 6, characterized in that: The method further comprises: If it is determined to adjust the window size of the first stream executor, the current window size of the first stream executor is updated according to the current window adjustment step size.

9. A window control device, characterized in that: include: A distribution unit, configured to distribute stream executors in a multi-node distributed system to multiple data streams; At least two of the plurality of data streams have different input rates; Each node in the multi-node distributed system includes one or more stream executors; at least two stream executors in the multi-node distributed system have different processing rates; A window determination unit is used to determine whether to adjust the window size of the first stream executor based on the correlation between the first stream executor and one or more second stream executors; the first stream executor and the second stream executor are stream executors allocated in the multi-node distributed system, and the first stream executor and the second stream executor are on the same node or process the same data stream.

10. The device according to claim 9, wherein The window determining unit includes: An analyzing unit is configured to analyze, based on a correlation between the first stream executor and one or more second stream executors, whether adjusting the current window size of the first stream executor based on the current window adjustment step size affects the processing of the data stream by the second stream executor.

11. The device according to claim 10, wherein The analysis unit comprises: an adjustment coefficient determining unit, configured to determine a first adjustment coefficient based on a covariance between the first stream executor and the second stream executor, and a first window size; the first window size comprising a window size determined based on a current window size of the first stream executor and the current window adjustment step size; a window size determining unit, configured to determine a second window size according to a first correlation coefficient between the first stream executor and the second stream executor, and the first adjustment coefficient; a coefficient determination unit, configured to determine whether the first correlation coefficient is 0 and whether the second window size exceeds a window value interval of the second stream executor; If the coefficient determination unit determines that the first correlation coefficient is not 0, and the second window size exceeds the window value interval of the second stream executor, adjusting the current window size of the first stream executor based on the current window adjustment step size to affect the processing of the data stream by the second stream executor; If the coefficient determination unit determines that the first correlation coefficient is 0, or the second window size does not exceed the window value interval of the second stream executor, then adjusting the current window size of the first stream executor based on the current window adjustment step does not affect the processing of the data stream by the second stream executor.

12. The device according to claim 11, wherein The window size determining unit is specifically configured to: determining a direction of window adjustment according to a first correlation coefficient between the first stream executor and the second stream executor; A second window size is determined according to the first adjustment coefficient and the direction of the window adjustment.

13. The device according to any one of claims 10 to 12, characterized in that The window determination unit further includes: a factor updating unit, configured to update an impact factor according to a first window size; the first window size comprising a window size determined according to a current window size of the first stream executor and the current window adjustment step size; A factor determination unit is configured to determine whether the updated impact factor is within a confidence factor threshold range of the first flow executor.

14. The device according to claim 13, wherein The impact factor includes a local impact factor or a global impact factor; The local impact factor is determined according to the window value interval and the current window size of the first stream executor; The global impact factor is determined based on a global window manager cost and a local window manager cost of the first stream executor.

15. The device according to any one of claims 10 to 12 and 14, characterized in that: The window determination unit further includes: a cost determining unit, configured to determine whether a first global window manager cost is less than a current global window manager cost, and whether the first global window manager cost is closer to 1 than the current global window manager cost; The first global window manager cost includes a cost determined according to a first window size; the first window size includes a window size determined according to a current window size of the first stream executor and the current window adjustment step size.

16. The device according to any one of claims 9 to 12 and 14, characterized in that: The device further comprises: A window size updating unit is configured to update the current window size of the first stream executor according to the current window adjustment step if the window determining unit determines to adjust the window size of the first stream executor.

17. A window control device, characterized in that: The method comprises a processor and a memory; the memory is used to store a program; the processor is used to execute the program stored in the memory, and when the program is executed, the processor is used to execute the method according to any one of claims 1 to 8.

18. A computer-readable storage medium storing instructions, wherein when the instructions are executed on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 8.

Images