Capacity expanding and shrinking method and device of stream processing system, computer program product and medium

Abstract

The invention discloses a capacity expanding and shrinking method and device of a stream processing system, a computer program product and a medium, and the capacity expanding and shrinking method of the stream processing system comprises the steps: obtaining a cloud resource configuration optimization target, wherein the cloud resource configuration optimization target comprises the maximum throughput and the shortest search time; and determining target cloud resource configuration corresponding to the cloud resource configuration optimization target. By adopting the scheme, the target cloud resource configuration with the maximum throughput can be found in the shortest search time.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for scaling up and down a stream processing system, a computer program product, and a medium. Background Art

[0002] Big data stream processing systems represented by Storm and Flink realize real-time processing of massive data through directed acyclic graphs (DAGs) composed of operators. The resource management module in the stream processing system is particularly important. It is responsible for determining the resource configuration (such as the number of containers, the number of CPU cores, the memory size) and the deployment location of the operator. However, the existing resource management module lacks efficient automatic elastic expansion and contraction strategies. Stream processing jobs are often pre-assigned with redundant computing resources, resulting in low efficiency in cloud resource allocation.

[0003] Therefore, how to design automatic elastic scaling algorithms for stream processing systems, how to optimize cloud resource configuration in real time as the workload changes for stream processing jobs, maximize job throughput and ensure high resource utilization has attracted widespread attention from researchers and developers. Summary of the invention

[0004] The present invention aims at providing at least a method for scaling up and down a stream processing system, which can shorten the time for searching for cloud resource configuration with maximum throughput.

[0005] In a first aspect, the present invention provides a method for scaling up and down a stream processing system, comprising: obtaining a cloud resource configuration optimization target, wherein the cloud resource configuration optimization target includes: maximum throughput and shortest search time; determining a target cloud resource configuration corresponding to the cloud resource configuration optimization target, comprising: obtaining candidate cloud resource configurations within the first t-1 time slots from the configured search space; obtaining the job throughput, throughput of each operator, and resource utilization of each operator obtained by running a target stream processing job using the candidate cloud resource configuration within the first t-1 time slot; t is a positive integer; determining the target stream processing job based on the job throughput, throughput of each operator, and resource utilization of each operator obtained by running the target stream processing job using the candidate cloud resource configuration within the first t-1 time slots. The bottleneck operator of the target stream processing job is determined; based on the bottleneck operator, the throughput of the bottleneck operator in the first t-1 time slots, and the resource utilization efficiency of the bottleneck operator in the first t-1 time slots, the predicted processing speed of the bottleneck operator is determined; based on the bottleneck operator and the predicted processing speed of the bottleneck operator, the candidate cloud resource configuration in the t-th time slot is selected from the search space; when the candidate cloud resource configuration in the t-th time slot is used to run the target stream processing job, the predicted processing speed of the bottleneck operator meets the traffic demand generated by the upstream operator; if the candidate cloud resource configuration in the t-th time slot is the same as the candidate cloud resource configuration in the t-1-th time slot, the candidate cloud resource configuration in the t-th time slot is used as the target cloud resource configuration.

[0006] Obtain a cloud resource configuration optimization target, which includes a maximum throughput and / or a shortest search time. Based on the cloud resource configuration optimization target, determine the corresponding target cloud resource configuration. Thus, the maximum throughput and the shortest search time are used as constraints, and the target cloud resource configuration finally obtained can obtain the maximum throughput in the shortest search time.

[0007] Optionally, the cloud resource configuration optimization goal is: Among them, x i (t) is the candidate cloud resource configuration of the ith operator in the tth time slot, x t ={x i (t)} i is the vector of candidate cloud resource configurations of all operators in the tth time slot, y t (x t )={y i (x i (t))} i For x t The vector of processing speeds of operators obtained by the corresponding candidate cloud resource configuration when running the target stream processing job, f t (y t (xt )) is a mapping function from the operator processing speed of the target stream processing job to the job throughput, Characterization x t The cumulative throughput of the target stream processing job can be maximized within the time from 1 to T.

[0008] Optionally, use the following formula to observe the operator processing speed yi(t): Among them, cpu i (x i (t)) is the i-th operator using x in the t-th time slot i (t) The CPU utilization obtained, is the throughput from the ith operator to the jth operator, c i (t) is the processing speed y of the ith operator in the tth time slot i (t) sampling, c i (t) is subject to the expectation y i (t) is a normal distribution.

[0009] Optional, y i The corresponding Gaussian process is GP(μ(x), k(x, x′)), where: μ(x) is y i The average value of k(x, x′) is y i The covariance of .

[0010] Optionally, determining the predicted processing speed of the bottleneck operator based on the bottleneck operator, the throughput of the bottleneck operator in the first t-1 time slots, and the resource utilization efficiency of the bottleneck operator in the first t-1 time slots includes: based on y i The corresponding Gaussian process GP(μ(x), k(x, x′)), and the candidate cloud resource configuration in the first t-1 time slots to run the target stream processing job, use the prediction operation to obtain the expected μ of the predicted processing speed t (x) and variance Among them, μ t (x) = k t (x) T (K t +σ 2 I) -1 y t , k t (x, x') = k(x, x') - k t (x)(K t +σ 2 I) -1 k t (x′); kt (x)=[k(x1,x),k(x2,x),...,k(x t , x)] T ; I is the unit matrix, K t is a positive definite kernel matrix A t is the set of candidate cloud resource configurations in the first t time slots, and A t ={x1, x2, ..., x t}, x1 is the candidate cloud resource configuration in the first time slot, x2 is the candidate cloud resource configuration adopted in the second time slot, x t is the candidate cloud resource configuration in the tth time slot, and x′ is the candidate cloud resource configuration in the next time slot after the tth time slot.

[0011] Optionally, the selecting, from the search space based on the bottleneck operator and the predicted processing speed of the bottleneck operator, a candidate cloud resource configuration in the t-th time slot comprises: selecting, from the search space, a candidate cloud resource configuration in the t-th time slot using the following formula: where Π χ It refers to selecting the candidate cloud resource configuration in the tth time slot from the search space X, β t is the upper confidence interval weight, β t =2log(|X|t 2 π 2 δ / 6), δ∈(1,∞); μ t-1 (x) is y i Corresponding to the expected posterior distribution, σ t-1 (x t ) is y i The posterior distribution of the corresponding variance.

[0012] In a second aspect, the present invention provides a device for scaling up and down a stream processing system, comprising: an acquisition unit, used to acquire a cloud resource configuration optimization target, wherein the cloud resource configuration optimization target comprises: maximum throughput and shortest search time; a determination unit, used to determine a target cloud resource configuration corresponding to the cloud resource configuration optimization target, comprising: acquiring candidate cloud resource configurations within the first t-1 time slots from the configured search space; acquiring the job throughput, throughput of each operator and resource utilization rate of each operator obtained by running the target stream processing job using the candidate cloud resource configuration within the first t-1 time slot; t is a positive integer; based on the job throughput, throughput of each operator and resource utilization rate of each operator obtained by running the target stream processing job using the candidate cloud resource configuration within the first t-1 time slot, determining The bottleneck operator of the target stream processing job; based on the bottleneck operator, the throughput of the bottleneck operator in the first t-1 time slots, and the resource utilization efficiency of the bottleneck operator in the first t-1 time slots, determine the predicted processing speed of the bottleneck operator; based on the bottleneck operator and the predicted processing speed of the bottleneck operator, select the candidate cloud resource configuration in the t-th time slot from the search space; when the candidate cloud resource configuration in the t-th time slot is used to run the target stream processing job, the predicted processing speed of the bottleneck operator meets the traffic requirements generated by the upstream operator; when it is detected that the candidate cloud resource configuration in the t-th time slot is the same as the candidate cloud resource configuration in the t-1-th time slot, the candidate cloud resource configuration in the t-th time slot is used as the target cloud resource configuration.

[0013] In a third aspect, the present invention provides a computer-readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for scaling up and down a stream processing system are executed.

[0014] In a fourth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-mentioned methods for scaling up and down a stream processing system.

[0015] In a fifth aspect, the present invention provides another device for scaling up and down a stream processing system, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor runs the computer program, the processor executes the steps of any of the above-mentioned methods for scaling up and down a stream processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a method for scaling up and down a stream processing system in an embodiment of the present invention;

[0017] Figure 2 is a flow chart for determining target cloud resource configuration in an embodiment of the present invention;

[0018] Figure 3 It is a structural schematic diagram of a capacity expansion and contraction device of a stream processing system in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In the prior art, for stream processing systems (such as Flink), how to automatically elastically scale capacity in real time with changes in traffic load is very important for stream processing jobs deployed on the cloud. However, the problem of automatic elastic scaling of stream processing jobs is fraught with difficulties. On the one hand, stream processing jobs perform parallel computing through a directed acyclic graph (DAG) composed of operators. Operators are interdependent, complex in structure, and link-sensitive. There may be hundreds of operators in a DAG, and each operator may have multiple upstream and downstream operators. The service quality (throughput and latency) of a single operator can greatly affect the overall service quality of the job. On the other hand, the service quality of an operator is determined by the resource configuration of the operator (such as the number of containers, the number of CPU cores, and the memory size). However, the mapping relationship between resource configuration and service quality is difficult to characterize (non-linear, multimodal). Adding an extra container or CPU core to an operator does not necessarily mean an improvement in service quality, but rather a decrease in resource utilization. At the same time, in cloud computing, computing resources are shared and oversold to multiple users. Therefore, the quality of service of a job is highly dynamic, and the throughput and latency of a job may vary by more than 20% under the same resource configuration. This makes the mapping relationship between resource configuration and service quality more difficult to predict.

[0020] In an embodiment of the present invention, a cloud resource configuration optimization target is obtained, and the cloud resource configuration optimization target includes a maximum throughput and / or a shortest search time. Based on the cloud resource configuration optimization target, a corresponding target cloud resource configuration is determined. Thus, the maximum throughput and the shortest search time are used as constraints, and the target cloud resource configuration finally obtained can obtain the maximum throughput within the shortest search time.

[0021] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0022] The embodiment of the present invention provides a method for scaling a stream processing system. Figure 1 The following is a detailed description through specific steps.

[0023] Step 101: Obtain cloud resource configuration optimization goals.

[0024] In a specific implementation, the cloud resource configuration optimization problem may be modeled first to obtain the cloud resource configuration optimization target. The cloud resource configuration optimization target may include: maximum throughput and shortest search time. In some embodiments, the cloud resource configuration optimization target includes both maximum throughput and shortest search time.

[0025] In the specific implementation, a stream processing system model can be built first. The stream processing system model can include the following three components: source operator, conversion operator and output operator. The source operator reads the real-time incoming stream data from the external message queue and sends the stream data to the downstream operator. The conversion operator can perform calculation operations on the stream data and output the calculation results of the stream data to the downstream operator. The output operator can receive the processed stream data and write it to the distributed file system (HDFS) or the remote dictionary service (Redis).

[0026] The downstream operator mentioned above may be a relative concept. For example, a conversion operator may be a downstream operator of a source operator, and accordingly, the upstream operator of the conversion operator is the source operator. For another example, an output operator may be a downstream operator of a conversion operator, and accordingly, the upstream operator of the output operator is the conversion operator.

[0027] In the embodiment of the present invention, it is set is the throughput from the ith operator to the jth operator. Represents the vector of throughputs of all upstream operators received by operator j. The mapping relationship between the inflow and outflow throughputs of operator i can be expressed by the following function:

[0028]

[0029] Among them, y i is the maximum processing speed of operator i. i，j (e i ) is the mapping relationship between the inflow and outflow throughput of operator i when the processing speed is not limited. In actual scenarios, h i，j (e i ) in the form of equation (1), the maximum processing speed y i Limitation (or truncation).

[0030] By repeatedly iterating the above formula (1), the cloud resource configuration optimization target can be obtained as:

[0031]

[0032] Among them, x i (t) is the candidate cloud resource configuration of the ith operator in the tth time slot, x t ={x i (t)}i is the vector of candidate cloud resource configurations of all operators in the tth time slot, y t (x t )={y i (x i (t))} i For x t The vector of processing speeds of operators obtained by the corresponding candidate cloud resource configuration when running the target stream processing job, f t (y t (x t )) is a mapping function from the operator processing speed of the target stream processing job to the job throughput, Characterization x t The cumulative throughput of the target stream processing job can be maximized within the time from 1 to T.

[0033] In a specific implementation, the following formula can be used to determine the processing speed y of the i-th operator in the t-th time slot: i (t):

[0034]

[0035] Among them, cpu i (x i (t)) is the number of times the ith operator uses x in the tth time slot. i (t) The CPU utilization obtained, is the throughput from the ith operator to the jth operator, c i (t) is y i (t) sampling, c i (t) is subject to the expectation y i (t) is a normal distribution.

[0036] In the specific implementation, y i The corresponding Gaussian process is GP(μ(x), k(x, x′)), where μ(x) is y i The average value of k(x, x′) is y i The covariance of .

[0037] Step 102: determine the target cloud resource configuration corresponding to the cloud resource configuration optimization target.

[0038] In an embodiment of the present invention, after determining the cloud resource configuration optimization target, the corresponding target cloud resource configuration is determined. The target cloud resource configuration can achieve the maximum throughput of the target stream processing job within the shortest search time.

[0039] Reference Figure 2 , a flow chart of determining target cloud resource configuration in an embodiment of the present invention is given. Figure 2 , step 102 is described in detail.

[0040] Step 121, obtaining the candidate cloud resource configurations within the first t-1 time slots from the configured search space.

[0041] In a specific implementation, there may be N candidate cloud resource configurations in the configured search space. From the configured search space, candidate cloud resource configurations within the first t-1 time slots may be selected. N and t are both positive integers, and N ≥ t.

[0042] For each time slot, there may be a corresponding candidate cloud resource configuration. Different time slots may correspond to different candidate cloud resource configurations.

[0043] In a specific implementation, a time slot may refer to a time period. For example, time t0 to time t1 is one time slot, and time t1 to time t2 is another time slot.

[0044] Step 122, obtaining the job throughput, the throughput of each operator, and the resource utilization rate of each operator obtained by configuring the candidate cloud resources in the first t-1 time slots to run the target stream processing job.

[0045] In a specific implementation, after selecting the candidate cloud resource configurations within the first t-1 time slots from the search space, the target stream processing job can be run using the candidate cloud resource configurations within the first t-1 time slots respectively to obtain the job throughput, throughput of each operator, and resource utilization of each operator corresponding to the candidate cloud resource configurations within the first t-1 time slots.

[0046] Step 123: determine the bottleneck operator of the target stream processing job.

[0047] In the embodiment of the present invention, the job throughput, the throughput of each operator, and the resource utilization rate of each operator obtained by running the target stream processing job based on the candidate cloud resource configuration in the first t-1 time slots can be used to model the online optimization problem. By solving the online optimization problem, the bottleneck operator of the target stream processing job can be obtained.

[0048] Step 124, determining the predicted processing speed of the bottleneck operator based on the bottleneck operator, the throughput of the bottleneck operator in the first t-1 time slots, and the resource utilization efficiency of the bottleneck operator in the first t-1 time slots.

[0049] After the bottleneck operator is determined, the throughput of the bottleneck operator in the first t-1 time slots and the resource utilization efficiency of the bottleneck operator in the first t-1 time slots can be obtained, and then the predicted processing speed of the bottleneck operator can be determined.

[0050] In the embodiment of the present invention, based on y iThe corresponding Gaussian process GP(μ(x), k(x, x′)), and the candidate cloud resource configuration in the first t-1 time slots to run the target stream processing job, use the prediction operation to obtain the expected μ of the predicted processing speed t (x) and variance

[0051] Among them, μ t (x) = k t (x) T (K t +σ 2 I) -1 y t , k t (x, x') = k(x, x') - k t (x)(K t +σ 2 I) -1 k t (x′); k t (x)=[k(x1,x),k(x2,x),…,k(x t , x)] T ; I is the unit matrix, K t is a positive definite kernel matrix A t is the set of candidate cloud resource configurations in the first t time slots, and A t ={x1, x2, ..., x t}, x1 is the candidate cloud resource configuration in the first time slot, x2 is the candidate cloud resource configuration adopted in the second time slot, x t is the candidate cloud resource configuration in the tth time slot, and x′ is the candidate cloud resource configuration in the next time slot.

[0052] Step 125 , based on the bottleneck operator and the predicted processing speed of the bottleneck operator, select a candidate cloud resource configuration in the t-th time slot from the search space.

[0053] Step 126: Use the candidate cloud resource configuration in the tth time slot as the target cloud resource configuration.

[0054] In a specific implementation, the following formula is used to select the candidate cloud resource configuration in the t-th time slot from the search space:

[0055]

[0056] where Π χ It refers to selecting the candidate cloud resource configuration in the tth time slot from the search space X, β t is the upper confidence interval weight, β t =2log(|X|t2 π 2 δ / 6), 6∈(1,∞); μ t-1 (x) is y i Corresponding to the expected posterior distribution, σ t-1 (x t ) is y i The posterior distribution of the corresponding variance.

[0057] In an embodiment of the present invention, if the candidate cloud resource configuration in the tth time slot is the same as the candidate cloud resource configuration in the t-1th time slot, the candidate cloud resource configuration in the tth time slot is used as the target cloud resource configuration.

[0058] In an embodiment of the present invention, the above-mentioned method for scaling up and down of a stream processing system may be performed by a scaling up and down device, which may be composed of an optimization engine, a database, a job monitor, and the like.

[0059] In the specific implementation, the optimization engine is responsible for determining the current operator resource configuration and communicating with Kubernetes and Flink to complete elastic scaling. It mainly includes the following three functions: using Kubernetes horizontal (HPA) and vertical (VPA) elastic scaling to adjust the number and size of TaskManager containers (CPU cores and memory size); using ApacheFlink Checkpoint to pause and restart stream processing jobs; when the stream processing job is paused, it communicates with the JobManager APIServer to assign TaskManager containers to different operators, and then restarts the stream processing job.

[0060] Database: Responsible for storing timestamped historical information, including operator input and output throughput, latency, and resource usage of pods and servers.

[0061] Job monitor: responsible for monitoring all operators and servers. It mainly includes the following two functions: using REST API to communicate with JobManager to monitor operator throughput and latency; using Kubernetes Metrics Server to monitor the computing resource usage of pods and servers.

[0062] Through the interaction between the above-mentioned expansion and contraction device and multiple systems such as Kubernetes and Flink, the ability of dynamic optimization of resource configuration is provided. The present invention can also be migrated to other stream processing systems such as Storm and Heron.

[0063] The method for scaling the stream processing system provided in the above embodiment of the present invention is run on the Kubernetes container management platform to optimize resource allocation for Flink stream processing jobs. The algorithm is verified by organizing a large number of real system experiments. In the Nexmark benchmark test and the Yahoo stream processing benchmark test, compared with the most advanced algorithm Dhalion, the scaling of the stream processing system can achieve a 1.8X-2.2X convergence speed acceleration, increase the throughput by 20.0% to 25.8%, and reduce the cloud computing resource cost by 14.6% to 15.6%.

[0064] Reference Figure 3 , a capacity expansion and contraction device 30 of a stream processing system in an embodiment of the present invention is provided, comprising: an acquisition unit 301 and a determination unit 302, wherein:

[0065] The acquisition unit 301 is used to acquire a cloud resource configuration optimization target, where the cloud resource configuration optimization target includes: maximum throughput and shortest search time;

[0066] The determination unit 302 is used to determine the target cloud resource configuration corresponding to the cloud resource configuration optimization target, including: obtaining the candidate cloud resource configuration in the first t-1 time slots from the configured search space; obtaining the job throughput, the throughput of each operator and the resource utilization rate of each operator obtained by running the target stream processing job using the candidate cloud resource configuration in the first t-1 time slot; t is a positive integer; based on the job throughput, the throughput of each operator and the resource utilization rate of each operator obtained by running the target stream processing job using the candidate cloud resource configuration in the first t-1 time slot, determining the bottleneck operator of the target stream processing job; based on the bottleneck operator, the bottleneck operator in the first t-1 time slot The predicted processing speed of the bottleneck operator is determined based on the throughput of the bottleneck operator and the resource utilization efficiency of the bottleneck operator in the first t-1 time slots; based on the bottleneck operator and the predicted processing speed of the bottleneck operator, the candidate cloud resource configuration in the t-th time slot is selected from the search space; when the candidate cloud resource configuration in the t-th time slot is used to run the target stream processing job, the predicted processing speed of the bottleneck operator meets the traffic demand generated by the upstream operator; when it is detected that the candidate cloud resource configuration in the t-th time slot is the same as the candidate cloud resource configuration in the t-1-th time slot, the candidate cloud resource configuration in the t-th time slot is used as the target cloud resource configuration.

[0067] In a specific implementation, the specific execution process of the acquisition unit 301 and the determination unit 302 may correspond to the above steps 101 to 102, which will not be described in detail here.

[0068] In a specific implementation, each module / unit included in each device or product described in the above embodiments may be a software module / unit or a hardware module / unit, or may be partly a software module / unit and partly a hardware module / unit.

[0069] For example, for each device or product applied to or integrated in a chip, each module / unit contained therein may be implemented in the form of hardware such as circuits, or at least some of the modules / units may be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits; for each device or product applied to or integrated in a chip module, each module / unit contained therein may be implemented in the form of hardware such as circuits, and different modules / units may be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units may be implemented in the form of software programs. The element can be implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.

[0070] An embodiment of the present invention also provides a computer-readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for scaling up and down the stream processing system provided in any of the above embodiments are executed.

[0071] An embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for scaling up and down a stream processing system provided in any of the above embodiments.

[0072] An embodiment of the present invention also provides another device for scaling up and down a stream processing system, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor runs the computer program, the steps of the method for scaling up and down a stream processing system provided in any of the above embodiments are executed.

[0073] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which may include: ROM, RAM, disk or CD, etc.

[0074] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A method for scaling up and down a stream processing system, characterized in that: include: Obtaining cloud resource configuration optimization goals, where the cloud resource configuration optimization goals include: maximum throughput and shortest search time; Determining a target cloud resource configuration corresponding to the cloud resource configuration optimization target includes: From the configured search space, obtain the candidate cloud resource configurations within the first t-1 time slots; Get the job throughput, throughput of each operator, and resource utilization of each operator obtained by running the target stream processing job in the candidate cloud resource configuration in the first t-1 time slots; t is a positive integer; Determine the bottleneck operator of the target stream processing job based on the job throughput, the throughput of each operator, and the resource utilization rate of each operator obtained by running the target stream processing job using the candidate cloud resource configuration in the first t-1 time slots; Determining a predicted processing speed of the bottleneck operator based on the bottleneck operator, the throughput of the bottleneck operator in the first t-1 time slots, and the resource utilization efficiency of the bottleneck operator in the first t-1 time slots; Based on the bottleneck operator and the predicted processing speed of the bottleneck operator, selecting a candidate cloud resource configuration in the t-th time slot from the search space; when the candidate cloud resource configuration in the t-th time slot is used to run the target stream processing job, the predicted processing speed of the bottleneck operator meets the traffic demand generated by the upstream operator; If the candidate cloud resource configuration in the t-th time slot is the same as the candidate cloud resource configuration in the t-1-th time slot, the candidate cloud resource configuration in the t-th time slot is used as the target cloud resource configuration.

2. The method for scaling up and down a stream processing system according to claim 1, wherein: The cloud resource configuration optimization goal is: Among them, x i (t) is the candidate cloud resource configuration of the ith operator in the tth time slot, x t ={x i (t)} i is the vector of candidate cloud resource configurations of all operators in the tth time slot, y t (x t )={y i (x i (t))} i For x t The vector of processing speeds of operators obtained by the corresponding candidate cloud resource configuration when running the target stream processing job, f t (y t (x t )) is a mapping function from the operator processing speed of the target stream processing job to the job throughput, Characterization x t The cumulative throughput of the target stream processing job can be maximized within the time from 1 to T.

3. The method for scaling up and down a stream processing system according to claim 2, wherein: The operator processing speed y is determined using the following formula: i (t): Among them, cpu i (x i (t)) is the i-th operator using x in the t-th time slot i (t) The CPU utilization obtained, is the throughput from the ith operator to the jth operator, c i (t) is the processing speed y of the ith operator in the tth time slot i (t) sampling, c i (t) is subject to the expectation y i (t) is a normal distribution.

4. The method for scaling up and down a stream processing system according to claim 3, wherein: y i The corresponding Gaussian process is GP(μ(x),k(x,x′)), where: μ(x) is y i The average value of k(x,x′) is y i The covariance of .

5. The method for scaling up and down a stream processing system according to claim 4, wherein: The determining of the predicted processing speed of the bottleneck operator based on the bottleneck operator, the throughput of the bottleneck operator in the first t-1 time slots, and the resource utilization efficiency of the bottleneck operator in the first t-1 time slots comprises: i The corresponding Gaussian process GP(μ(x), k(x, x′)), and the candidate cloud resource configuration in the first t-1 time slots to run the target stream processing job, use the prediction operation to obtain the expected μ of the predicted processing speed t (x) and variance Among them, m t (x)=k t (x) T (K t +s 2 I) -1 y t ,k t (x,x′)=k(x,x′)-k t (x)(K t +s 2 I) -1 k t (x′); k t (x) = [k(x x ,x),k(x2,x),…,k(x t ,x)] T ; I is the unit matrix, K t is a positive definite kernel matrix A t is the set of candidate cloud resource configurations in the first t time slots, and A t ={x1,x2,…,x t }, x1 is the candidate cloud resource configuration in the first time slot, x2 is the candidate cloud resource configuration adopted in the second time slot, x t is the candidate cloud resource configuration in the tth time slot, x ′ The candidate cloud resource configuration in the next time slot after the t-th time slot.

6. The automatic elastic expansion and contraction algorithm of the stream processing system according to claim 5, characterized in that: The selecting, from the search space, a candidate cloud resource configuration in the t-th time slot based on the bottleneck operator and the predicted processing speed of the bottleneck operator includes: The candidate cloud resource configuration in the t-th time slot is selected from the search space using the following formula: where Π χ It refers to selecting the candidate cloud resource configuration in the tth time slot from the search space X, β t is the upper confidence interval weight, β t =2log(|X|t 2 π 2 δ / 6), δ∈(1,∞); μ t-1 (x) is y i Corresponding to the expected posterior distribution, σ t-1 (x t ) is y i The posterior distribution of the corresponding variance.

7. A device for expanding and contracting a stream processing system, characterized in that: include: An acquisition unit, used to acquire a cloud resource configuration optimization target, wherein the cloud resource configuration optimization target includes: maximum throughput and shortest search time; A determination unit is used to determine the target cloud resource configuration corresponding to the cloud resource configuration optimization target, including: obtaining candidate cloud resource configurations within the first t-1 time slots from the configured search space; obtaining the job throughput, throughput of each operator and resource utilization rate of each operator obtained by running the target stream processing job using the candidate cloud resource configuration within the first t-1 time slot; t is a positive integer; based on the job throughput, throughput of each operator and resource utilization rate of each operator obtained by running the target stream processing job using the candidate cloud resource configuration within the first t-1 time slot, determining the bottleneck operator of the target stream processing job; based on the bottleneck operator, the bottleneck operator within the first t-1 time slot The predicted processing speed of the bottleneck operator is determined based on the throughput of the bottleneck operator and the resource utilization efficiency of the bottleneck operator in the first t-1 time slots; based on the bottleneck operator and the predicted processing speed of the bottleneck operator, the candidate cloud resource configuration in the t-th time slot is selected from the search space; when the candidate cloud resource configuration in the t-th time slot is used to run the target stream processing job, the predicted processing speed of the bottleneck operator meets the traffic demand generated by the upstream operator; when it is detected that the candidate cloud resource configuration in the t-th time slot is the same as the candidate cloud resource configuration in the t-1-th time slot, the candidate cloud resource configuration in the t-th time slot is used as the target cloud resource configuration.

8. A computer-readable storage medium, wherein the computer-readable storage medium is a non-volatile storage medium or a non-transient storage medium, and a computer program is stored thereon, wherein: When the computer program is executed by a processor, the steps of the method for scaling up and down a stream processing system according to any one of claims 1 to 6 are executed.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method for scaling up and down a stream processing system according to any one of claims 1 to 6 are implemented.

10. A device for scaling a stream processing system, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the processor executes the steps of the method for scaling up and down a stream processing system according to any one of claims 1 to 6.

Images