Elastic expansion and contraction method, electronic device and storage medium for stream processing platform
By predicting traffic peak and valley time points based on historical traffic, expanding capacity and reducing the servers and partitions of the stream processing platform in advance, the insufficient processing capacity and resource waste caused by traffic fluctuations are solved, and more efficient resource utilization is achieved.
Patent Information
- Application Number
- CN202011206048.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-02
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2040-11-02
AI Technical Summary
The stream processing platform lacks servers during peak traffic requests, resulting in insufficient processing capabilities; server redundancy during trough periods leads to waste of resources.
Based on historical traffic prediction, the server is expanded in advance and partitions are migrated to improve processing power, and the server is reduced to reduce resource waste when the traffic valley is reduced.
It improves the processing capability of the stream processing platform at traffic peaks, and reduces server resource waste during traffic valleys, achieving flexible response to traffic fluctuations.
Smart Images
Figure CN112445613B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an elastic scaling method, electronic device and storage medium for a stream processing platform. Background Art
[0002] Stream processing platforms can process all action stream data of users on websites, so they are widely used. However, when deploying stream processing platforms, the number of servers is usually fixed. Therefore, when the number of traffic requests increases during peak hours, the servers in the stream processing platform may not be enough to handle the sudden increase in requests. When the number of traffic requests is small during low hours, the server redundancy of the stream processing platform will lead to resource waste. Summary of the invention
[0003] The main technical problem solved by the present application is to provide an elastic scaling method, electronic device and storage medium for a stream processing platform, which can expand the server and migrate partitions based on the predicted traffic before the peak traffic of the stream processing platform arrives to improve the processing capacity of the stream processing platform, and reduce the server and migrate partitions after the valley traffic of the stream processing platform arrives to reduce the waste of server resources.
[0004] To solve the above technical problems, the first aspect of the present application provides a method for elastic scaling of a stream processing platform, the method comprising: obtaining the predicted traffic corresponding to the stream processing platform in each time period in the current time period based on the historical traffic of the stream processing platform before the current time period; determining the valley time point and the peak time point from the current time period according to the predicted traffic; when the current time point is a first preset time point before the peak time point, expanding the server for the stream processing platform, and transferring part of the partitions deployed in the server of the stream processing platform to the expanded server; when the current time point is a second preset time point after the valley time point, reducing the server for the stream processing platform, and transferring the partitions deployed in the server to be reduced to the server to be retained.
[0005] Among them, the step of obtaining the corresponding predicted traffic of the stream processing platform in each time period in the current time period based on the historical traffic of the stream processing platform before the current time period includes: obtaining multiple first traffic samples of the server of the stream processing platform in the same time period in multiple time periods before the current time period; using the multiple first traffic samples to obtain the first traffic average value of the stream processing platform in each time period in the time period, and using the first traffic average value as the predicted traffic of the stream processing platform in each time period in the current time period.
[0006] Among them, the step of obtaining the predicted traffic corresponding to each time period in the current time period based on the historical traffic of the stream processing platform before the current time period includes: obtaining multiple second traffic samples in the same time period of each partition in the server of the stream processing platform in multiple time periods before the current time period; adding the second traffic samples in the same time period of all the partitions in the same server in the same time period to obtain the first traffic sample of the server in the same time period in the same time period; using the multiple first traffic samples corresponding to the multiple time periods before the current time period to obtain the first traffic average value of the stream processing platform in each time period in the time period, and using the first traffic average value as the predicted traffic of the stream processing platform in each time period in the current time period.
[0007] The method further includes: obtaining multiple second traffic samples in the same time period of each partition in multiple time periods before the current time period in each partition of the server; using the multiple second traffic samples to obtain the second traffic average value of each partition in each time period in the time period, and using the second traffic average value as the estimated traffic of the partition in each time period in the current time period.
[0008] Among them, the step of transferring part of the partitions deployed in the servers of the stream processing platform to the expanded servers includes: transferring part of the partitions deployed in the servers of the stream processing platform to the expanded servers in a preset manner according to the estimated traffic, so that the maximum traffic difference between all the servers in the current stream processing platform is less than a first threshold; the step of transferring the partitions deployed in the servers to be reduced to the servers to be retained includes: transferring the partitions deployed in the servers to be reduced to the servers to be retained in a preset manner according to the estimated traffic, so that the maximum traffic difference between all the servers in the current stream processing platform is less than a second threshold.
[0009] Among them, the step of transferring part of the partitions deployed in the servers of the stream processing platform to the expanded servers in a preset manner according to the estimated traffic includes: transferring the part of the partitions to the expanded servers according to the estimated traffic so that the variance between the estimated traffic corresponding to all servers in the current stream processing platform is minimized; the step of transferring the partitions deployed in the servers to be reduced to the servers to be retained in a preset manner according to the estimated traffic includes: transferring the partitions deployed in the servers to be reduced to the servers to be retained according to the estimated traffic so that the variance between the estimated traffic corresponding to all servers in the current stream processing platform is minimized; wherein, the estimated traffic is the sum of the estimated traffic of all partitions on the current server.
[0010] Among them, the step of determining the valley time point and the peak time point from the current time period according to the predicted traffic includes: sorting and numbering the predicted traffic of the stream processing platform in each time period in the current time period in ascending order according to the numerical value of the traffic; multiplying the maximum number by the first percentage and rounding to obtain a first integer; taking the time point corresponding to the first integer number of predicted traffic among all the sorted predicted traffic as the valley time point, and taking the time point corresponding to the last integer number of predicted traffic among all the sorted predicted traffic as the peak time point.
[0011] Among them, the step of determining the valley time point and the peak time point from the current time period according to the predicted traffic includes: using the predicted traffic in each time period in the current time period to obtain the predicted traffic average for a single time period in the current time period; multiplying the predicted traffic average by a second percentage to obtain a valley threshold, and multiplying the predicted traffic average by a third percentage to obtain a peak threshold; taking the time points corresponding to all predicted traffic less than the valley threshold as the valley time points, and taking the time points corresponding to all predicted traffic greater than the peak threshold as the peak time points.
[0012] In order to solve the above-mentioned technical problems, the second aspect of the present application provides an electronic device, which includes a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the elastic scaling method for a stream processing platform of the above-mentioned first aspect.
[0013] In order to solve the above technical problems, the third aspect of the present application provides a computer storage medium, on which program data is stored. When the program data is executed by a processor, the elastic scaling method for a stream processing platform of the first aspect is implemented.
[0014] The beneficial effects of the present application are as follows: the present application estimates the corresponding predicted traffic in each time period in the current time period based on the historical traffic of the stream processing platform before the current time period, determines the valley time points and peak time points in the predicted traffic based on the periodicity of the traffic, expands the server before the traffic peak of the stream processing platform arrives and migrates some partitions to the expanded server in advance, thereby improving the processing capacity of the stream processing platform, and reduces the server after the traffic valley of the stream processing platform arrives and transfers the partitions in the server to be reduced to the server to be retained to reduce the waste of server resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:
[0016] Figure 1 It is a flow chart of an implementation method of an elastic expansion and contraction method for a stream processing platform provided by the present application;
[0017] Figure 2 yes Figure 1 A schematic flow chart of an implementation method corresponding to step S101;
[0018] Figure 3 It is a flow chart of another implementation method of the elastic expansion and contraction method for a stream processing platform provided by the present application;
[0019] Figure 4 yes Figure 3 A schematic diagram of a flow chart of an implementation method corresponding to step S304;
[0020] Figure 5 yes Figure 3 A flowchart of another implementation method corresponding to step S304 in FIG.
[0021] Figure 6 It is a structural schematic diagram of an implementation method of partition transfer when expanding a server;
[0022] Figure 7 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;
[0023] Figure 8 It is a structural schematic diagram of an implementation scheme of a computer storage medium provided in the present application. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0025] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.
[0026] See also Figure 1 , Figure 1 : is a flow chart of an implementation method of an elastic expansion and contraction method for a stream processing platform provided by the present application, the method comprising:
[0027] Step S101: Based on the historical traffic of the stream processing platform before the current time period, obtain the corresponding predicted traffic of the stream processing platform in each time period in the current time period.
[0028] Specifically, a stream processing platform such as Kafka is a distributed publish-subscribe messaging system that can process all action stream data of consumers on a website. Kafka has the characteristics of high throughput and horizontal scalability, so it is widely used. For the deployed stream processing platform, a period of time is set as the time period, and the historical traffic of the stream processing platform in different time periods in multiple time periods is recorded. Based on the periodicity of traffic repetition on the network, the traffic of the stream processing platform in the current time period is predicted based on the historical traffic to obtain the predicted traffic.
[0029] In one application, a day is used as a time period, and the time period is divided into multiple time periods. The traffic of the deployed stream processing platform in multiple periods is recorded, and the traffic of the same time period in multiple periods is superimposed and the average value is taken as the predicted traffic of the corresponding time period. Since the website is visited repeatedly in the time period, the traffic in each time period in the current time period can be predicted by recording the historical traffic before the current time period and using the historical traffic.
[0030] Specifically, see Figure 2 , Figure 2 yes Figure 1 The flowchart of an implementation method corresponding to step S101 in FIG. 1 is a flowchart of an implementation method corresponding to step S101 in FIG. 1 , wherein step S101 specifically includes:
[0031] Step S201: Acquire a plurality of first traffic samples of a server of a stream processing platform in the same period in a plurality of time periods before a current time period.
[0032] Specifically, the stream processing platform includes at least one server, and takes the current time period as a node, obtains the traffic in the same time period in the time period before the current time period for each server of the stream processing platform, records it as the first traffic sample, and records multiple first traffic samples in the same time period in multiple time periods before the current time period. The most accurate first traffic sample corresponding to the current server can be obtained by directly reading the traffic corresponding to the server from the server.
[0033] In a specific application scenario, a 24-hour day is used as a time period, and the time period is divided into 12 time periods, each of which is 2 hours. In a single time period, the average flow rate of each server of the deployed stream processing platform in each time period is recorded as the first flow sample. The flow rates of all time periods in multiple time periods are recorded to obtain multiple first flow samples in multiple time periods before the current time period.
[0034] Step S202: using a plurality of first traffic samples, obtaining a first traffic average value of the flow processing platform in each time period in the time cycle, and using the first traffic average value as the predicted traffic of the flow processing platform in each time period in the current time cycle.
[0035] Specifically, the traffic of each time period in multiple time periods before the current time period of the stream processing platform is averaged to obtain an average value, and the periodicity of traffic changes on the stream processing platform is used to take the average value as the predicted traffic of the current time period.
[0036] In one application method, multiple first traffic samples of the same server in the same time period in multiple time cycles are added together and then divided by the corresponding number of time periods to obtain a first average value, the first average values of all servers in the stream processing platform in the same time period are added together to obtain the first traffic average value, the corresponding first traffic average values in all time periods are calculated, and then the first traffic average value is used as the predicted traffic of the stream processing platform in each time period in the current time cycle.
[0037] Step S102: Determine the valley time point and the peak time point in the current time period according to the predicted flow.
[0038] Specifically, obtain the predicted traffic in each time period in the current time cycle, obtain the highest and lowest values therein, subtract the first value from the highest value to obtain the second value, add the first value to the lowest value to obtain the third value, and take the time point corresponding to the time period when the value of the predicted traffic in each time period in the current time cycle is greater than the second value as the peak time point, and take the time point corresponding to the time period when the value of the predicted traffic in each time period in the current time cycle is less than the third value as the valley time point.
[0039] Step S103: When the current time point is the first preset time point before the peak time point, a server is expanded for the stream processing platform, and part of the partitions deployed in the server of the stream processing platform are transferred to the expanded server.
[0040] Specifically, when the current time point reaches a first preset time point before the peak time point, a server is added to the stream processing platform to expand the capacity of the stream processing platform, wherein the server includes at least one partition and a server without a partition cannot receive requests. Therefore, before the peak time point arrives, some partitions in the servers deployed on the stream processing platform are transferred to the expanded servers, so that partitions exist on the expanded servers before the peak time point arrives, and then all servers in the stream processing platform are available after the peak time point arrives, thereby improving the processing capacity of the stream processing platform.
[0041] Furthermore, since partition migration will affect the processing efficiency of the server, starting the migration at the first preset time point before the peak time point can allow partition migration to be within the period before the peak time point, thereby reducing the impact of partition migration on the processing efficiency of the stream processing platform.
[0042] Step S104: when the current time point is a second preset time point after the valley time point, reduce the server for the stream processing platform, and transfer the partitions deployed in the server to be reduced to the server to be retained.
[0043] Specifically, when the current time point reaches a second preset time point after the valley time point, servers are reduced in the stream processing platform to shrink the capacity of the stream processing platform, servers to be reduced are selected and partitions in the servers to be reduced are transferred to servers to be retained to reduce waste of server resources.
[0044] Furthermore, when the predicted traffic reaches a valley time point and continues to remain at a low value, the server to be reduced is selected and the partition on the server to be reduced is transferred to the server to be retained. When the second preset time point after the valley time point is reached, the server to be reduced whose partition has been removed is removed from the stream processing platform to save server resources of the stream processing platform.
[0045] In a specific application scenario, the stream processing platform kafka includes multiple server brokers, each server broker includes at least one partition, and a 24-hour day is used as a time period, and the time period is equally divided into 12 periods, each period is 2 hours, the first preset time point is 1 hour before the peak time point, and the second preset time point is 1 hour after the valley time point. The flow data of each period in multiple time periods before the current time period is used to calculate the average flow of each period, and the average flow in each period is used as the predicted flow of the corresponding period in the current time period. The maximum value in the predicted flow is subtracted from a fixed value to determine the peak threshold and then determine the peak time point. The minimum value in the predicted flow is added to a fixed value to determine the valley threshold and then determine the valley time point. One hour before the peak time point arrives, the stream processing platform kafka expands the server broker, and some partitions are transferred from the existing server broker to the expanded server broker. One hour after the valley time point arrives, some server brokers in the current stream processing platform kafka are reduced, and the partitions on the reduced server broker are transferred to the reserved server broker.
[0046] The elastic scaling method for a stream processing platform provided in this embodiment estimates the corresponding predicted traffic in each time period in the current time period based on the historical traffic of the stream processing platform before the current time period, determines the valley time points and peak time points in the predicted traffic based on the periodicity of the traffic, expands the server before the traffic peak of the stream processing platform arrives and migrates some partitions to the expanded server in advance, thereby improving the processing capacity of the stream processing platform, and reduces the server after the traffic valley of the stream processing platform arrives and transfers the partitions in the server to be reduced to the server to be retained, thereby reducing the waste of server resources.
[0047] See also Figure 3 , Figure 3 : is a flow chart of another embodiment of the elastic expansion and contraction method for a stream processing platform provided by the present application, the method comprising:
[0048] Step S301: Acquire multiple second traffic samples in the same time period in multiple time periods before the current time period for each partition in the server of the stream processing platform.
[0049] Specifically, the stream processing platform includes at least one server, each server includes at least one partition, and taking the current time period as a node, the traffic in the same time period in the time period before the current time period is obtained for all partitions corresponding to each server in the stream processing platform, which is recorded as the second traffic sample, and multiple second traffic samples of the same time period in multiple time periods before the current time period are recorded.
[0050] Step S302: Add the second traffic samples of all partitions in the same server in the same time period in the same time cycle to obtain the first traffic samples of the server in the same time period in the same time cycle.
[0051] Specifically, the flow corresponding to the server can be obtained by superimposing the flows of all partitions on the server, and the second flows of all partitions on the same server in the same time period in the same time period are added together to obtain the first flow sample of the server in the same time period in the time period. Repeat the above steps S301-S302 for multiple time periods before the current time period, and multiple first flow samples of the server in different time periods in different time periods can be obtained.
[0052] Step S303: Utilize multiple first traffic samples corresponding to multiple time periods before the current time period to obtain the first traffic average value of the flow processing platform in each time period in the time period, and use the first traffic average value as the predicted traffic of the flow processing platform in each time period in the current time period.
[0053] Specifically, the traffic of each time period in multiple time periods before the current time period of the stream processing platform is averaged to obtain an average value, and based on the periodic changes in traffic changes on the stream processing platform, the calculated average value is used as the predicted traffic for the current time period.
[0054] In one application method, all first traffic samples of all servers in the stream processing platform in the same time period are added together to obtain a first added value, the first added value is divided by the corresponding number of time periods to obtain a first traffic average value, the corresponding first traffic average values in all time periods are calculated, and then the first traffic average value is used as the predicted traffic of the stream processing platform in each time period in the current time period.
[0055] Step S304: Determine the valley time point and the peak time point in the current time period according to the predicted flow.
[0056] Specifically, in the predicted traffic in each time period of the current time cycle, the time points corresponding to the smaller and larger values need to be selected to determine the valley time point and the peak time point.
[0057] In one application, see Figure 4 , Figure 4 yes Figure 3 A flowchart of an implementation method corresponding to step S304 in FIG. 1 is shown in FIG. 1 , wherein step S304 includes:
[0058] Step S401: sort and number the predicted traffic of the stream processing platform in each time period in the current time period in ascending order according to the numerical value of the traffic.
[0059] Specifically, after the predicted traffic in each time period in the current time cycle is obtained, the predicted traffic in each time period is sorted in ascending order according to the numerical value and numbered.
[0060] Step S402: multiply the maximum number by a first percentage and round it to obtain a first integer.
[0061] Specifically, the largest number is multiplied by a preset first percentage (eg, 20%, 30%), and then the corresponding value is rounded to obtain a first integer.
[0062] Step S403: The time point corresponding to the first integer number of predicted flows among all the sorted predicted flows is taken as the valley time point, and the time point corresponding to the last integer number of predicted flows among all the sorted predicted flows is taken as the peak time point.
[0063] Specifically, for example, a time period is divided into 12 time periods, and the first percentage is 20%. The time points of the time periods corresponding to the two predicted traffic with the smallest predicted traffic values in the 12 time periods are taken as valley time points, and the time points of the time periods corresponding to the two predicted traffic with the largest predicted traffic values are taken as peak time points. The peak time points and valley time points can be quickly found in the above manner, and if the number of peak time points and valley time points is the same, the number of expansion and reduction of servers in the stream processing platform is the same, and thus the number of servers in the stream processing platform is relatively stable between different periods.
[0064] In a specific application scenario, a day of 24 hours is used as a time cycle, and the time cycle is divided into 12 time periods, each of which is 2 hours. Please combine Table 1 and Table 2. Table 1 is a distribution table of the predicted traffic of the stream processing platform in each time period in the current time cycle, and Table 2 is a sorting table of the predicted traffic of the stream processing platform in each time period in the current time cycle. Among them, the data in Table 2 can be obtained by arranging the data in Table 1 in ascending order. The integer obtained by multiplying the largest number 12 in Table 2 by 30% is 3, and then the time periods corresponding to the smallest three predicted traffic are obtained (0-2, 2-4, 22-24), then the valley time points are 0 o'clock, 2 o'clock and 22 o'clock, and the time periods corresponding to the largest three predicted traffic are obtained (10-12, 12-14, 14-16), then the peak time points are 10 o'clock, 12 o'clock and 14 o'clock.
[0065] Table 1: Distribution of traffic predicted by the stream processing platform in each time period during the current time period
[0066] Time 0-2 2-4 4-6 6-8 8-10 10-12 12-14 14-16 16-18 18-20 20-22 22-24 flow 0.2 0.3 0.4 0.8 1.6 1.8 2.0 1.7 1.6 1.4 1.2 0.3
[0067] Table 2: Ranking table of traffic predictions of the stream processing platform in each time period in the current time period
[0068] serial number 1 2 3 4 5 6 7 8 9 10 11 12 flow 0.2 0.3 0.3 0.4 0.8 1.2 1.4 1.6 1.6 1.7 1.8 2.0
[0069] It is understandable that in other embodiments, the predicted traffic in each time period is sorted in descending order according to the numerical value and numbered, and the valley time point and the peak time point can also be determined in the same manner.
[0070] In another application, see Figure 5 , Figure 5 yes Figure 3 A flowchart of another implementation method corresponding to step S304 in FIG. 1 is shown in FIG. 1 , wherein step S304 includes:
[0071] Step S501: using the predicted traffic in each time period in the current time period, obtain the average predicted traffic in a single time period in the current time period.
[0072] Specifically, all predicted flows in each time period in the current time cycle are added together and then divided by the number of time periods to obtain the predicted flow mean.
[0073] Step S502: multiply the predicted traffic mean by the second percentage to obtain a valley threshold, and multiply the predicted traffic mean by the third percentage to obtain a peak threshold.
[0074] Specifically, the predicted flow mean is multiplied by a preset second percentage (eg, 40%, 50%, 60%) to obtain a valley threshold, and the predicted flow is multiplied by a preset third percentage (eg, 140%, 150%, 160%) to obtain a peak threshold.
[0075] Step S503: taking all time points corresponding to predicted flows less than the valley threshold as valley time points, and taking all time points corresponding to predicted flows greater than the peak threshold as peak time points.
[0076] Specifically, the predicted flow in each time period in the current time cycle is compared with the valley threshold and the peak threshold. For the predicted flow less than the valley threshold, the predicted flow value of the corresponding time period is determined to be small, and for the predicted flow greater than the valley threshold, the predicted flow value of the corresponding time period is determined to be large. In the above manner, when determining the valley time point and the peak time point, the relationship between the flow values can be emphasized, so that the valley threshold and the peak threshold are closer to the higher and lower parts of the predicted flow, and thus the valley time point and the peak time point are more accurate.
[0077] In a specific application scenario, 24 hours a day is taken as a time period, and the time period is equally divided into 12 time periods, each of which is 2 hours. Please refer to Table 1 again, and the average value of all predicted traffic in Table 1 is added to obtain the average value of 1.1. The average value 1.1 is multiplied by 40%, and the valley threshold value is 0.44. The average value 1.1 is multiplied by 160%, and the peak threshold value is 1.76. The values less than the valley threshold value are 0.2, 0.3 and 0.4, and the corresponding time periods are 0-2, 2-4, 4-6 and 22-24. The valley time points are 0 o'clock, 2 o'clock, 4 o'clock and 22 o'clock. The values greater than the peak threshold value are 1.8 and 2.0, and the corresponding time periods are 10-12 and 12-14. The peak time points are 10 o'clock and 12 o'clock.
[0078] Furthermore, the elastic scaling method for the stream processing platform provided in this embodiment also includes: using multiple second traffic samples to obtain the second traffic average value of each partition in each time period in the time period, and using the second traffic average value as the estimated traffic of the partition in each time period in the current time period.
[0079] Specifically, in step S301, a plurality of second flow samples have been obtained, and then the second flow samples in a plurality of different time periods can be used to obtain the second flow average value for each time period within the length of a time period, and the second flow average value is used as the estimated flow of the partition on the server in each time period in the current time period. The step of obtaining the estimated flow can be performed after any step in steps S301-S304.
[0080] Furthermore, in this embodiment, the second traffic sample corresponding to the partition is directly obtained in step S301. The second traffic sample can be used to obtain the first traffic sample corresponding to the server through calculation, or the estimated traffic corresponding to the partition can be obtained through calculation, thereby reducing the number of times data is directly read and improving the system's operating speed.
[0081] Step S305: When the current time point is the first preset time point before the peak time point, a server is expanded for the stream processing platform, and part of the partitions deployed in the server of the stream processing platform are transferred to the expanded server.
[0082] Specifically, when the current time point reaches a first preset time point before the peak time point, servers are added to the stream processing platform to expand the capacity of the stream processing platform, and some partitions in the servers deployed on the stream processing platform are transferred to the expanded servers.
[0083] Step S306: When the current time point is a second preset time point after the valley time point, reduce the server for the stream processing platform, and transfer the partitions deployed in the server to be reduced to the server to be retained.
[0084] Specifically, when the current time point reaches a second preset time point after the valley time point, servers are reduced in the stream processing platform to shrink the capacity of the stream processing platform, servers to be reduced are selected and partitions in the servers to be reduced are transferred to servers to be retained.
[0085] In a specific application scenario, the step of transferring some partitions deployed in the server of the stream processing platform to the expanded server includes: transferring some partitions deployed in the server of the stream processing platform to the expanded server in a preset manner according to the estimated traffic, so that the maximum traffic difference between all servers in the current stream processing platform is less than a first threshold. The step of transferring partitions deployed in the server to be reduced to the server to be retained includes: transferring partitions deployed in the server to be reduced to the server to be retained in a preset manner according to the estimated traffic, so that the maximum traffic difference between all servers in the current stream processing platform is less than a second threshold.
[0086] Specifically, the traffic of the partitions of the server in each time period of different time periods before the current time period is obtained, and then the maximum traffic difference between the partitions is used as the first threshold value according to the traffic difference between different partitions in the time period corresponding to the peak time point and the two adjacent time periods, and the maximum traffic difference between the partitions is used as the second threshold value according to the traffic difference between different partitions in the time period corresponding to the valley time point and the two adjacent time periods, and then when the partition is transferred at the peak time point, the traffic between the servers in the stream processing platform after the server is expanded is made less than the first threshold value according to the estimated traffic on the partition. Similarly, when the partition is transferred at the valley time point, the traffic between the servers in the stream processing platform after the server is expanded is made less than the second threshold value according to the estimated traffic on the partition, so that the traffic difference between the servers in the stream processing platform after the server is expanded will not be too large, thereby improving the processing capacity of the stream processing platform.
[0087] In another specific application scenario, transferring part of the partitions deployed in the servers of the stream processing platform to the expanded servers in a preset manner according to the estimated traffic includes: transferring part of the partitions to the expanded servers according to the estimated traffic so that the variance between the estimated traffic corresponding to all servers in the current stream processing platform is minimized; transferring the partitions deployed in the servers to be reduced to the servers to be retained in a preset manner according to the estimated traffic includes: transferring the partitions deployed in the servers to be reduced to the servers to be retained according to the estimated traffic so that the variance between the estimated traffic corresponding to all servers in the current stream processing platform is minimized; wherein the estimated traffic is the sum of the estimated traffic of all partitions on the current server.
[0088] Specifically, see Figure 6 , Figure 6 is a structural diagram of an implementation method of partition transfer when expanding a server. After obtaining the estimated traffic corresponding to each partition, when the expanded server performs partition transfer, some partitions are selected from the deployed partitions, and after being transferred to the expanded server, the variance of the estimated traffic between the servers is minimized. For example, Figure 6 As shown in , when transferring the partition on server 1 to server 2, the transfer is performed in a manner with the minimum variance. Partition 1 and partition 2 can be transferred to server 2 together, or partition 3 can be transferred to server 2 to minimize the variance. Similarly, when reducing the server and transferring the partitions, when transferring the partitions on the server to be reduced to the server to be saved, the variance of the expected traffic between the servers is minimized. Among them, the expected traffic on the server can be obtained by summing the estimated traffic corresponding to the partitions on the server. Furthermore, after the partitions are transferred, the traffic between the servers in the stream processing platform is relatively averaged, thereby improving the ability of the stream processing platform to cope with traffic fluctuations.
[0089] Optionally, when the time periods corresponding to multiple peak time points are adjacent and continuous time periods, the server is expanded only at the first preset time before the earliest time point; when the time periods corresponding to multiple valley time points are adjacent and continuous time periods, the server is reduced only at the second preset time after the earliest time point.
[0090] Optionally, if the server has been expanded or reduced in the current time period, when the current time point is neither in the period corresponding to the valley time point nor the peak time point in the current time period, the servers and partitions in the stream processing platform are restored to their initial state without expansion or reduction.
[0091] The elastic scaling method for the stream processing platform provided in this embodiment estimates the corresponding predicted traffic of the stream processing platform in each time period in the current time period through historical traffic, determines the peak time point and valley time point of the traffic, expands the server at the peak time point and reduces the server at the valley time point, and makes the traffic difference between the servers as even as possible, thereby improving the ability of the stream processing platform to handle sudden fluctuations in traffic.
[0092] See also Figure 7 , Figure 7 It is a structural diagram of an embodiment of an electronic device provided in the present application, wherein the electronic device 70 includes a memory 701 and a processor 702 coupled to each other, wherein the memory 701 stores program data (not shown), and the processor 702 calls the program data to implement the elastic scaling method for the stream processing platform in any of the above embodiments. For an explanation of the relevant contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.
[0093] See also Figure 8 , Figure 8 It is a structural diagram of an implementation method of a computer storage medium provided in the present application. The computer storage medium 80 stores program data 800. When the program data 800 is executed by the processor, the elastic scaling method for the stream processing platform in any of the above-mentioned embodiments is implemented. For an explanation of the relevant contents, please refer to the detailed description of the above-mentioned method embodiments, which will not be repeated here.
[0094] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present implementation scheme.
[0095] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
[0097] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A method for elastic expansion and contraction of a stream processing platform, characterized in that: The method comprises: Based on the historical traffic of the stream processing platform before the current time period, obtaining the corresponding predicted traffic of the stream processing platform in each time period in the current time period; Determining a valley time point and a peak time point from the current time period according to the predicted flow; When the current time point is a first preset time point before the peak time point, expanding a server for the stream processing platform, and transferring part of the partitions deployed in the server of the stream processing platform to the expanded server; When the current time point is a second preset time point after the valley time point, reducing the server for the stream processing platform, and transferring the partitions deployed in the server to be reduced to the server to be retained; Among them, the step of obtaining the predicted traffic corresponding to each time period in the current time period based on the historical traffic of the stream processing platform before the current time period includes: obtaining the predicted traffic corresponding to each time period in the current time period based on the historical traffic of the stream processing platform before the current time period; obtaining multiple second traffic samples in the same time period of each partition in the server of the stream processing platform in multiple time periods before the current time period; adding the second traffic samples in the same time period of all the partitions in the same server in the same time period to obtain the first traffic sample of the server in the same time period in the same time period; using the multiple first traffic samples corresponding to the multiple time periods before the current time period, obtaining the first traffic average value of the stream processing platform in each time period in the time period, and using the first traffic average value as the predicted traffic of the stream processing platform in each time period in the current time period.
2. The method according to claim 1, characterized in that Also includes: Acquire a plurality of second traffic samples of each partition in the server within the same time period in a plurality of time periods before the current time period; The plurality of second flow samples are used to obtain a second flow average value of each partition in each time period in the time cycle, and the second flow average value is used as the estimated flow of the partition in each time period in the current time cycle.
3. The method according to claim 2, characterized in that The step of transferring part of the partitions deployed in the server of the stream processing platform to the expanded server includes: According to the estimated traffic, part of the partitions deployed in the server of the stream processing platform are transferred to the expanded server in a preset manner, so that the maximum traffic difference between all the servers in the current stream processing platform is less than a first threshold; The step of transferring the partition deployed in the server to be reduced to the server to be retained includes: According to the estimated traffic, the partitions deployed in the servers to be reduced are transferred to the servers to be retained in a preset manner, so that the maximum traffic difference between all the servers in the current stream processing platform is less than a second threshold.
4. The method according to claim 2, characterized in that: The step of transferring part of the partitions deployed in the server of the stream processing platform to the expanded server in a preset manner according to the estimated traffic includes: Transferring the part of the partitions to the expanded server according to the estimated traffic, so as to minimize the variance between the estimated traffic corresponding to all servers in the current stream processing platform; The step of transferring the partitions deployed in the server to be reduced to the server to be retained in a preset manner according to the estimated traffic includes: According to the estimated traffic, the partitions deployed in the server to be reduced are transferred to the server to be retained, so as to minimize the variance of the estimated traffic corresponding to all servers in the current stream processing platform; The estimated traffic is the sum of the estimated traffic of all partitions on the current server.
5. The method according to claim 1, characterized in that The step of determining the valley time point and the peak time point from the current time period according to the predicted flow rate includes: The predicted traffic of the stream processing platform in each time period in the current time period is sorted in ascending order according to the numerical value of the traffic and numbered; Multiply the largest number by the first percentage and round it to obtain a first integer; The time point corresponding to the first integer number of predicted flows among all the predicted flows after sorting is taken as the valley time point, and the time point corresponding to the last integer number of predicted flows among all the predicted flows after sorting is taken as the peak time point.
6. The method according to claim 1, characterized in that The step of determining the valley time point and the peak time point from the current time period according to the predicted flow rate includes: Using the predicted traffic in each time period in the current time period, obtaining the average predicted traffic in a single time period in the current time period; Multiplying the predicted flow mean by the second percentage to obtain a valley threshold, and multiplying the predicted flow mean by the third percentage to obtain a peak threshold; The time points corresponding to all predicted flows less than the valley threshold are taken as the valley time points, and the time points corresponding to all predicted flows greater than the peak threshold are taken as the peak time points.
7. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 6.
8. A computer storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 6 is implemented.