Method and device for determining the number of computing servers in a big data platform

Abstract

An embodiment of the present application provides a method and device for determining the number of computing servers in a big data platform, the method comprising: predicting the number of first computing tasks of the big data platform in a first time period of a second cycle based on application load data of the big data platform in each time period of at least one first cycle, wherein the first cycle is a time period that has occurred and the second cycle is an ongoing time period; determining whether to expand or reduce the number of running servers in the big data platform based on a relationship between the first number of computing tasks and the processing capabilities of n servers in operation in the big data platform, wherein n is an integer greater than or equal to 1.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for determining the number of computing servers in a big data platform. Background Art

[0002] Yet Another Resource Negotiator (YARN) is a resource management and job scheduling technology in the open source Hadoop distributed processing framework. As one of the core components of Hadoop, YARN is responsible for allocating system resources to various applications running in the Hadoop cluster and scheduling tasks to be executed on different cluster nodes.

[0003] The basic idea of ​​YARN is to decompose the functions of resource management and job scheduling / monitoring into separate daemons, which have a global resource manager (ResourceManager, RM) and each application manager (ApplicationMaster, AM). The application can be a single job or a directed acyclic graph (Directed Acyclic Graphene, DAG) of jobs.

[0004] YARN, as an important global functional component, also has defects. The main problem with the currently running YARN open source components is that the configuration rules are fixed, and when a large number of computing tasks are running in parallel, the performance of other components in the cluster will be reduced. For example, in actual use, when a large number of YARN computing tasks start running, the number of requests to the HBASE system will decrease, resulting in reduced cluster performance, and in severe cases, business backlogs. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method and device for determining the number of computing servers in a big data platform, which can solve the problem of reduced performance of the entire cluster caused by large-scale data parallel computing.

[0006] In order to solve the above technical problems, this application adopts the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a method for determining the number of computing servers in a big data platform, comprising: predicting the number of first computing tasks of the big data platform in a first time period of a second cycle based on application load data of the big data platform in each time period of at least one first cycle, wherein the first cycle is a time period that has occurred and the second cycle is an ongoing time period; determining whether to expand or reduce the number of running servers in the big data platform based on the first number of servers and the processing capacity of n running servers in the big data platform, where n is an integer greater than or equal to 1.

[0008] In a second aspect, an embodiment of the present invention provides a device for determining the number of computing servers in a big data platform, comprising: a prediction module, used to predict the number of first computing tasks of the big data platform in a first time period of a second cycle based on application load data of the big data platform in each time period of at least one first cycle, wherein the first cycle is a time period that has occurred and the second cycle is an ongoing time period; a determination module, used to determine whether to expand or reduce the number of servers in operation in the big data platform based on the first number of servers and the processing capacity of n servers in operation in the big data platform, where n is an integer greater than or equal to 1.

[0009] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and computer executable instructions stored in the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of the method for connecting to a base station as described in the first aspect above.

[0010] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the method for connecting to a base station as described in the first aspect above are implemented.

[0011] An embodiment of the present application provides a method and device for determining the number of computing servers in a big data platform. The method predicts the number of first computing tasks of the big data platform in the first time period of a second cycle based on application load data of the big data platform in each time period of at least one first cycle, and then determines whether to expand or reduce the number of running servers in the big data platform based on the relationship between the first number of computing tasks and the processing capabilities of n servers in operation in the big data platform, thereby effectively solving the problem of reduced performance of the entire cluster due to large-scale data parallel computing in the big data platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0013] Figure 1 It is a flow chart of a method for determining the number of computing servers in a big data platform disclosed in an embodiment of the present application;

[0014] Figure 2 It is a structural schematic diagram of a device for determining the number of computing servers in a big data platform disclosed in an embodiment of the present application;

[0015] Figure 3 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0017] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0018] In conjunction with the accompanying drawings, the method for determining the number of computing servers in the big data platform provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios.

[0019] Figure 1 1 is a flow chart of a method for determining the number of computing servers in a big data platform disclosed in an embodiment of the present application. The method 100 can be executed by an electronic device. In other words, the method can be executed by software or hardware installed in the electronic device. Figure 1 As shown, the method may include the following steps.

[0020] S110: Predicting the number of first computing tasks of the big data platform in a first time period of a second cycle based on application load data of the big data platform in each time period of at least one first cycle.

[0021] In the embodiment of the present application, the first cycle is a time cycle that has occurred, and the second cycle is an ongoing time cycle. The first cycle can be one or more. Take one first cycle as an example. For example, the first cycle is 8:00am-20:00pm of the previous day, the second cycle is 8:00am-20:00pm of the current day, and the first time period of the second cycle is 10:00am-10:10am. Because the first cycle is a time cycle that has occurred, it records the application load data of the entire cycle. To predict the number of first computing tasks in the first time period of 10:00am-10:10am of the second cycle, it is only necessary to find the application load data corresponding to the time period of 10:00am-10:10am of the first cycle, and the number of first computing tasks can be predicted.

[0022] Taking the case where there are multiple first cycles, when there are multiple first cycles, the application load data recorded in multiple first cycles can be averaged to make the predicted number of first computing tasks in the first time period closer to the actual number of tasks calculated in the first time period in the big data platform.

[0023] S120: Determine whether to expand or reduce the number of servers in operation in the big data platform based on the relationship between the number of the first computing tasks and the processing capabilities of the n servers in operation in the big data platform.

[0024] In an embodiment of the present application, the processing capacity of a server refers to the number of tasks that the server can calculate. The relationship between the first number of computing tasks and the processing capacity of the n servers in operation in the big data platform is used to determine whether to expand or reduce the number of servers in operation in the big data platform. When the first number of computing tasks is greater than the processing capacity of the n servers, the number of servers in operation in the big data platform is expanded. When the first number of computing tasks is equal to the processing capacity of the n servers, the number of servers in operation in the big data platform is maintained. When the first number of computing tasks is less than the processing capacity of the n servers, the number of servers in operation in the big data platform is reduced, wherein n is an integer greater than or equal to 1.

[0025] An embodiment of the present application provides a method for determining the number of computing servers in a big data platform. By predicting the number of first computing tasks of the big data platform in the first time period of the second cycle, and comparing in real time the relationship between the number of first computing tasks and the processing capabilities of n servers in operation in the big data platform, it is determined whether to expand or reduce the number of servers in operation in the big data platform. This overcomes the service delay problem caused by the big data platform's real-time analysis of application load data, and effectively solves the problem of reduced performance of the entire cluster caused by a large number of computing tasks being performed in parallel in the big data platform.

[0026] In a possible implementation, after predicting the number of first computing tasks in the first time period of the second cycle of the big data platform, the method may further include: crawling the number of second computing tasks in the first time period by crawler technology within the first time period, and recording the second computing task number as application load data in the first time period of the second cycle. In this possible implementation, the predicted number of first computing tasks may be inconsistent with the number of computing tasks actually processed by the server in the big data platform. In order to make the number of computing tasks predicted in the same time period of the next cycle more accurate, the number of second computing tasks in the corresponding first time period is crawled in real time by crawler technology, that is, the number of computing tasks actually processed by the server in the big data platform. In addition, collecting and analyzing data by crawler technology will not affect the cluster performance in the big data platform.

[0027] In another possible implementation, before predicting the number of first computing tasks of the big data platform in the first time period of a second cycle based on the application load data of the big data platform in each time period of at least one first cycle, the method may also include: determining the second time period based on the application load data of the big data platform in each time period of at least one first cycle, wherein, within the at least one first cycle, the difference between the application load data of the second time period and the application load data of the third time period is greater than a first predetermined value, and the third time period is the previous time period adjacent to the second time period; determining the time period corresponding to the second time period within the second cycle as the first time period.

[0028] In the above possible implementation methods, the second time period of the first cycle may be a time period with a large number of computing tasks, or it may be a time period with a small number of computing tasks. Taking the second time period as an example where the number of computing tasks is large, the difference between the application load data of the second time period and the application load data of the third time period is calculated. When the difference is greater than the first predetermined value, it means that the number of computing tasks in the second time period is large, so the number of first computing tasks in the first time period corresponding to the second time period is also large. In the case of a large number of computing tasks, the servers of the big data platform are not enough to calculate the number of tasks, so the number of running servers is expanded through corresponding rules.

[0029] In this possible implementation, determining the second time period based on the application load data of the big data platform in each time period of at least one first cycle can include: mapping the application load data in each time period of at least one first cycle to target data according to a predetermined mapping rule; determining the second time period based on the target data in each time period of at least one first cycle, wherein, in the at least one first cycle, the difference between the target data of the second time period and the target data of the third time period is greater than a second predetermined value.

[0030] In a specific application, the crawler program collects data every 300 seconds, collects enough application load data for at least one first cycle, and maps the application load data in each time period of at least one first cycle to target data according to a predetermined mapping rule. The predetermined mapping rule is:

[0031] e(m)=d(m)-X(m);

[0032] W(m+1)=W(m)+ηX(m)e(m);

[0033] Initialize, set m = 0, assign w(0) a small random non-zero value;

[0034] Among them, X(m) is the training sample, which here represents at least one first cycle of application load data; W(m) is the target data; e(m) is the deviation; d(m) is the expected output; y(m) is the actual output; η is the learning rate; and m is the number of iterations.

[0035] Taking the first cycle as Q as an example, Q is an integer greater than 1, each time period of each first cycle has corresponding application load data, when the predetermined mapping rule calculates the application load data of the first first cycle, m is increased by 1, and the predetermined mapping rule continues to calculate the application load data of the second first cycle until m equals Q, and the algorithm ends. The target data W(m) obtained by the predetermined mapping rule is sequentially compared to see whether the difference between the target data of a time period in the first cycle and the target data of the previous time period is greater than the second predetermined value. When it is greater than the second predetermined value, the time period is the second time period.

[0036] In this implementation, the data is filtered and adaptively adjusted in a cyclic calculation manner to approach the optimal result and predict the number of the first computing tasks more accurately.

[0037] In the above possible implementations, the second time period may be a time period with the maximum and / or minimum application load data within the first cycle. When the second time period is a time period with the maximum application load data within the first cycle, so that the predicted number of first tasks is the maximum, and the servers in the big data platform are insufficient to handle, the number of servers in operation in the big data platform is expanded; when the second time period is a time period with the minimum application load data within the first cycle, so that the predicted number of first tasks is the minimum, and there are too many servers in operation in the big data platform, the number of servers in operation in the big data platform is reduced.

[0038] In a possible implementation, S120 may include:

[0039] Step 1, determining whether the number of the first computing tasks is greater than the first processing capacity, if yes, executing step 2, otherwise, executing step 3;

[0040] Step 2, updating β servers in the dormant server group in the big data platform to a running state, and returning to step 1;

[0041] Step 3, determining whether the number of the first computing tasks is less than the second processing capacity, if yes, executing step 4, otherwise, executing step 5;

[0042] Step 4, reducing α servers from the n servers in operation in the big data platform, and returning to step 3;

[0043] Step 5, maintaining the number of servers in operation in the big data platform.

[0044] In this possible implementation, the first processing capacity is the sum of the processing capacity of n-1 servers in operation and the processing capacity increment of the dormant server group; the second processing capacity is the processing capacity of n-1 servers in operation; β is the quotient of the first number of computing tasks and the first processing capacity rounded up; α is the quotient of the second processing capacity and the first number of computing tasks rounded down.

[0045] Specifically, the above steps can be expressed by the following formula:

[0046] Result=β+n,iftaskNum≥(n-1)p+Δp;

[0047] Result=n-α, taskNum≤(n-1)p;

[0048] Result=n, if (n-1)p≤taskNum≤(n-1)p+Δp;

[0049]

[0050] Among them, taskNum is the number of the first computing tasks; (n-1)p is the processing capacity of n-1 servers in operation, that is, the second processing capacity; Δp is the processing capacity increment of the dormant server group; (n-1)p+Δp is the processing capacity of n-1 servers in operation and the processing capacity increment of the dormant server group, that is, the first processing capacity; Result is the number of servers in operation after expansion or reduction of servers; β can be regarded as an intelligent increasing factor, and α can be regarded as an intelligent decreasing factor.

[0051] In this possible implementation, when it is determined that the number of first computing tasks is greater than the first processing capacity, the number of servers in operation in the big data platform is expanded, and the YARN configuration file of the dormant server is modified by a program, the relevant computing module is started, and the dormant server is added to the running server cluster to cope with the computing peak. When it is determined that the number of first computing tasks is less than the second processing capacity, the number of servers in operation in the big data platform is reduced, and the YARN configuration file of the running server is modified by a program, the relevant computing module is stopped, and the running server is returned to the dormant server group.

[0052] In the above possible implementation methods, by introducing the intelligent increasing factor β and the intelligent decreasing factor α, the servers running in the big data platform can be increased or decreased, avoiding system instability caused by sudden increase or decrease in the number of servers, thereby effectively coping with the diverse fluctuations of application load data in the big data platform.

[0053] In addition, in the above possible implementation, step 4: reducing α servers from the n servers in operation in the big data platform may include: making the α servers enter a cooling state; and after a preset cooling time, adding the α servers to the dormant server group. By setting a preset cooling time, the reduced α servers enter a cooling state until the preset cooling time has passed, and the α servers join the dormant server group, thereby ensuring the stability of the cluster in the big data platform.

[0054] The above-mentioned application embodiment crawls the application load data in the big data platform through crawler technology, monitors the application load data in the cluster in a contactless manner, filters and adaptively adjusts the application load data in a cyclic calculation manner to approach the optimal result, and obtains the optimal first number of computing tasks. Then, the corresponding expansion and reduction activities are generated through the optimal first number of computing tasks, so as to realize dynamic and intelligent expansion and reduction of servers in the cluster, so as to solve the problem of reduced performance of the entire cluster caused by large-scale data parallel calculation, while saving costs and improving efficiency.

[0055] Based on the above-mentioned method for determining the number of computing servers in a big data platform, the present application embodiment discloses a device 200 for determining the number of computing servers in a big data platform, such as Figure 2 As shown, the device 200 mainly includes:

[0056] A prediction module 210, configured to predict the number of first computing tasks of the big data platform in a first time period of a second cycle according to application load data of the big data platform in each time period of at least one first cycle, wherein the first cycle is a time period that has occurred and the second cycle is an ongoing time period;

[0057] The determination module 220 is used to determine whether to expand or reduce the number of servers in operation in the big data platform based on the first number of servers and the processing capacity of the n servers in operation in the big data platform, where n is an integer greater than or equal to 1.

[0058] In a possible implementation, the prediction module 210 is also used to crawl the number of second computing tasks within the first time period through crawler technology within the first time period, and record the number of second computing tasks as application load data in the first time period of the second cycle.

[0059] In another possible implementation, the prediction module 210 can also be used to determine a second time period based on application load data of the big data platform in each time period of at least one first cycle, wherein, within the at least one first cycle, the difference between the application load data of the second time period and the application load data of the third time period is greater than a first predetermined value, and the third time period is the previous time period adjacent to the second time period; and the time period corresponding to the second time period within the second cycle is determined to be the first time period.

[0060] In this possible implementation, the prediction module 210 determines the second time period based on the application load data of the big data platform in each time period of at least one first cycle, which may include: mapping the application load data in each time period of at least one first cycle to target data according to a predetermined mapping rule; determining the second time period based on the target data in each time period of at least one first cycle, wherein, in the at least one first cycle, the difference between the target data of the second time period and the target data of the third time period is greater than a second predetermined value.

[0061] In the above possible implementation manners, the second time period may be a time period in which the application load data in the first cycle is maximum and / or minimum.

[0062] In one implementation, the determination module 220 determines whether to expand or reduce the number of second servers in operation in the big data platform according to the number of the first servers and the processing capabilities of the n servers in operation in the big data platform, including:

[0063] Step 1, determining whether the number of the first computing tasks is greater than the first processing capacity, if yes, executing step 2, otherwise, executing step 3;

[0064] Step 2, updating β servers in the dormant server group in the big data platform to a running state, and returning to step 1;

[0065] Step 3, determining whether the number of the first computing tasks is less than the second processing capacity, if yes, executing step 4, otherwise, executing step 5;

[0066] Step 4, reducing α servers from the n servers in operation in the big data platform, and returning to step 3;

[0067] Step 5, maintaining the number of servers in operation in the big data platform;

[0068] Among them, the first processing capacity is the sum of the processing capacity of n-1 servers in running state and the processing capacity increment of the dormant server group; the second processing capacity is the processing capacity of n-1 servers in running state; β is the quotient of the first number of computing tasks and the first processing capacity rounded up; α is the quotient of the second processing capacity and the first number of computing tasks rounded down.

[0069] In the above implementation, the determination module reduces α servers from the n servers in operation in the big data platform, including: putting the α servers into a cooling state; after a preset cooling time, adding the α servers to the dormant server group.

[0070] The determination device 200 provided in the embodiment of the present application can execute the various methods described in the foregoing method embodiments, and realize the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be repeated here.

[0071] Optional, such as Figure 3 As shown, an embodiment of the present application also provides an electronic device 300, including a processor 301, a memory 302, and a program or instruction stored in the memory 302 and executable on the processor 301. When the program or instruction is executed by the processor 301, each process of the embodiment of the method for determining the number of computing servers in the above-mentioned big data platform is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0072] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned test state detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0073] The processor 301 is the processor in the electronic device 300 described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0074] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run network-side device programs or instructions to implement the various processes of the embodiment of the method for determining the number of servers in the above-mentioned big data platform, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0075] An embodiment of the present application also provides a computer program product, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, each process of the embodiment of the method for determining the number of computing servers in the above-mentioned big data platform is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0076] The above embodiments of the present application focus on describing the differences between the various embodiments. As long as the different optimization features between the various embodiments are not contradictory, they can be combined to form a better embodiment. Considering the simplicity of the text, they will not be repeated here.

[0077] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for determining the number of computing servers in a big data platform, characterized in that: include: Predicting the number of first computing tasks of the big data platform in a first time period of a second cycle according to application load data of the big data platform in each time period of at least one first cycle, wherein the first cycle is a time period that has occurred and the second cycle is an ongoing time period; Determine whether to expand or reduce the number of servers in operation in the big data platform according to the relationship between the number of the first computing tasks and the processing capabilities of the n servers in operation in the big data platform, where n is an integer greater than or equal to 1; Before predicting the number of first computing tasks of the big data platform in the first time period of the second cycle according to the application load data of the big data platform in each time period of at least one first cycle, the method further includes: Determine a second time period according to application load data of the big data platform in each time period of at least one first cycle, wherein within the at least one first cycle, a difference between the application load data of the second time period and the application load data of a third time period is greater than a first predetermined value, and the third time period is a previous time period adjacent to the second time period; Determine a time period within the second cycle corresponding to the second time period as the first time period; The determining the second time period according to the application load data of the big data platform in each time period of at least one first cycle includes: According to a predetermined mapping rule, mapping the application load data in each time period of at least one first cycle to target data; Determining the second time period according to target data in each time period of at least one first cycle, wherein, in the at least one first cycle, a difference between the target data in the second time period and the target data in the third time period is greater than a second predetermined value; The predetermined mapping rule is: e(m)=d(m)-X(m); W(m+1)=W(m)+ηX(m)e(m); initialization, let m=0, assign a small random non-zero value to w(0); wherein, X(m) is a training sample, representing at least one first cycle of application load data; W(m) is the target data; e(m) is the deviation; d(m) is the expected output; y(m) is the actual output; η is the learning rate; and m is the number of iterations.

2. The method according to claim 1, characterized in that After predicting the number of first computing tasks of the big data platform in the first time period of the second cycle, the method further includes: In the first time period, the number of second computing tasks in the first time period is crawled by using crawler technology, and the number of second computing tasks is recorded as application load data in the first time period of the second cycle.

3. The method according to claim 1, characterized in that The second time period is a time period in which the application load data in the first cycle is maximum and / or minimum.

4. The method according to claim 1, characterized in that Determining whether to expand or reduce the number of servers in operation in the big data platform according to a relationship between the number of the first computing tasks and the processing capabilities of the n servers in operation in the big data platform includes: Step 1, determining whether the number of the first computing tasks is greater than the first processing capacity, if yes, executing step 2, otherwise, executing step 3; Step 2, updating β servers in the dormant server group in the big data platform to a running state, and returning to step 1; Step 3, determining whether the number of the first computing tasks is less than the second processing capacity, if yes, executing step 4, otherwise, executing step 5; Step 4, reducing α servers from the n servers in operation in the big data platform, and returning to step 3; Step 5, maintaining the number of servers in operation in the big data platform; Among them, the first processing capacity is the sum of the processing capacity of n-1 servers in running state and the processing capacity increment of the dormant server group; the second processing capacity is the processing capacity of n-1 servers in running state; β is the quotient of the first number of computing tasks and the first processing capacity rounded up; α is the quotient of the second processing capacity and the first number of computing tasks rounded down.

5. The method according to claim 4, characterized in that Reducing α servers from the n servers in operation in the big data platform includes: Putting the α servers into a cooling state; After a preset cooling time, the α servers are added to the dormant server group.

6. A device for determining the number of computing servers in a big data platform, characterized in that: include: A prediction module, configured to predict the number of first computing tasks of the big data platform in a first time period of a second cycle according to application load data of the big data platform in each time period of at least one first cycle, wherein the first cycle is a time period that has occurred and the second cycle is an ongoing time period; a determination module, configured to determine whether to expand or reduce the number of servers in operation in the big data platform according to a relationship between the number of the first computing tasks and the processing capabilities of the n servers in operation in the big data platform, where n is an integer greater than or equal to 1; The prediction module is also used to: Determine a second time period according to application load data of the big data platform in each time period of at least one first cycle, wherein within the at least one first cycle, a difference between the application load data of the second time period and the application load data of a third time period is greater than a first predetermined value, and the third time period is a previous time period adjacent to the second time period; Determine a time period within the second cycle corresponding to the second time period as the first time period; In the prediction module, determining the second time period according to the application load data of the big data platform in each time period of at least one first cycle includes: According to a predetermined mapping rule, mapping the application load data in each time period of at least one first cycle to target data; Determining the second time period according to target data in each time period of at least one first cycle, wherein, in the at least one first cycle, a difference between the target data in the second time period and the target data in the third time period is greater than a second predetermined value; The predetermined mapping rule is: e(m)=d(m)-X(m); W(m+1)=W(m)+ηX(m)e(m); initialization, let m=0, assign a small random non-zero value to w(0); wherein, X(m) is a training sample, representing at least one first cycle of application load data; W(m) is the target data; e(m) is the deviation; d(m) is the expected output; y(m) is the actual output; η is the learning rate; and m is the number of iterations.

7. The device according to claim 6, characterized in that The prediction module is also used to: In the first time period, the number of second computing tasks in the first time period is crawled by using crawler technology, and the number of second computing tasks is recorded as application load data in the first time period of the second cycle.

Images