Data warehouse computing resource estimation method and device and storage medium

Abstract

The embodiment of the invention provides a data warehouse computing resource estimation method and device and a storage medium, and the method comprises the steps: obtaining historical input data of a data warehouse, and estimating the number of newly added stages in future time according to the historical input data; calculating a complex coefficient according to the newly added process number and the complexity correction coefficient; calculating a fragment expansion coefficient according to the specific value of the fragment size, the sum of the new process numbers in the future time, the process number for modifying and calculating the fragment size of the engine in the future time and the fragment deviation correction coefficient; calculating to obtain a data expansion coefficient according to the data storage amount, the data block size and the data correction coefficient in the future time; a CPU usage amount regression model function and a memory usage amount regression model function of the computing resources are obtained, and the CPU resource usage amount in the future time and the memory resource usage amount in the future time are obtained through estimation according to the CPU usage amount regression model function, the complex coefficient, the fragmentation expansion coefficient and the data expansion coefficient.

Description

Technical Field

[0001] The present application belongs to the field of computer software and big data technology, and specifically relates to a method, device and storage medium for estimating computing resources of a data warehouse. Background Art

[0002] In the field of big data, data warehouses often need to estimate the capacity demand for cluster storage and computing resources in the future, apply for budgets or business procurement, etc., to achieve rapid and orderly expansion of cluster physical machines to meet normal data warehouse expansion needs and ensure the stability and effectiveness of data production.

[0003] Existing computing resource estimation methods are based on single-factor linear estimation or perceptual estimation. Since they cannot apply multiple factors of the data warehouse to estimate computing resources, they cannot accurately estimate the computing resources of the data warehouse, resulting in inaccurate computing resource estimation. Summary of the invention

[0004] The present application provides a data warehouse computing resource estimation method, device and storage medium, which can fully consider multiple factors such as the data volume, computing complexity and shard size of the data warehouse, and use multiple factors to estimate the computing resources of the data warehouse, thereby improving the accuracy of computing resource estimation.

[0005] In a first aspect, the present application provides a method for estimating computing resources of a data warehouse, the method comprising the following steps:

[0006] Obtain the historical input data of the data warehouse, and estimate the number of new stages p in the future based on the historical input data i ; Get the storage increment a of the previous time period in the future i-1 , get the user increment v in the future time i 、Total number of users u i , the user coverage in the future time C i , the monthly increase of business processes in the future time wi, the total number of business processes in the future time mi and the amount of data storage to be deleted S i ', based on the user increment v in the future i 、Total number of users u i , the user coverage in the future time C i , the monthly increase in business process of the previous month in the future i-1 , the total number of business processes mi in the future and the amount of data storage to be deleted S i 'Predict the amount of data stored in the future S i ;

[0007] According to the number of new processes p i And the complexity correction coefficient A is calculated to get the complexity coefficient Kci ; Modify the number of processes t that compute the engine shard size based on future time i , the ratio of the default input shard size of the computing engine to the modified shard size of the computing engine; based on the shard size ratio, the total number of new processes in the future, and the number of processes that modify the shard size of the computing engine in the future t i And the slice correction coefficient B is calculated to get the slice expansion coefficient K si ; Based on the amount of data stored in the future S i , data block size and data correction coefficient C to calculate the data expansion coefficient K pi ;

[0008] Get the CPU usage regression model function f of computing resources Vi (x), memory usage regression model function f mi (x), based on the CPU usage regression model function f Vi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the CPU resource usage in the future and regress the model function f based on the memory usage mi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the memory resource usage in the future.

[0009] In a second aspect, a data warehouse computing resource estimation device is provided, wherein the device comprises:

[0010] An acquisition unit, used to acquire historical input data of the data warehouse;

[0011] The computing unit is used to estimate the number of new stages p in the future based on the historical input data. i ; Get the storage increment a of the previous time period in the future i-1 ;

[0012] The acquisition unit is also used to acquire the user increment v in the future time. i 、Total number of users u i , the user coverage in the future time C i , the monthly increase of business processes in the future time wi, the total number of business processes in the future time mi and the amount of data storage to be deleted S i ';

[0013] The calculation unit is also used to calculate the user increment v in the future time. i 、Total number of users u i, the user coverage in the future time C i , the monthly increase in business process of the previous month in the future i-1 , the total number of business processes mi in the future and the amount of data storage to be deleted S i 'Predict the amount of data stored in the future S i ; According to the number of new processes p i And the complexity correction coefficient A is calculated to get the complexity coefficient K ci ; Modify the number of processes t that compute the engine shard size based on future time i , the ratio of the default input shard size of the computing engine to the modified shard size of the computing engine; based on the shard size ratio, the total number of new processes in the future, and the number of processes that modify the shard size of the computing engine in the future t i And the slice correction coefficient B is calculated to get the slice expansion coefficient K si ; Based on the amount of data stored in the future S i , data block size and data correction coefficient C to calculate the data expansion coefficient K pi ;

[0014] The acquisition unit is also used to obtain the CPU usage regression model function f of the computing resource Vi (x), memory usage regression model function f mi (x);

[0015] The computing unit is also used to regress the model function f according to the CPU usage. Vi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the CPU resource usage in the future and regress the model function f based on the memory usage mi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the memory resource usage in the future.

[0016] In a third aspect, the present application provides a computer storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute some or all of the steps described in the first aspect of the present application.

[0017] The embodiments of the present application have the following beneficial effects:

[0018] The technical solution of the present application integrates the complexity coefficient, the shard expansion coefficient and the data expansion coefficient, fully considers the estimation factors of future computing resources, improves the accuracy and reliability of computing resource estimation, can avoid the impact of too few estimated resources on the stability and effectiveness of future data output, and can also avoid excessive estimated resources causing obvious waste, which is of great significance to the rapid and orderly expansion of big data computing clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a structural diagram of the data center provided by this application;

[0021] Figure 2 It is a flowchart of a data warehouse computing resource estimation method provided by the present application;

[0022] Figure 3 A curve diagram of the change between a complex coefficient and the comprehensive stage growth rate provided for this application;

[0023] Figure 4 A schematic diagram of a variation curve of a slice correction coefficient B provided in this application;

[0024] Figure 5 A schematic diagram of a curve showing the change of the data correction coefficient C provided in this application;

[0025] Figure 6 A flowchart of a method for estimating computing resources of a data warehouse provided for implementation of the present application;

[0026] Figure 7 A schematic diagram of the structure of a data warehouse computing resource estimation device provided in this application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, systems, products or devices.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] The following is an introduction to the relevant terms involved in this application.

[0031] MapReduce is a programming model for parallel computing of large data sets (greater than 1TB). The concepts "Map" and "Reduce" are the main ideas, which are borrowed from functional programming languages, and features are borrowed from vector programming languages. It greatly facilitates programmers to run their programs on distributed systems without knowing distributed parallel programming.

[0032] A stage (Execution Stage) is a data processing logic unit in an execution plan, representing a group of independently executed operations. Multiple stages are scheduled to be executed in refrigerators or serially on computing resources through execution plans.

[0033] A job (Execution Job) is responsible for executing specific data processing tasks and is the basic computing task unit in the execution plan. A job usually corresponds to a collection of all operations in a stage.

[0034] Mapper (Mapping Function) maps input data into key-value pairs in the MapReduce model.

[0035] Reducer (Reducing Function) receives the output (key-value pairs) of mapper in the MapReduce model and processes all values ​​with the same key, such as summarizing, aggregating, and sorting.

[0036] The following describes the system architecture of a data center related to an embodiment of the present application.

[0037] The present application also provides a data center. The data warehouse of the present application can be set in the data center. The data center can include multiple computers 10, such as Figure 1 As shown, Figure 1 The structural diagram of the data center provided in the present application, each computer 10 may include at least one processor 11 and a memory 12, and may also include a communication interface 14 and a bus 13. Among them, the processor 11, the memory 12 and the communication interface 14 can communicate with each other through the bus 13. The communication interface 14 can transmit information. The processor 11 can call the logic instructions in the memory 12 to execute the method in the above embodiment.

[0038] In addition, the logic instructions in the above-mentioned memory 12 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. The above-mentioned logic instructions can also become an operating system, application program or computer program, etc.

[0039] The memory 12 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 11 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 12, that is, implementing the methods in the above embodiments.

[0040] The memory 12 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the computer 10, etc. In addition, the memory 12 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, may also be a transient storage medium.

[0041] It is understandable that the computer in this embodiment can be understood as a server, for example, the server includes a cloud server or an edge server, and no unique limitation is made here. Of course, in an optional technical scenario, the above-mentioned computer can also be a personal computer (PC), a tablet computer, a laptop computer, and other smart devices. This application does not limit the specific form of the above-mentioned computer, as long as the above-mentioned computer has an operating system and shortcuts.

[0042] The specific methods are introduced in detail below.

[0043] See also Figure 2 ,This application also provides a data warehouse computing resource estimation method, Figure 2 A flowchart of a data warehouse computing resource estimation method provided in this application, the above method can be used in Figure 1 The computer may be a personal computer, a server, a tablet computer, a laptop computer or other smart device. The method may be performed on the data center shown in FIG. Figure 2 As shown, the following steps are included:

[0044] Step S201: Obtain historical input data from the data warehouse, and estimate the number of new stages p in the future based on the historical input data. i ; Get the storage increment a of the previous time period in the future i-1 , get the user increment v in the future time i 、Total number of users u i , the user coverage in the future time C i , the monthly increase of business processes in the future time wi, the total number of business processes in the future time mi and the amount of data storage to be deleted S i ', based on the user increment v in the future i 、Total number of users u i , the user coverage in the future time C i , the monthly increase in business process of the previous month in the future i-1 , the total number of business processes mi in the future and the amount of data storage to be deleted S i 'Predict the amount of data stored in the future S i ;

[0045] For example, the above estimates the number of new stages p in the future based on the historical input data i Specifically, they may include:

[0046] According to the number of newly added stages in each time interval in the historical input data, a new change curve of pi is constructed, and the future time ti is obtained. The number of newly added stages pi corresponding to the future time ti is obtained from the new change curve.

[0047] In the above scheme, it is assumed that the number of newly added stages changes nonlinearly, so it is necessary to construct a new change curve. Of course, in actual applications, the number of newly added stages can also change linearly, that is, the number of newly added stages p estimated in the future based on the historical input data i It may also include:

[0048] According to the newly added stage numbers at two time intervals in the historical input data, a new change line of pi is constructed, and the future time ti is obtained. The newly added stage number pi corresponding to the future time ti is queried from the newly added change line.

[0049] The above estimates the number of new stages p in the future based on the historical input data i Specifically, they may include:

[0050]

[0051] Among them, n is the number of months to be accumulated, generally 12 months, and h is the specific month value.

[0052] The above user increment v based on future time i 、Total number of users u i , the user coverage in the future time C i , the monthly increase of the business process in the future time wi, the monthly increase of the business process in the previous month in the future time w i-1 and the amount of data storage to be deleted S i 'Predict the amount of data stored in the future S i Specifically, they may include:

[0053] Calculate the storage increment a in the time interval between the future time and the first time i-1 The user growth rate is calculated based on the user increment and total number of users in the future, and the user growth rate is calculated based on the total number of users u i , the user coverage in the future time C i , the monthly increase of the business process in the future time wi, the monthly increase of the business process in the previous month in the future time w i-1 Calculate the business process growth rate, calculate the product of the total increase and the user growth rate and the business process growth rate to get the product result, and the data storage volume Si in the future time is equal to the sum of the current total data S0 and the product result minus the data storage volume S' to be deleted.

[0054] In one optional solution, the calculation method of the above user growth rate may specifically include:

[0055]

[0056] Among them, v i is the user increment in the future time, u i-1 The total number of users in the previous month in the future.

[0057] In an optional solution, the calculation method of the business process growth rate may specifically include:

[0058] Business usage growth rate = 1 + w i *c i / m i-1 / u i

[0059] Among them, wi is the monthly increase of the business process in the future, w i-1 The business process month increment for the previous month in the future, u i is the total number of users in the future, C i User coverage in the future.

[0060] Among them, in an optional solution, the above-mentioned predicted data storage volume S for future time i The specific calculation methods may include:

[0061]

[0062] Step S202: according to the number of newly added processes p i And the complexity correction coefficient A is calculated to get the complexity coefficient K ci ; Modify the number of processes t that compute the engine shard size based on future time i , the ratio of the default input shard size of the computing engine to the modified shard size of the computing engine; based on the shard size ratio, the total number of new processes in the future, and the number of processes that modify the shard size of the computing engine in the future t i And the slice correction coefficient B is calculated to get the slice expansion coefficient K si ; Based on the amount of data stored in the future S i , data block size and data correction coefficient C to calculate the data expansion coefficient K pi ;

[0063] In an optional application scenario, the above-mentioned additional process number p i And the complexity correction coefficient A is calculated to get the complexity coefficient K ci Specifically, they may include:

[0064] Calculate the sum of the number of new processes in the time interval between the future time and the first time, and calculate the number of new processes p i The first product is obtained by multiplying the first ratio and the complexity correction coefficient A by the sum of the number of newly added processes. The complexity coefficient K ciis the first power of a constant.

[0065] For example, the above complexity coefficient K ci The calculation formula can be shown as follows:

[0066] The constant in this calculation formula is e.

[0067] Of course, in other application scenarios, the above complexity coefficient K ci The calculation formula can be as follows: It can also be:

[0068] The above constant may be 2.8. The present application does not limit the specific value of the above constant, but only requires that the range of the above constant be in the range of [2.7, 2.8].

[0069] The complexity correction coefficient A can be an empirical coefficient, for example, 0.7, 0.8, 1.0, etc. Of course, in other application scenarios, in order to improve the calculation accuracy of the complexity coefficient, the above A can also be obtained by querying the change curve of A, and the change curve of A can be obtained through historical input data. Specifically, assuming that the historical input data has the same data volume, the data center has the same parameters and hardware configuration, if the number of stages at the first time is f1 before, and the number of stages at the second time after the change is f 2i (For example, if 4 groups of adjustment values ​​are set, the accumulated resource usage at the first time is V1, and the accumulated resource usage at the second time after the change is V 2i ,So

[0070]

[0071] Where n is the number of months to be accumulated.

[0072] By extracting different time periods of historical input data, we can calculate the change curve between A and the comprehensive stage growth rate. The above comprehensive stage growth rate is the average of the stage growth rates of n months. Figure 3 , Figure 3 A curve diagram of the change between a complex coefficient and a comprehensive stage growth rate is provided for this application.

[0073] In an optional application scenario, the above-mentioned ratio of the shard size, the total number of processes added in the future time, and the number of processes t that modify the shard size of the computing engine in the future time are i And the slice correction coefficient B is calculated to get the slice expansion coefficient K si Specifically, they may include:

[0074] Calculate the sum of the number of new processes in the time interval between the future time and the first time, subtract 1 from the first coefficient of the slice correction coefficient B by the slice size ratio, and calculate the number of processes t that modify the slice size of the computing engine in the future time based on the slice size ratio i The second coefficient of , calculate the product of the shard correction coefficient B and the first coefficient to get the second product, calculate the number of processes t that will modify the shard size of the computing engine in the future i The product of the second coefficient is the third product, the product of the second product and the third product is the fourth product, the ratio of the fourth product to the sum of the number of newly added processes is calculated to obtain the second ratio, the shard expansion coefficient K si is the sum of the second ratio and 1.

[0075] The specific calculation method of the second coefficient may include:

[0076] The second coefficient = ln(1+1 / (1+e^(-l m / l tm );

[0077] Among them, l m The default input fragment size for the engine; l tm Modify the compute engine shard size for a future time.

[0078] For example, the above slice expansion coefficient K si The calculation formula can specifically include:

[0079]

[0080] For example, the above B may be an empirical coefficient, for example, the value of B may be 0.8, 0.9, 1.0, etc. Of course, the above B may also be obtained by querying the change curve of B. The change curve of B is as follows: Figure 4 As shown, Figure 4 A schematic diagram of the change curve of the shard correction coefficient B provided in this application. The above change curve can be obtained through historical input data. Specifically, the calculation adopts the control single variable method. For example, for the same data in the same table, the same calculation logic (the number of jobs, the number of stages, etc. are exactly the same), and the same cluster environment, by changing the input mapper or reducer shard size, observe the change rate of the calculation time. Set the default shard size to l m , after adjustment, it is l tmi (For example, if you set 4 groups of stage number shards with gradually decreasing adjustment values, the number of mappers and reducers in each stage is consistent), the previous cumulative resource usage is V1, and after the shard adjustment, it is V 2i , V 2i / V1 is used as Ksi and substituted into the slice expansion coefficient formula. The value of each group is averaged to calculate B.

[0081]

[0082] Where n is the number of months to be accumulated.

[0083] By extracting different time periods of historical input data, the change curve between B and the comprehensive shard expansion rate can be calculated.

[0084] The above comprehensive shard expansion rate = B*(l m / l tm -1)*ln(1+1 / (1+e^(-l m / l tm )).

[0085] For example, the above data storage volume S based on the future time i , data block size Z and data correction coefficient C to calculate the data expansion coefficient K pi Specifically, they may include:

[0086] Based on the amount of data stored in the future S i The number of data blocks in the future time is calculated based on the data storage volume S of the previous time in the future time and the data block size Z. i-1 The data expansion coefficient K is calculated based on the data block size Z and the data block size Z. The ratio of the data block number in the future time to the data block number in the previous time is calculated to obtain a third ratio. The data expansion coefficient K is calculated based on the third ratio and the data correction coefficient C. pi .

[0087] For example, the data expansion coefficient K is calculated based on the third ratio and the data correction coefficient C. pi Specifically, they may include:

[0088] The product of the third ratio and the data correction coefficient C is calculated to obtain the fifth product, and the fifth power of the constant is calculated. The data expansion coefficient Kpi is the product of the fifth product and the fifth power of the constant.

[0089] The values ​​of the above constants can be found in the values ​​of the constants for calculating the complexity coefficient Kci, which will not be repeated here.

[0090] See also Figure 5 , Figure 5 This is a schematic diagram of the change curve of the data correction coefficient C provided in this application. The above C can be based on the following Figure 5 The change curve shown is obtained. Of course, in some application scenarios, the value of the data correction coefficient C can also be an empirical coefficient, for example, 0.01, 0.05, etc. Figure 5The X-axis is the comprehensive data expansion rate = C*|S i / Z∣ / ∣S i-1 / Z∣.

[0091] The method for obtaining the variation line of the data correction coefficient C may specifically include:

[0092] The calculation is determined based on the specific configuration of different cloud service clusters combined with actual test data. The calculation adopts the control single variable method, for example, when the calculation logic, parameters, cluster environment, and data content are the same, the data storage capacity is S i-1 (larger than the size of the mapper input shard), the amount of data stored after the increase is S ij (If 4 sets of incremental adjustment values ​​are set, the data form remains the same), the previous cumulative resource usage is V1, and after the shard adjustment, it is V 2j , V 2j / V1 as K pi Substitute the data expansion coefficient formula and take the average of the values ​​in each group to calculate C;

[0093]

[0094] in,

[0095] ∣S i-1 / Z| means round down, |S ij / Z| means round down.

[0096] Step S203: Obtain the CPU usage regression model function f of the computing resources Vi (x), memory usage regression model function f mi (x), based on the CPU usage regression model function f Vi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the CPU resource usage in the future and regress the model function f based on the memory usage mi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the memory resource usage in the future.

[0097] The technical solution of the present application integrates the complexity coefficient, the shard expansion coefficient and the data expansion coefficient, fully considers the estimation factors of future computing resources, improves the accuracy and reliability of computing resource estimation, can avoid the impact of too few estimated resources on the stability and effectiveness of future data output, and can also avoid excessive estimated resources causing obvious waste, which is of great significance to the rapid and orderly expansion of big data computing clusters.

[0098] In an optional technical scenario, the above CPU usage regression model function f Vi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi The estimated CPU resource usage in the future may include:

[0099] Get the time interval that needs to be estimated and calculate the CPU regression model function f Vi (x) The definite integral value within the time interval, calculate the definite integral value and the complex coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi The product of gets the estimated CPU resource usage in this time interval.

[0100] For example, the calculation formulas for the CPU resource usage and memory resource usage may specifically include:

[0101]

[0102] Of course, in some actual application scenarios, the calculation formulas for the above CPU resource usage and memory resource usage may specifically include:

[0103]

[0104] Where T1 is the start time, T2 is the future time, P vi and P mi Respectively represent the cumulative resource usage of CPU and memory.

[0105] For example, the memory usage regression model function f mi (x) and CPU usage regression model function f Vi (x) may be a preset regression model function, and the above regression model function is not limited here.

[0106] Taking the time interval from 0:00 to 8:00 as an example, the above calculation of the cumulative CPU resource usage from 0:00 to 8:00 next month is as follows:

[0107]

[0108] Among them, K c1 is the complexity coefficient of the next month, K s1 is the expansion coefficient for the next month, K p1 is the data expansion coefficient for the next month; P v1is the estimated cumulative CPU resource usage from 0:00 to 8:00 next month, and fv1(x) is the regression model function of the CPU usage for the next month.

[0109] In an optional technical scenario, the above memory usage regression model function f mi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi The estimated memory resource usage in the future may include:

[0110] Get the time interval that needs to be estimated and calculate the memory usage regression model function f mi (x) The definite integral value within the time interval, calculate the definite integral value and the complex coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi The product of gets the estimated memory resource usage for this time interval.

[0111] Taking the time interval from 0:00 to 8:00 as an example, the above calculation of the cumulative memory resource usage from 0:00 to 8:00 next month is as follows:

[0112]

[0113] Among them, K c1 is the complexity coefficient of the next month, K s1 is the expansion coefficient for the next month, K p1 is the data expansion coefficient for the next month, P m1 is the estimated cumulative memory resource usage from 0:00 to 8:00 next month; fm1(x) is the regression model function of memory usage for the next month.

[0114] For example, the above method may further include:

[0115] The cumulative CPU resource usage rate at the future time is calculated based on the CPU resource usage at the future time, and the cumulative memory resource usage rate at the future time is calculated based on the memory resource usage at the future time.

[0116] The above calculation formula is as follows:

[0117] R vi =100%*P vi / [(T2-T1)*V]

[0118] R mi =100%*P mi / [(T2-T1)*M]

[0119] R viis the cumulative CPU resource usage, R mi is the cumulative memory resource usage rate, V is the maximum number of CPU cores in the application hardware cluster (such as a data center), and M is the maximum memory of the application hardware cluster (such as a data center); T1 is the start time, and T2 is the future time.

[0120] Embodiment 1

[0121] The first embodiment of the present application provides a method for estimating computing resources of a data warehouse, referring to Figure 6 , Figure 6 A flow chart of a data warehouse computing resource estimation method provided in Example 1 of the present application. In this example, the resource utilization rate for the next month is calculated, that is, n=1, the storage capacity S0 of the previous month is 10^9MB, the storage increment of the previous month is 10^8MB, and the user growth rate v i / u i-1 is 10%, and the business process growth rate w i / m i-1 is 10%, and the user coverage rate is c i / u i is 100%, the capacity management deleted data S' is 10^8MB, the number of stages f1 in the previous month is 10, and f 2i (i=1,2,3,4) are 12,14,16,18 respectively (the number of maps and reduce in each stage is consistent), and the test results are V 2i / V1(i=1,2,3,4) are 1.16,1.35,1.54,1.73 respectively. The total number of newly added stages as of last month is 100000. The number of newly added stages from the first month to the twelfth month in the past 12 months is 10000,10100,10300,10400,10450,10600,10650,10700,10900,11100,11300,11550 respectively. lm is 256MB, ltmi(i=1,2,3,4) are 128MB, 64MB, 32MB, 16MB respectively. The test results show that V 2i / V1(i=1,2,3,4) are 1.6,2.9,5.6,10.8 respectively; the data volume of the previous month S i-1 The adjusted data volume is S ij (j=1, 2, 3, 4) are 12GB, 14GB, 16GB, 18GB respectively (the data format remains the same), Z is 256MB, and the test results are V 2j / V1(j=1, 2, 3, 4) are 1.205, 1.406, 1.608, 1.811 respectively. The maximum number of CPU cores in this data center cluster is 50,000.

[0122] Step S601: Based on the storage capacity S0 of the previous month, the storage increment of the previous month, the user growth rate, the business process growth rate, and the user coverage rate c i / u i Calculate the storage capacity S for the next month i ;

[0123] Specifically, according to the above data, Si=1.021*10^9MB is calculated;

[0124] Step S602: Based on the stage number f1 of the previous month, the stage number of the next month after the change is f 2i , the tested V 2i The complexity correction coefficient A calculated by / V1 is used to calculate pi based on the total number of new stages added last month; the complexity coefficient K is calculated based on the complexity correction coefficient A and pi ci ;

[0125] Specifically, according to the data of this embodiment, p is calculated. i =15487; A=0.72;K ci =1.118.

[0126] Step S603: Calculate the slice expansion coefficient, and calculate the slice expansion coefficient K based on the slice expansion coefficient, the default input slice size, and the modified slice size. si .

[0127] According to the data of this embodiment, B=0.942, K si =1.595.

[0128] Step S604: Calculate the data expansion coefficient. The data expansion coefficient K is calculated based on the data expansion coefficient, the data volume of the previous month, the data block size, and the adjusted data volume. pi ;

[0129] According to the data of this embodiment, C=0.00326; K ci =1.024.

[0130] Step S605: Based on the CPU usage regression model function and the complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Calculate the CPU resource usage for the next month.

[0131] Specifically, they may include:

[0132] According to the CPU vcore usage of the cluster at each time point (1, 2, 3, 4, 5, 6, 7, 8) at the end of last month, which is 10000, 15000, 30000, 45000, 40000, 35000, 30000, 20000, the polynomial fitting regression function is:

[0133] f Vi (x)=-63.13131313*x*3+-1320.34632035*x*2+18147.54689755*x+-9642.85714286

[0135] Calculate the cumulative CPU resource usage from 0:00 to 8:00 next month:

[0136]

[0137] Step S606: Calculate the cumulative resource utilization rate for the next month based on the CPU resource usage for the next month.

[0138] R v1 =100%*390022 / [(8-0)*50000]=97%.

[0139] The technical solution of the present application integrates the complexity coefficient, the shard expansion coefficient and the data expansion coefficient, fully considers the estimation factors of future computing resources, improves the accuracy and reliability of computing resource estimation, can avoid the impact of too few estimated resources on the stability and effectiveness of future data output, and can also avoid excessive estimated resources causing obvious waste, which is of great significance to the rapid and orderly expansion of big data computing clusters.

[0140] See also Figure 7 , Figure 7 A schematic diagram of the structure of a data warehouse computing resource estimation device provided in the present application, the device comprising:

[0141] An acquisition unit 701 is used to acquire historical input data of a data warehouse;

[0142] The calculation unit 702 is used to estimate the number of new stages p in the future based on the historical input data. i ; Get the storage increment a of the previous time period in the future i -1;

[0143] The acquisition unit 701 is also used to obtain the user increment v in the future time i 、Total number of users u i , the user coverage in the future time C i , the monthly increase of business processes in the future time wi, the total number of business processes in the future time mi and the amount of data storage to be deleted S i ';

[0144] The calculation unit 702 is further configured to calculate the user increment v at a future time. i 、Total number of users u i , the user coverage in the future time C i , the monthly increase in business process of the previous month in the future i -1. The total number of business processes mi in the future and the amount of data storage to be deleted S i 'Predict the amount of data stored in the future S i ; According to the number of new processes p i And the complexity correction coefficient A is calculated to get the complexity coefficient K ci ; Modify the number of processes t that compute the engine shard size based on future time i , the ratio of the default input shard size of the computing engine to the modified shard size of the computing engine; based on the shard size ratio, the total number of new processes in the future, and the number of processes that modify the shard size of the computing engine in the future t i And the slice correction coefficient B is calculated to get the slice expansion coefficient K si ; Based on the amount of data stored in the future S i , data block size and data correction coefficient C to calculate the data expansion coefficient K pi ;

[0145] The acquisition unit 701 is also used to obtain the CPU usage regression model function f of the computing resource Vi (x), memory usage regression model function f mi (x);

[0146] The calculation unit 702 is also used to regress the model function f according to the CPU usage. Vi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the CPU resource usage in the future and regress the model function f based on the memory usage mi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the memory resource usage in the future.

[0147] For example, the calculation unit 702 is specifically used to calculate the storage increment a within the time interval between the future time and the first time. i-1 The user growth rate is calculated based on the user increment and total number of users in the future, and the user growth rate is calculated based on the total number of users u i , the user coverage in the future time C i, the monthly increase of the business process in the future time wi, the monthly increase of the business process in the previous month in the future time w i-1 Calculate the business process growth rate, calculate the product of the total increase and the user growth rate and the business process growth rate to get the product result, and the data storage volume Si in the future time is equal to the sum of the current total data S0 and the product result minus the data storage volume S' to be deleted.

[0148] For example, the calculation unit 702 is specifically used to calculate the sum of the number of new processes in the time interval between the future time and the first time, and calculate the number of new processes p i The first product is obtained by multiplying the first ratio and the complexity correction coefficient A by the sum of the number of newly added processes. The complexity coefficient K ci is the first power of a constant.

[0149] For example, the calculation unit 702 is specifically used to calculate the sum of the number of newly added processes in the time interval between the future time and the first time, and the first coefficient of the slice correction coefficient B minus 1 of the slice size ratio is calculated according to the slice size ratio to obtain the number of processes t of modifying the slice size of the computing engine in the future time. i The second coefficient of , calculate the product of the shard correction coefficient B and the first coefficient to get the second product, calculate the number of processes t that will modify the shard size of the computing engine in the future i The product of the second coefficient is the third product, the product of the second product and the third product is the fourth product, the ratio of the fourth product to the sum of the number of newly added processes is calculated to obtain the second ratio, the shard expansion coefficient K si is the sum of the second ratio and 1.

[0150] For example, the calculation unit 702 is specifically configured to store data S at a future time. i The number of data blocks in the future time is calculated based on the data storage volume S of the previous time in the future time and the data block size Z. i-1 The data expansion coefficient K is calculated based on the data block size Z and the data block size Z. The ratio of the data block number in the future time to the data block number in the previous time is calculated to obtain a third ratio. The data expansion coefficient K is calculated based on the third ratio and the data correction coefficient C. pi .

[0151] For example, the calculation unit 702 is specifically used to calculate the product of the third ratio and the data correction coefficient C to obtain a fifth product, and calculate the fifth product power of the constant, the data expansion coefficient K pi It is the product of the fifth product and a constant raised to the fifth power.

[0152] For example, the calculation unit 702 is further used to obtain the time interval to be estimated and calculate the CPU regression model function f Vi(x) The definite integral value within the time interval, calculate the definite integral value and the complex coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi The product of gets the estimated CPU resource usage in this time interval;

[0153] The calculation unit 702 is also used to obtain the time interval to be estimated and calculate the memory usage regression model function f mi (x) The definite integral value within the time interval, calculate the definite integral value and the complex coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi The product of gets the estimated memory resource usage for this time interval.

[0154] The calculation unit 702 is further used to calculate the cumulative CPU resource usage rate in the future time according to the CPU resource usage in the future time, and to calculate the cumulative memory resource usage rate in the future time according to the memory resource usage in the future time.

[0155] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0156] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method recorded in the above method embodiments, and the above computer includes an electronic device.

[0157] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0158] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0159] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0162] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a volatile memory or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM) and direct RAM bus RAM (DR RAM). Various media that can store program code are available.

[0163] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.

Claims

1. A method for estimating computing resources of a data warehouse, characterized in that: The method comprises the following steps: Obtain the historical input data of the data warehouse, and estimate the number of new stages p in the future based on the historical input data i ; Get the storage increment a of the previous time period in the future i -1, get the user increment v in the future time i 、Total number of users u i , the user coverage in the future time C i , the monthly increase of business processes in the future time wi, the total number of business processes in the future time mi and the amount of data storage to be deleted S i ', based on the user increment v in the future i 、Total number of users u i , the user coverage in the future time C i , the monthly increase in business process of the previous month in the future i -1. The total number of business processes mi in the future and the amount of data storage to be deleted S i 'Predict the amount of data stored in the future S i ; According to the number of new processes p i And the complexity correction coefficient A is calculated to get the complexity coefficient K ci ; Modify the number of processes t that compute the engine shard size based on future time i , the ratio of the default input shard size of the computing engine to the modified shard size of the computing engine; Based on the ratio of the shard size, the total number of new processes in the future, and the number of processes that modify the shard size of the calculation engine in the future, t i And the slice correction coefficient B is calculated to get the slice expansion coefficient K si ; Based on the amount of data stored in the future S i , data block size and data correction coefficient C to calculate the data expansion coefficient K pi ; Get the CPU usage regression model function f of computing resources Vi (x), memory usage regression model function f mi (x), based on the CPU usage regression model function f Vi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the CPU resource usage in the future and regress the model function f based on the memory usage mi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the memory resource usage in the future.

2. The data warehouse computing resource estimation method according to claim 1, characterized in that: The user increment v based on the future time i 、Total number of users u i , the user coverage in the future time C i , the monthly increase in business process of the previous month in the future i -1. The total number of business processes mi in the future and the amount of data storage to be deleted S i 'Predict the amount of data stored in the future S i Specifically include: Calculate the storage increment a in the time interval between the future time and the first time i-1 The user growth rate is calculated based on the user increment and total number of users in the future, and the user growth rate is calculated based on the total number of users u i , the user coverage in the future time C i , the monthly increase of the business process in the future time wi, the monthly increase of the business process in the previous month in the future time w i-1 Calculate the business process growth rate, calculate the product of the total increase, the user growth rate and the business process growth rate to get the product result, and the data storage volume S in the future time i It is equal to the sum of the current total amount of data S0 and the product result minus the amount of data storage to be deleted S'.

3. The data warehouse computing resource estimation method according to claim 2, characterized in that: According to the number of new processes p i And the complexity correction coefficient A is calculated to get the complexity coefficient K ci Specifically include: Calculate the sum of the number of new processes in the time interval between the future time and the first time, and calculate the number of new processes p i The first product is obtained by multiplying the first ratio and the complexity correction coefficient A by the sum of the number of newly added processes. The complexity coefficient K ci is the first power of a constant.

4. The data warehouse computing resource estimation method according to claim 2, characterized in that: The ratio of the slice size, the total number of processes added in the future, and the number of processes for modifying the slice size of the computing engine in the future t i And the slice correction coefficient B is calculated to get the slice expansion coefficient K si Specifically include: Calculate the sum of the number of new processes in the time interval between the future time and the first time, subtract 1 from the first coefficient of the slice correction coefficient B by the slice size ratio, and calculate the number of processes t that modify the slice size of the computing engine in the future time based on the slice size ratio i The second coefficient of , calculate the product of the shard correction coefficient B and the first coefficient to get the second product, calculate the number of processes t that will modify the shard size of the computing engine in the future i The product of the second coefficient is the third product, the product of the second product and the third product is the fourth product, the ratio of the fourth product to the sum of the number of newly added processes is calculated to obtain the second ratio, the shard expansion coefficient K si is the sum of the second ratio and 1.

5. The data warehouse computing resource estimation method according to claim 2, characterized in that: The amount of data stored in the future time S i , data block size Z and data correction coefficient C to calculate the data expansion coefficient K pi Specifically include: Based on the amount of data stored in the future S i The number of data blocks in the future time is calculated based on the data storage volume S of the previous time in the future time and the data block size Z. i-1 The data expansion coefficient K is calculated based on the data block size Z and the data block size Z. The ratio of the data block number in the future time to the data block number in the previous time is calculated to obtain a third ratio. The data expansion coefficient K is calculated based on the third ratio and the data correction coefficient C. pi .

6. The data warehouse computing resource estimation method according to claim 5, characterized in that: The data expansion coefficient K is calculated based on the third ratio and the data correction coefficient C. pi Specifically include: Calculate the product of the third ratio and the data correction coefficient C to get the fifth product, calculate the fifth product power of the constant, the data expansion coefficient K pi It is the product of the fifth product and a constant raised to the fifth power.

7. The method for estimating computing resources of a data warehouse according to any one of claims 1 to 6, characterized in that: The CPU usage regression model function f Vi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi The estimated CPU resource usage in the future includes: Get the time interval that needs to be estimated and calculate the CPU regression model function f Vi (x) The definite integral value within the time interval, calculate the definite integral value and the complex coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi The product of gets the estimated CPU resource usage in this time interval; The memory usage regression model function f mi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi The estimated memory resource usage in the future includes: Get the time interval that needs to be estimated and calculate the memory usage regression model function f mi (x) The definite integral value within the time interval, calculate the definite integral value and the complex coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi The product of gets the estimated memory resource usage for this time interval.

8. The data warehouse computing resource estimation method according to claim 7, characterized in that: The method further comprises: The cumulative CPU resource usage rate at the future time is calculated based on the CPU resource usage at the future time, and the cumulative memory resource usage rate at the future time is calculated based on the memory resource usage at the future time.

9. A data warehouse computing resource estimation device, characterized in that: The device comprises: An acquisition unit, used to acquire historical input data of the data warehouse; The computing unit is used to estimate the number of new stages p in the future based on the historical input data. i ; Get the storage increment a of the previous time period in the future i-1； The acquisition unit is also used to acquire the user increment v in the future time. i 、Total number of users u i , the user coverage in the future time C i , the monthly increase of business processes in the future time wi, the total number of business processes in the future time mi and the amount of data storage to be deleted S i '; The calculation unit is also used to calculate the user increment v in the future time. i 、Total number of users u i , the user coverage in the future time C i , the monthly increase in business process of the previous month in the future i-1 , the total number of business processes mi in the future and the amount of data storage to be deleted S i 'Predict the amount of data stored in the future S i ; According to the number of new processes p i And the complexity correction coefficient A is calculated to get the complexity coefficient K ci ; Modify the number of processes t that compute the engine shard size based on future time i , the ratio of the default input shard size of the computing engine to the modified shard size of the computing engine; based on the shard size ratio, the total number of new processes in the future, and the number of processes that modify the shard size of the computing engine in the future t i And the slice correction coefficient B is calculated to get the slice expansion coefficient K si ; Based on the amount of data stored in the future S i , data block size and data correction coefficient C to calculate the data expansion coefficient K pi ; The acquisition unit is also used to obtain the CPU usage regression model function f of the computing resource Vi (x), memory usage regression model function f mi (x); The computing unit is also used to regress the model function f according to the CPU usage. Vi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the CPU resource usage in the future and regress the model function f based on the memory usage mi (x), complexity coefficient K ci , Slice expansion coefficient K si , data expansion coefficient K pi Estimate the memory resource usage in the future.

10. A computer storage medium storing a computer program for electronic data exchange, wherein: The computer program enables a computer to execute part or all of the steps of the method according to any one of claims 1 to 8.

Images