Dynamic allocation method and device for log storage partitions
By dynamically allocating log storage partitions, the application's write traffic prediction and grouping problems are solved, and the log storage partition fixation problems are improved, and write and search efficiency is improved.
Patent Information
- Application Number
- CN202111319868.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-09
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2041-11-09
AI Technical Summary
In the prior art, the log storage partition is fixed, resulting in problems such as low writing performance, high concurrency pressure, and low search efficiency during high data volume and peak periods.
By obtaining the historical write traffic data of each application, predicting the write traffic in the next period, dynamically allocating the log storage partition, grouping and allocating the partition size according to the write traffic size, improving write performance and search efficiency.
It realizes flexible allocation based on write traffic, improves log write storage performance, alleviates the pressure of concurrency during peak periods, and improves data search efficiency.
Smart Images
Figure CN114003178B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of data storage and finance, and in particular to a method and device for dynamically allocating log storage partitions. Background Art
[0002] The process of writing the log data generated by each application every day to the cluster's disk is called log storage. Common clusters contain three types of nodes: management nodes, client nodes, and data nodes. The application's log data will be distributed to the corresponding data nodes through the client nodes according to the instructions of the management nodes. In the prior art, the size of the shards allocated to each application to store log data is usually set according to the number of data nodes. Generally speaking, the log storage partitions corresponding to the log data of each application are fixed, which leads to the following problems:
[0003] 1) The data storage partitions and shards of each application are fixed and single. When the amount of log data is large, its write storage performance is low;
[0004] 2) During the peak period of log storage, the concurrent pressure of writing log data is high;
[0005] 3) The search efficiency for log data stored by each application is low. Summary of the invention
[0006] The present application provides a method for dynamically allocating log storage partitions, which relates to the fields of data storage and finance. The method includes:
[0007] Respectively obtain a first write flow of log data of each application in any time period of a plurality of historical periods and a total write flow of log data of each application in each of the historical periods;
[0008] Predicting a second write flow of log data of each application in the time period of the next cycle according to the first write flow and the total write flow;
[0009] The applications are divided into M groups according to the second write traffic, and the size of the log storage partition allocated to each group of applications during the time period of the next cycle is determined according to the second write traffic of the applications in each group.
[0010] In one embodiment, predicting the second write flow of log data of each application in the time period of the next cycle according to the first write flow and the total write flow includes:
[0011] Determine the total write traffic cycle representative value of log data of each application in multiple historical cycles respectively;
[0012] Determine respectively a first period ratio of the first write flow of log data of each application in the period of each historical period to the total write flow of the corresponding historical period;
[0013] Determine a second time period ratio of the write traffic of each application in the time period of the next cycle according to the first time period ratio;
[0014] The second write flow of log data of each application in the time period of the next cycle is determined according to the second time period ratio of each application and the total write flow period representative value.
[0015] In one embodiment, dividing the applications into M groups according to the second write traffic includes:
[0016] sorting each application according to the second write traffic;
[0017] The sorted applications are grouped according to a preset grouping ratio to obtain M groups.
[0018] In one embodiment, determining the size of the log storage partition allocated to the application in each group during the time period of the next cycle according to the second write traffic of the application in each group includes:
[0019] Determine a write flow representative value of the corresponding group according to the second write flow of the application in each group;
[0020] According to the ratio of the write traffic representative value of each group to the total write traffic representative values of the M groups, the size of the log storage partition allocated to the application in the corresponding group in the first period of the next cycle is determined.
[0021] In one embodiment, determining the write traffic representative value of the corresponding group according to the second write traffic of the application in each group includes:
[0022] sorting the second write traffic of the applications included in each group respectively;
[0023] The median of the sorted second write traffic is used as the representative value of the write traffic of the corresponding group.
[0024] In one embodiment, determining the write traffic representative value of the corresponding group according to the second write traffic of the application in each group includes:
[0025] respectively calculating an average value of the second write traffic of the applications included in each group;
[0026] The average value of the second write flow rate of each group is taken as the representative value of the write flow rate of the corresponding group.
[0027] In one embodiment, when writing log data, each application in the same group automatically occupies the available storage space of the log storage partition of the corresponding group.
[0028] The present application also provides a device for dynamically allocating log storage partitions, the device comprising:
[0029] A historical write flow acquisition module, used to respectively acquire the first write flow of log data of each application in any period of multiple historical cycles and the total write flow of log data of each application in each of the historical cycles;
[0030] A write traffic prediction module, configured to predict a second write traffic of log data of each application in the time period of the next cycle according to the first write traffic and the total write traffic;
[0031] The log storage partition allocation module is used to divide each application into M groups according to the second write flow, and determine the size of the log storage partition allocated to each group of applications during the time period of the next cycle according to the second write flow of the applications in each group.
[0032] In one embodiment, the write traffic prediction module includes:
[0033] A periodic write traffic prediction unit, used to respectively determine a periodic representative value of the total write traffic of log data of each application in a plurality of historical periods;
[0034] A period write traffic ratio determination unit, used to respectively determine a first period ratio of the first write traffic of the log data of each application in the period of each historical cycle to the total write traffic of the corresponding historical cycle; and determine a second period ratio of the write traffic of each application in the period of the next cycle according to the first period ratio;
[0035] The time period write traffic prediction unit is used to determine the second write traffic of log data of each application in the time period of the next cycle according to the second time period ratio of each application and the total write traffic cycle representative value.
[0036] In one embodiment, the log storage partition allocation module includes an application grouping unit, which is used to:
[0037] sorting each application according to the second write traffic;
[0038] The sorted applications are grouped according to a preset grouping ratio to obtain M groups.
[0039] In one embodiment, the log storage partition allocation module further includes:
[0040] a write traffic representative value determining unit, configured to determine a write traffic representative value of a corresponding group according to the second write traffic of the application in each group;
[0041] The log storage partition allocation unit is used to determine the size of the log storage partition allocated to the application in the corresponding group in the first time period of the next cycle according to the proportion of the write traffic representative value of each group to the total write traffic representative values of the M groups.
[0042] The dynamic allocation method and device of log storage partitions of the present application can flexibly allocate the log storage partitions of the application according to the size of the write traffic, thereby improving the performance of log writing storage, alleviating the concurrency pressure of some data nodes on the storage end during peak periods, and improving the efficiency of searching the stored data. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0044] Figure 1 A schematic diagram of a method for dynamically allocating log storage partitions of the present application.
[0045] Figure 2 Another diagram of a dynamic allocation method for log storage partitions.
[0046] Figure 3 Another diagram of a dynamic allocation method for log storage partitions.
[0047] Figure 4 Schematic diagram of a dynamic allocation mechanism for log storage partitions.
[0048] Figure 5 Another schematic diagram of a dynamic allocation device for log storage partitions.
[0049] Figure 6 Another schematic diagram of a dynamic allocation device for log storage partitions.
[0050] Figure 7 A schematic diagram of an electronic device. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] This application provides a method for dynamically allocating log storage partitions, such as Figure 1 As shown, the method comprises the following steps:
[0053] Step S101 , respectively obtaining the first write flow of log data of each application in any time period of a plurality of historical periods and the total write flow of log data of each application in each of the historical periods.
[0054] Specifically, assuming that a week is a cycle, Monday in each cycle is a time period, Tuesday is a time period, ..., Sunday is a time period, then a cycle is divided into 7 time periods. In the dynamic allocation process of log storage partitions, different allocation strategies are determined for each time period in a cycle. Those skilled in the art will appreciate that the definition of the cycle and the division of time periods in each cycle in the present embodiment are only a feasible solution. In practical applications, a cycle can be defined as a month, and the 1st to 10th of each month is the first time period, and the 11th to 20th is the second time period, etc. In other words, the cycle in this application can be a time period that is longer or shorter than the cycle in the above example, and more or fewer time periods can also be divided in each cycle, and this application is not limited to this.
[0055] In this embodiment, taking the Monday period of three historical cycles (i.e., three weeks) as an example, the first write flow rate of log data of each application in any period of multiple historical cycles in step S101 is obtained respectively, specifically: the first write flow rate T of log data of application A on Monday of the first week is obtained. 11A , the first write traffic T of log data of application A on Monday of the second week 21A , the first write traffic T of log data of application A on Monday of the third week 31A ; The first write traffic T of log data of application B on Monday of the first week 11B , the first write traffic T of log data of application B on Monday of the second week 21B , the first write traffic T of log data of application B on Monday of the third week 31B ; and the first write traffic T of log data of application C on Monday of the first week 11C , the first write traffic T of log data of application C on Monday of the second week 21C , the first write traffic T of log data of application C on Monday of the third week 31C In this embodiment, a total of three applications are taken as an example for description, and actually more or fewer applications may be included, and this application is not limited to this.
[0056] The total write traffic of log data of each application in each historical period in step S101 is obtained, specifically: the total write traffic T of application A in the 7 time periods in the first week is obtained. 1A, the total write traffic T of application A in the seven periods from Monday to Friday in the second week 2A , the total write traffic T of application A in the 7 periods in the third week 3A ; Get the total write traffic T of application B in the 7 periods in the first week 1B , the total write traffic T of application B in the 7 periods of the second week 2B , the total write traffic T of application B in the 7 periods in the third week 3B ; and obtain the total write traffic T of application C in the 7 periods in the first week 1C , the total write traffic T of application C in the 7 periods of the second week 2C , the total write traffic T of application C in the 7 periods of the third week 3C .
[0057] Table 1 shows the relationship between the write traffic of application A in different time periods of each cycle and the total write traffic of each cycle:
[0058] Table 1: Historical cycle write traffic table of application A
[0059]
[0060] The relationship between the write traffic of Application B and Application C in different time periods of each cycle and the total write traffic of each cycle is similar to that of Application A, see Table 2 and Table 3.
[0061] Table 2: Historical period write traffic table of application B
[0062]
[0063]
[0064] Table 3: Historical cycle write traffic table of application C
[0065]
[0066] Step S102: predicting a second write flow of log data of each application in the time period of the next cycle according to the first write flow and the total write flow.
[0067] Following the above example, the second write traffic of each application in the next cycle (i.e., the next week) is predicted as follows: Based on the write traffic T of application A on Monday of the first week, 11A and the total write traffic T in the first week 1A Calculate the write traffic ratio R of application A on Monday of the first week 11 , based on the write traffic T of application A on Monday of the second week 21A and the total write traffic T in the second week 2ACalculate the write traffic ratio R of application A on Monday of the second week 21 , and the write traffic T of application A on Monday of the third week 31A and the total write traffic T in the third week 3A Calculate the write traffic ratio R of application A on Monday of the third week 31 , according to the ratio R 11A , R 21A , R 31A And the total write traffic T 1A , T 2A , T 3A The second write traffic of application A on Monday of the next cycle (ie, the next week) is predicted. There are many ways to predict, which will be described one by one in the subsequent embodiments.
[0068] Similarly, the second write traffic and prediction method of log data of application B and application C in the said period of the next cycle are the same as those of application A. It is only necessary to replace the relevant data of application A with the data of application B and application C. This application will not be repeated here.
[0069] Step S103: divide the applications into M groups according to the second write traffic, and determine the size of the log storage partition allocated to each group of applications during the time period of the next cycle according to the second write traffic of the applications in each group.
[0070] The dividing each application into M groups according to the second write traffic specifically includes: sorting each application according to the second write traffic; and grouping each sorted application according to a preset grouping ratio to obtain M groups.
[0071] For example, the preset grouping rule in the system is to sort the total log write traffic predicted for each application every day (i.e., a period of a cycle) from low to high, and divide the applications into M=5 groups at a ratio of 30%, 25%, 20%, 15%, and 10% of the total number of applications, respectively. Assuming that there are 300 applications in this example, the total log write traffic (i.e., the second write traffic) of each application on Monday of the next cycle (i.e., the next week) is sorted from low to high and divided into 5 groups: the first group contains 90 applications, the second group contains 75 applications, the third group contains 60 applications, the fourth group contains 45 applications, and the fifth group contains 30 applications.
[0072] After the grouping is completed, the size of the log storage partition allocated to the group of applications on Monday of the next cycle (i.e., the next week) is determined based on the second flow write value of the applications included in each group. The log storage partition here is allocated in groups, that is, all applications in each group share the log storage partition allocated to the group. When each application in the same group writes log data, it automatically occupies the available storage space of the log storage partition of the corresponding group.
[0073] The dynamic allocation method and device of log storage partitions of the present application can flexibly allocate the log storage partitions of the application according to the size of the write traffic, thereby improving the performance of log writing storage, alleviating the concurrency pressure of some data nodes on the storage end during peak periods, and at the same time ensuring that applications in each group flexibly occupy log storage partitions within the group.
[0074] In one embodiment, if Figure 2 As shown, step S102, predicting the second write flow of log data of each application in the time period of the next cycle according to the first write flow and the total write flow, specifically includes the following steps S1021 to S1024:
[0075] Step S1021 , respectively determining a total write traffic cycle representative value of log data of each application in a plurality of historical cycles.
[0076] In a specific example, the cycle average value of the total write flow of log data of each application in multiple historical periods is determined as the cycle representative value of the total write flow. Continuing with the example of calculating the second write flow of application A, application B and application C in the previous embodiment, refer to Table 1. The cycle average value T of the total write flow of log data of application A in the past three historical periods (the first week, the second week and the third week) is A =(T 1A +T 2A +T 3A ) / 3; Similarly, the total write traffic cycle average value T of application B’s log data in the past three historical cycles (the first week, the second week, and the third week) B =(T 1B +T 2B +T 3B ) / 3; the total write traffic cycle average value T of log data of application C in the past three historical cycles (first week, second week and third week) C =(T 1C +T 2C +T 3C ) / 3.
[0077] In another specific example, the median of the total write flow of log data of each application in multiple historical periods is determined as the total write flow period representative value. Continuing with the example of calculating the second write flow of application A, application B, and application C in the previous embodiment, refer to Table 1. The total write flow of log data of application A in the past three historical periods (the first week, the second week, and the third week) T 1A , T 2A and T 3A Sort and determine the median of the three, and write the total into flow T 1A , T 2A and T3A The median of is taken as the representative value of the total write flow cycle corresponding to application A; similarly, the total write flow of log data of application B in the past three historical cycles (the first week, the second week, and the third week) is taken as the representative value of the total write flow T 1B 、T 2B and T 3B Sort and determine the median of the three, and write the total into flow T 1B 、T 2B and T 3B The median of is taken as the representative value of the total write flow cycle corresponding to application B; the total write flow T of log data of application C in the past three historical cycles (first week, second week and third week) is taken as the representative value of the total write flow cycle corresponding to application B; 1C 、T 2C and T 3C Sort and determine the median of the three, and write the total into flow T 1C 、T 2C and T 3C The median is taken as the representative value of the total write stream cycle corresponding to application C.
[0078] Step S1022 , respectively determining a first period ratio of the first write flow of log data of each application in the period of each historical period to the total write flow of the corresponding historical period.
[0079] Specifically, the first write traffic of log data of application A on Monday of the first week accounts for the first period ratio of the total write traffic of the first week. 11A =T 11A / T 1A The first write traffic of log data of application A on Monday of the second week accounts for the second period of the total write traffic of the second week. 21A =T 21A / T 2A The first write traffic of log data of application A on Monday of the third week accounts for the first period of the total write traffic of the third week. 31A =T 31A / T 3A ;
[0080] Similarly, the first write traffic of log data of application B on Monday of the first week accounts for the first period of the total write traffic of the first week. 11B =T 11B / T 1B The first write traffic of log data of application B on Monday of the second week accounts for the second period of the total write traffic of the second week. 21B =T 21B / T 2B The first write traffic of log data of application B on Monday of the third week accounts for the first period of the total write traffic of the third week. 31B =T 31B / T3B .
[0081] The first write traffic of log data of application C on Monday of the first week accounts for the first period of the total write traffic of the first week. 11C =T 11C / T 1C The first write traffic of log data of application C on Monday of the second week accounts for the second period of the total write traffic of the second week. 21C =T 21C / T 2C The first write traffic of log data of application C on Monday of the third week accounts for the first period of the total write traffic of the third week. 31C =T 31C / T 3C .
[0082] Step S1023: determining a second time period ratio of the write traffic of each application in the time period of the next cycle according to the first time period ratio.
[0083] Specifically, the median of the first period ratio of each application in the period of each historical cycle is used as the second period ratio of the write traffic in the period of the next cycle. 41A For R 11A , R 21A and R 31A The median of the second period ratio R of application B 41B For R 11B , R 21B and R 31B The median of the second period of application C 41C For R 11C , R 21C and R 31C of the median.
[0084] For another example, the average of the first period ratios of each application in each historical period is used as the second period ratio of the write traffic in the period of the next period. 41A =(R 11A +R 21A +R 31A ) / 3; the second period ratio R of application B 41B =(R 11B +R 21B +R 31B ) / 3; the second period ratio R of application C 41C =(R 11C +R 21C +R 31C ) / 3.
[0085] Step S1024, determining the second write flow of log data of each application in the time period of the next cycle according to the second time period ratio of each application and the total write flow period representative value.
[0086] Specifically, the second write traffic T of the log data of application A on Monday of the next cycle (ie, the next week) is 41A =R 41A ×T A The second write traffic T of log data of application B on Monday of the next cycle (i.e., the next week) 41B =R 41B ×T B The second write traffic T of log data of application C on Monday of the next cycle (i.e., the next week) 41C =R 41C ×T C .
[0087] In one embodiment, if Figure 3 As shown, step S103, dividing each application into M groups according to the second write traffic, and determining the size of the log storage partition allocated to the application in each group during the time period of the next cycle according to the second write traffic of the application in each group, includes:
[0088] Step S1031, sorting the applications according to the second write traffic; grouping the sorted applications according to a preset grouping ratio to obtain M groups;
[0089] Step S1032 , determining a write traffic representative value of the corresponding group according to the second write traffic of the application in each group.
[0090] Specifically, assuming that there are 300 applications in total, after each application is divided into M=5 groups, the first group contains 90 applications, the second group contains 75 applications, the third group contains 60 applications, the fourth group contains 45 applications, and the fifth group contains 30 applications. The second write flow corresponding to each application is calculated according to the method in the aforementioned embodiment of the present application (see step S1021 to step S1024). Then, the write flow representative value W11 of the first group of applications, the write flow representative value W21 of the second group of applications, the write flow representative value W31 of the third group of applications, the write flow representative value W41 of the fourth group of applications, and the write flow representative value W51 of the fifth group of applications are determined respectively. Among them, the write flow representative value of each group of applications is described in detail in subsequent embodiments.
[0091] Step S1033, according to the ratio of the write traffic representative value of each group to the total write traffic representative values of the M groups, determine the size of the log storage partition allocated to the application in the corresponding group in the first period of the next cycle. The ratio of the write traffic representative value of each group to the total write traffic representative values of the M groups is the ratio of the log storage partition allocated to the group of applications to the total log storage partition space.
[0092] Specifically, assuming that the total space of the log storage partition is Y, the log storage partition Y1 allocated to the first group of applications is = W11 / (W11+W21+W31+W41+W51)×Y; the log storage partition Y2 allocated to the second group of applications is = W21 / (W11+W21+W31+W41+W51)×Y; the log storage partition Y3 allocated to the third group of applications is = W31 / (W11+W21+W31+W41+W51)×Y; the log storage partition Y4 allocated to the fourth group of applications is = W41 / (W11+W21+W31+W41+W51)×Y; the log storage partition Y5 allocated to the fifth group of applications is = W51 / (W11+W21+W31+W41+W51)×Y.
[0093] In one embodiment, step S1032, determining a write traffic representative value of a corresponding group according to the second write traffic of the application in each group, includes:
[0094] The second write flows of the applications included in each group are sorted respectively; and the median of the sorted second write flows is used as a representative value of the write flow of the corresponding group.
[0095] Specifically, for the 90 applications in the first group, the second write traffic values corresponding to the 90 applications in this group are sorted and the median is taken, and the median is used as the representative value of the write traffic for the first group; the method for determining the representative value of the write traffic for other groups is the same as the first group, and will not be repeated here.
[0096] In one embodiment, step S1032, determining a write traffic representative value of a corresponding group according to the second write traffic of the application in each group, includes:
[0097] The average value of the second write traffic of the applications included in each group is calculated respectively; and the average value of the second write traffic of each group is used as the representative value of the write traffic of the corresponding group.
[0098] Specifically, for the 90 applications in the first group, the second write traffic values corresponding to the 90 applications in this group are summed and the average value is calculated, and the average value is used as the representative value of the write traffic for the first group; the method for determining the representative value of the write traffic for other groups is the same as the first group, and will not be repeated here.
[0099] In the above-mentioned embodiment of the present application, the allocation strategy of the log storage partitions for each time period of the next cycle will be calculated before the next cycle arrives, and when the next cycle arrives, the allocation of the log storage partitions for each time period will be completed in sequence according to the determined allocation strategy. Generally speaking, within the preset time before the arrival of the nth time period of a cycle, the allocation strategy of the log storage partition for the nth time period determined in advance is imported into the system, so that each application performs log storage according to the corresponding allocation strategy in the nth time period. However, since the allocation strategy of the log storage partition is obtained according to the log write traffic prediction of each application in the historical cycle, there is a certain error between it and the log write traffic actually generated by each application. Therefore, in another embodiment, the dynamic allocation method of the log storage partition of the present application also includes:
[0100] Get the actual log write traffic of each application in the nth period of the current cycle;
[0101] According to the actual log write traffic, each application is grouped to obtain the actual grouping of each application in the nth time period of the current cycle; wherein, the grouping rule can refer to the grouping rule in the aforementioned embodiment of the present application, that is, the actual log write traffic is sorted from low to high, and the applications are divided into M=5 groups in the proportion of 30%, 25%, 20%, 15%, and 10% of the total number of applications, wherein the actual log write traffic size relationship of the applications in each group is: the first group (actual group) < the second group (actual group) < the third group (actual group) < the fourth group (actual group) < the fifth group (actual group);
[0102] Calculate the gear difference between the actual grouping and the predicted grouping of each application respectively, wherein the predicted grouping is the grouping obtained according to the predicted second write flow of each application in the corresponding period when determining the allocation strategy of the log storage partition before the current cycle arrives;
[0103] When the actual grouping of at least one application is two or more levels higher than the predicted grouping corresponding to the application, the allocation strategy of the log storage partition of each application in the nth time period of the current cycle is re-determined. That is, when the actual grouping of application X is group 5, and the predicted grouping of application X is group 2, the actual grouping of application X is 3 levels higher than the predicted grouping corresponding to the application. At this time, application X is reallocated to the corresponding group.
[0104] The above-mentioned embodiments of the present application are all described by taking a certain period of a cycle as an example, and the allocation of log storage partitions for other periods of the same cycle can be implemented using the same method. The dynamic allocation method of log storage partitions of the present application can flexibly allocate the log storage partitions of the application according to the size of the write traffic, improve the performance of log writing storage, alleviate the concurrent pressure of some data nodes on the storage end during peak periods, and improve the efficiency of data stored according to search.
[0105] Based on the same inventive concept, the embodiments of the present application also provide a kind of dynamic allocation device of log storage partition, which can be used to implement the method described in the above embodiment, as described in the following embodiments. Since the principle of solving the problem by the dynamic allocation device of log storage partition is similar to the dynamic allocation method of log storage partition, the implementation of the dynamic allocation device of log storage partition can refer to the implementation of the dynamic allocation method of log storage partition, and the repetitions are not repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware of predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.
[0106] The present application also provides a device for dynamically allocating log storage partitions, such as Figure 4 As shown, the device comprises:
[0107] The historical write flow acquisition module 401 is used to respectively acquire the first write flow of log data of each application in any period of multiple historical periods and the total write flow of log data of each application in each of the historical periods;
[0108] A write traffic prediction module 402, configured to predict a second write traffic of log data of each application in the time period of the next cycle according to the first write traffic and the total write traffic;
[0109] The log storage partition allocation module 403 is used to divide the applications into M groups according to the second write traffic, and determine the size of the log storage partition allocated to each group of applications during the time period of the next cycle according to the second write traffic of the applications in each group.
[0110] In one embodiment, if Figure 5 As shown, the write traffic prediction module 402 includes:
[0111] The periodic write traffic prediction unit 4021 is used to respectively determine the periodic representative value of the total write traffic of the log data of each application in multiple historical periods;
[0112] The period write traffic ratio determination unit 4022 is used to respectively determine the first period ratio of the first write traffic of the log data of each application in the period of each historical cycle to the total write traffic of the corresponding historical cycle; and determine the second period ratio of the write traffic of each application in the period of the next cycle according to the first period ratio;
[0113] The time period write traffic prediction unit 4023 is used to determine the second write traffic of log data of each application in the time period of the next cycle according to the second time period ratio of each application and the total write traffic cycle representative value.
[0114] In one embodiment, if Figure 6 As shown, the log storage partition allocation module 403 includes an application grouping unit 4031, which is used to:
[0115] sorting each application according to the second write traffic;
[0116] The sorted applications are grouped according to a preset grouping ratio to obtain M groups.
[0117] In one embodiment, if Figure 6 As shown, the log storage partition allocation module 403 also includes:
[0118] A write traffic representative value determining unit 4032, configured to determine a write traffic representative value of a corresponding group according to the second write traffic of the application in each group;
[0119] The log storage partition allocation unit 4033 is used to determine the size of the log storage partition allocated to the application in the corresponding group in the first period of the next cycle according to the ratio of the write traffic representative value of each group to the total write traffic representative values of the M groups.
[0120] The dynamic allocation method and device of log storage partitions of the present application can flexibly allocate the log storage partitions of the application according to the size of the write traffic, thereby improving the performance of log writing storage, alleviating the concurrency pressure of some data nodes on the storage end during peak periods, and improving the efficiency of searching the stored data.
[0121] The present invention also provides an electronic device, see Figure 7 , the electronic device 100 specifically includes:
[0122] A central processing unit (processor) 110 , a memory (memory) 120 , a communication module (Communications) 130 , an input unit 140 , an output unit 150 and a power supply 160 .
[0123] The memory 120, the communication module 130, the input unit 140, the output unit 150 and the power supply 160 are respectively connected to the central processing unit 110. The memory 120 stores a computer program, and the central processing unit 110 can call the computer program. When the central processing unit 110 executes the computer program, all the steps in the dynamic allocation method of log storage partitions in the above embodiment are implemented.
[0124] The embodiment of the present application further provides a computer storage medium for storing a computer program, wherein the computer program can be executed by a processor. When the computer program is executed by the processor, any method for dynamically allocating log storage partitions provided by the present invention is implemented.
[0125] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for hardware + program embodiments and system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiments. Although the embodiments of this specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way of executing the order of many steps, and does not represent the only order of execution.
[0126] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of this specification. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory. The above description is only an embodiment of the embodiments of this specification and is not intended to limit the embodiments of this specification. For those skilled in the art, the embodiments of this specification may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of this specification shall be included within the scope of the claims of the embodiments of this specification.
Claims
1. A method for dynamically allocating log storage partitions, characterized in that: include: Respectively obtain a first write flow of log data of each application in any time period of a plurality of historical periods and a total write flow of log data of each application in each of the historical periods; Predicting a second write flow of log data of each application in the time period of the next cycle according to the first write flow and the total write flow; Dividing each application into M groups according to the second write traffic, and determining the size of the log storage partition allocated to each group of applications during the time period of the next cycle according to the second write traffic of the applications in each group; The predicting, according to the first write flow and the total write flow, of the second write flow of the log data of each application in the time period of the next cycle includes: Determine the total write traffic cycle representative value of log data of each application in multiple historical cycles respectively; Determine respectively a first period ratio of the first write flow of log data of each application in the period of each historical period to the total write flow of the corresponding historical period; Determine a second time period ratio of the write traffic of each application in the time period of the next cycle according to the first time period ratio; The second write flow of log data of each application in the time period of the next cycle is determined according to the second time period ratio of each application and the total write flow period representative value.
2. The method for dynamically allocating log storage partitions according to claim 1, characterized in that: The dividing the applications into M groups according to the second write traffic includes: sorting each application according to the second write traffic; The sorted applications are grouped according to a preset grouping ratio to obtain M groups.
3. The method for dynamically allocating log storage partitions according to claim 2, characterized in that: The determining, according to the second write traffic of the applications in each group, the size of the log storage partition allocated to the applications in each group during the time period of the next cycle includes: Determine a write flow representative value of the corresponding group according to the second write flow of the application in each group; According to the ratio of the write traffic representative value of each group to the total write traffic representative values of the M groups, the size of the log storage partition allocated to the application in the corresponding group in the first period of the next cycle is determined.
4. The method for dynamically allocating log storage partitions according to claim 3, characterized in that: The determining the write flow representative value of the corresponding group according to the second write flow of the application in each group respectively includes: sorting the second write traffic of the applications included in each group respectively; The median of the sorted second write traffic is used as the representative value of the write traffic of the corresponding group.
5. The method for dynamically allocating log storage partitions according to claim 3, characterized in that: The determining the write flow representative value of the corresponding group according to the second write flow of the application in each group respectively includes: respectively calculating an average value of the second write traffic of the applications included in each group; The average value of the second write flow rate of each group is taken as the representative value of the write flow rate of the corresponding group.
6. The method for dynamically allocating log storage partitions according to claim 4 or 5, characterized in that: When applications in the same group write log data, they automatically occupy the available storage space of the log storage partition of the corresponding group.
7. A dynamic allocation device for log storage partitions, characterized in that: include: A historical write flow acquisition module, used to respectively acquire the first write flow of log data of each application in any period of multiple historical cycles and the total write flow of log data of each application in each of the historical cycles; A write traffic prediction module, configured to predict a second write traffic of log data of each application in the time period of the next cycle according to the first write traffic and the total write traffic; a log storage partition allocation module, configured to divide each application into M groups according to the second write traffic, and determine the size of the log storage partition allocated to each group of applications during the time period of the next cycle according to the second write traffic of the applications in each group; Wherein, the write traffic prediction module includes: A periodic write traffic prediction unit, used to respectively determine a periodic representative value of the total write traffic of log data of each application in a plurality of historical periods; A period write traffic ratio determination unit, used to respectively determine a first period ratio of the first write traffic of the log data of each application in the period of each historical cycle to the total write traffic of the corresponding historical cycle; and determine a second period ratio of the write traffic of each application in the period of the next cycle according to the first period ratio; The time period write traffic prediction unit is used to determine the second write traffic of log data of each application in the time period of the next cycle according to the second time period ratio of each application and the total write traffic cycle representative value.
8. The device for dynamically allocating log storage partitions according to claim 7, characterized in that: The log storage partition allocation module includes an application grouping unit, which is used to: sorting each application according to the second write traffic; The sorted applications are grouped according to a preset grouping ratio to obtain M groups.
9. The dynamic allocation device for log storage partitions according to claim 8, characterized in that: The log storage partition allocation module also includes: a write traffic representative value determining unit, configured to determine a write traffic representative value of a corresponding group according to the second write traffic of the application in each group; The log storage partition allocation unit is used to determine the size of the log storage partition allocated to the application in the corresponding group in the first time period of the next cycle according to the proportion of the write traffic representative value of each group to the total write traffic representative values of the M groups.
10. An electronic device, characterized in that: include: A central processing unit, a memory, and a communication module, wherein a computer program is stored in the memory, the central processing unit can call the computer program, and when the central processing unit executes the computer program, the dynamic allocation method for log storage partitions according to any one of claims 1 to 6 is implemented.
11. A computer storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, the method for dynamically allocating log storage partitions according to any one of claims 1 to 6 is implemented.