A method for constructing a hadoop parameter optimization model

Abstract

The present invention relates to the field of distributed processing technology, and in particular to a method for constructing a hadoop parameter optimization model, comprising: using a server to collect the amount of data generated by each data source within a certain period of time; analyzing the characteristics of a single data source, and assigning a characteristic value in proportion to the amount of data generated by the data source, and using the server to estimate the size of a file to be processed according to the characteristic value; the server collects the resource inventory of each preparation node in a normal operating state in a certain period of time, and groups them; the server estimates the number of nodes and processing time according to the size of the file to be processed; the server adjusts the parameters of hadoop according to the server's estimated number of nodes and processing time. By analyzing the characteristics of the data source, assigning a characteristic value, estimating the file size according to the characteristic value, grouping the hadoop distributed nodes, and adjusting the parameters of hadoop according to the file size and the node group, the resources of the hadoop project are saved.

Description

Technical Field

[0001] The invention relates to a Hadoop optimization method, and in particular to a method for constructing a Hadoop parameter optimization model. Background Art

[0002] As the data generated by information continues to expand, Hadoop is widely used as an important means of processing and solving large files. In applications, the adjustment of Hadoop configuration parameters plays a vital role in the overall computing efficiency and resource utilization. Chinese Patent Publication No. CN104317610A discloses "a method and device for automatic installation and deployment of Hadoop platform", which uses a host cluster to load the terminal node of Hadoop and adjusts the necessary parameters to the default parameters. Chinese Patent Publication No. CN103064664A discloses "a method and system for automatic optimization of Hadoop parameters based on performance estimation", which uses simulated operations on Hadoop projects to adjust Hadoop parameters so that it can reduce costs. Chinese Patent Publication No. CN104750780A discloses "a method for optimizing Hadoop configuration parameters based on statistical analysis", which uses classification of applications with strong characteristics and establishment of prediction models to guide the optimization of Hadoop parameters.

[0003] It can be seen that the above method and system have the following problems: when the information source of the project is in various states, it is difficult to judge the scale of the project, and it is also difficult to achieve the purpose of saving resources for the parameter optimization of Hadoop. Summary of the invention

[0004] To this end, the present invention provides a method for constructing a hadoop parameter optimization model, which is used to overcome the problem in the prior art that when the information source of the project is in multiple states, it is difficult to judge the scale of the project, and it is also difficult to achieve the purpose of saving resources for hadoop parameter optimization.

[0005] To achieve the above object, the present invention provides a method for constructing a hadoop parameter optimization model, comprising:

[0006] Step S1, using a server to collect the amount of data generated by each data source within a certain period of time and analyze the maximum and minimum values ​​of data generated by a single grid data source within a preset time;

[0007] Step S2, analyzing the characteristics of a single data source based on experience and entering the characteristics into the server, the server assigning a characteristic value to each data source in proportion to the amount of data generated by each data source, and estimating the size of the file to be processed based on the characteristic value;

[0008] Step S3, the server collects the resource inventory of each standby node in a normal operation state at a certain time period, and groups the standby nodes according to the size of the resource inventory and time;

[0009] Step S4, the server estimates the number of nodes and processing time according to the size of the file to be processed;

[0010] Step S5, adjusting the parameters of Hadoop according to the number of nodes and processing time estimated by the server.

[0011] Furthermore, the amount of data generated by the data source within a preset period is D, and D changes regularly within a preset time T;

[0012] For the amount of data Dij generated by the i-th data source when the preset time T is evenly divided into j time periods, there is a maximum value maxDij and a minimum value minDij, and i is set to 1, 2, 3, ..., N, j to 1, 2, 3, ..., m.

[0013] Furthermore, the maximum amount of data generated by the data source within the preset time T is recorded as maxDT, and the minimum amount of data generated within the preset time T is recorded as minDT, and maxDT=maxDij×n×m, minDT=minDij×n×m is set;

[0014] MinDT is used as the standard file size, and the hadoop standard parameter A corresponding to minDT is set, and grouping is performed based on A according to the running status of the standby node.

[0015] Furthermore, for a single standby node, the resource inventory R of the standby node has a maximum value maxR within a preset time T, and a first preset resource inventory R1 and a second preset resource inventory R2 are set, wherein R1=0.3maxR, R2=0.7maxR,

[0016] If R<R1, the server determines that the resource stock of the standby node is insufficient, and records the period of time when the node is in this situation as an unavailable period;

[0017] If R1≤R<R2, the server determines that the resource stock of the standby node is low, and records the period of the node in this situation as an inefficient period;

[0018] If R≥R2, the server determines that the resource stock of the standby node is relatively high, and records the period of time when the node is in this situation as a high-efficiency period.

[0019] Furthermore, for the kth preparation node, the state P in the jth time period kjAssignment, where k = 1, 2, 3, ..., n,

[0020] If the period is recorded as unavailable, P kj Assign a value of 0,

[0021] If the period is recorded as an inefficient period, P kj Assign a value of 1,

[0022] If the period is recorded as an efficient period, P kj Assign a value of 2,

[0023] The standard parameter A is used to group the preparation nodes according to the optimal number of nodes NA that meets A, so that the group of nodes is in P kj The project with the data volume minDT is completed after the optimal execution time tA under the state.

[0024] Further, for the kth preparation node,

[0025] When j+t≤T,

[0026] If P kj =P kj+1 =…=P kj+t ≥1, the server records the node as a stable node in the (j, j+t) period and includes it in the group;

[0027] If P kj =P kj+1 =…=P kj+t =0, the server regards the node as an unavailable node for the period (j, j+t);

[0028] When j+t>T, the server determines that the node is unavailable.

[0029] Furthermore, the average of the maximum data volume maxDT and the minimum data volume minDT is The hadoop parameter is A', where the optimal number of nodes of A' is NA', the optimal execution time is tA', and the function of the optimal number of nodes and the optimal execution time is set to f(D), where f(D) = N×t,

[0030] After obtaining the data volume D, the server determines the adjustment method of the hadoop parameters of the job according to the function f(D).

[0031] Furthermore, the data volume Dij corresponds to a characteristic attribute of the time period j, and its characteristic attribute affects the scale of the data volume Dij. In the time period T, Dij is multiplied by the proportion of the characteristic attribute of j in the time period T, and the approximate value of DT is obtained after summing up as the reference data for the estimated data volume to optimize the parameters of Hadoop.

[0032] Furthermore, if each of the nodes cannot be arranged when the maximum data volume maxDT is more than 70%, the node group operates normally.

[0033] Furthermore, the preset time is not continuous, and the processing time is continuous.

[0034] Compared with the prior art, the beneficial effect of the present invention lies in that, by analyzing the characteristics of the data source, assigning characteristic values, estimating the file size according to the characteristic values, grouping the Hadoop distributed nodes, allocating node groups according to the file size, and adjusting the Hadoop parameters according to the file size and node group, the resources of the Hadoop project are saved.

[0035] Furthermore, by classifying the amount of data generated by the data source according to time and scale, the classification efficiency of the data source is effectively improved while avoiding incorrect judgment of the data scale due to unclear data source classification, thereby further saving the resources of the Hadoop project.

[0036] Furthermore, by estimating the maximum and minimum scales that a data source can reach within a certain period of time and setting the minimum data volume as a benchmark group, this method effectively improves the rationality of computing resource allocation and further saves Hadoop project resources while avoiding the waste of computing power due to the lack of reference for file segmentation.

[0037] Furthermore, by classifying the nodes, the nodes with heavy workload outside the Hadoop project can be separated from the nodes with light workload. This can avoid the time waste caused by insufficient processing power of a single node and effectively improve the reliability of node work, thereby further saving resources of the Hadoop project.

[0038] Furthermore, by assigning values ​​to the working status of the nodes, the nodes are classified, which not only avoids uneven node processing capabilities, but also effectively improves the reliability of node work, thereby further saving resources of the Hadoop project.

[0039] Furthermore, by comparing the node work loading value with the working period, the period with the highest node work efficiency is determined, and the nodes are grouped. This avoids the increase in project duration due to different node processing efficiency, and effectively improves the reliability of node work, thereby further saving Hadoop project resources.

[0040] Furthermore, by setting the function of the number of nodes, execution time and data volume, the completion result of the Hadoop project is estimated, which reduces the amount of calculation and avoids resource shortage or waste caused by inaccurate data scale estimation, thereby further saving the resources of the Hadoop project.

[0041] Furthermore, by adjusting the estimated attributes of the data source through the time attribute, the stability of the Hadoop project is improved, thereby further saving the resources of the Hadoop project, while avoiding the abnormal increase in data volume due to special time, which makes the estimation inaccurate.

[0042] Furthermore, by obtaining the maximum amount of data, the uncertainty of project estimation caused by too small node grouping is avoided, and the system's resistance to interference is improved, thereby saving resources of the Hadoop project.

[0043] Furthermore, by setting the continuity of time, the abnormal increase of data volume caused by continuous information collection is avoided, and the stability of the Hadoop project is improved, thereby further saving the resources of the Hadoop project. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The present invention is a flowchart of a method for constructing a Hadoop parameter optimization model. DETAILED DESCRIPTION

[0045] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0047] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0048] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] See also Figure 1 As shown, it is a flow chart of a method for constructing a hadoop parameter optimization model according to the present invention, comprising:

[0050] Step S1, using a server to collect the amount of data generated by each data source within a certain period of time and analyze the maximum and minimum values ​​of data generated by a single grid data source within a preset time;

[0051] Step S2, analyzing the characteristics of a single data source based on the experience of the server and entering the characteristics into the server, wherein the server assigns a characteristic value to each data source in proportion to the amount of data generated by each data source, and estimates the size of the file to be processed based on the characteristic value;

[0052] Step S3, the server collects the resource inventory of each standby node in a normal operation state at a certain time period, and groups the standby nodes according to the size of the resource inventory and time;

[0053] Step S4, the server estimates the number of nodes and processing time according to the size of the file to be processed;

[0054] Step S5: the server adjusts the parameters of Hadoop according to the number of nodes and processing time estimated by the server.

[0055] By analyzing the characteristics of the data source, assigning characteristic values, estimating the file size based on the characteristic values, grouping the Hadoop distributed nodes, allocating node groups based on the file size, and adjusting the Hadoop parameters based on the file size and node group, the resources of the Hadoop project can be saved.

[0056] Specifically, the amount of data generated by the data source within a preset period is D, and D changes regularly within a preset time T;

[0057] For the amount of data Dij generated by the i-th data source when the preset time T is evenly divided into j time periods, there is a maximum value maxDij and a minimum value minDij, and i is set to 1, 2, 3, ..., N, j to 1, 2, 3, ..., m.

[0058] By classifying the amount of data generated by the data source according to time and scale, the classification efficiency of the data source is effectively improved, while avoiding incorrect judgment of the data scale due to unclear data source classification, thereby further saving the resources of the Hadoop project.

[0059] Specifically, the maximum amount of data generated by the data source within the preset time T is recorded as maxDT, and the minimum amount of data generated within the preset time T is recorded as minDT, and maxDT=maxDij×n×m, minDT=minDij×n×m is set;

[0060] MinDT is used as the standard file size, and the hadoop standard parameter A corresponding to minDT is set, and grouping is performed based on A according to the running status of the standby node.

[0061] By estimating the maximum and minimum scales that a data source can reach within a certain period of time and setting the minimum data volume as a benchmark group, this method avoids the waste of computing power due to the lack of reference for file segmentation and effectively improves the rationality of the allocation of computing power resources, thereby further saving the resources of the Hadoop project.

[0062] Specifically, for a single standby node, the resource inventory R of the standby node has a maximum value maxR within a preset time T, and a first preset resource inventory R1 and a second preset resource inventory R2 are set, wherein R1 = 0.3maxR, R2 = 0.7maxR,

[0063] If R<R1, the server may determine that the resource stock of the standby node is insufficient, and record the period of time when the node is in this situation as an unavailable period;

[0064] If R1≤R<R2, the server may determine that the resource stock of the standby node is low, and record the period of the node in this situation as an inefficient period;

[0065] If R≥R2, the server may determine that the resource stock of the standby node is relatively high, and record the period of time when the node is in this situation as a high-efficiency period.

[0066] By classifying the nodes, the nodes with heavy workload outside the Hadoop project are separated from the nodes with light workload. This avoids the time waste caused by insufficient processing power of a single node and effectively improves the reliability of node work, thereby further saving resources of the Hadoop project.

[0067] Specifically, for the kth preparation node, the state P in the jth time period kjAssignment, where k = 1, 2, 3, ..., n,

[0068] If the period is recorded as unavailable, P kj Assign a value of 0,

[0069] If the period is recorded as an inefficient period, P kj Assign a value of 1,

[0070] If the period is recorded as an efficient period, P kj Assign a value of 2,

[0071] The standard parameter A is used to group the preparation nodes according to the optimal number of nodes NA that meets A, so that the group of nodes is in P kj Under the optimal execution time tA, the project with a data volume of minDT can be completed.

[0072] By assigning values ​​to the working status of the nodes, the nodes are classified, which not only avoids uneven node processing capabilities, but also effectively improves the reliability of node work, thereby further saving resources of the Hadoop project.

[0073] Specifically, for the kth preparation node,

[0074] When j+t≤T,

[0075] If P kj =P kj+1 =…=P kj+t ≥1, the server records the node as a stable node in the (j, j+t) period and includes it in the group;

[0076] If P kj =P kj+1 =…=P kj+t =0, the server regards the node as an unavailable node for the period (j, j+t);

[0077] When j+t>T, the server determines that the node is unavailable.

[0078] By comparing the node work loading value with the working period, the period with the highest node work efficiency is determined and the nodes are grouped. This avoids the increase in project duration due to different node processing efficiency and effectively improves the reliability of node work, thereby further saving resources of the Hadoop project.

[0079] Specifically, the average of the maximum data volume maxDT and the minimum data volume minDT is The hadoop parameter is A', where the optimal number of nodes of A' is NA', the optimal execution time is tA', and the function of the optimal number of nodes and the optimal execution time is set to f(D), where f(D) = N×t,

[0080] After obtaining the data volume D, the adjustment method of the hadoop parameters of the job can be determined.

[0081] By setting the number of nodes, execution time and data volume as a function, the completion result of the Hadoop project is estimated. This reduces the amount of computation and avoids resource shortage or waste caused by inaccurate data scale estimation, thereby further saving the resources of the Hadoop project.

[0082] Specifically, the data volume Dij corresponds to a characteristic attribute of the time period j, and its characteristic attribute affects the scale of the data volume Dij. In the time period T, Dij is multiplied by the proportion of the characteristic attribute of j in the time period T, and the approximate value of DT is obtained after the sum as the estimated data volume for hadoop parameter optimization reference data. For example, when processing the information of a shopping software for 7 consecutive days, the 7 days include 2 rest days, 1 specific shopping festival, 1 travel discount day and 4 working days. The server can assign 1 to the working day, 1.5 to the rest day, 5 to the specific shopping festival, and 0.5 to the travel discount day. Then the data volume Di' generated by the data source in the 7 consecutive days is Di'=Di×(1×4+1.5×2+0.5×1+5×1). If the travel discount day overlaps with a rest day, then the data volume Di' generated by the data source in the 7 consecutive days is

[0083] By adjusting the estimated properties of the data source through the time attribute, the inaccurate estimation caused by the abnormal increase of data volume due to special time is avoided, and the stability of the Hadoop project is improved, thereby further saving the resources of the Hadoop project.

[0084] Specifically, if each of the nodes cannot be deployed when the maximum data volume maxDT is more than 70%, the node group can run. By obtaining the maximum data volume, the uncertainty of project estimation caused by too small node grouping is avoided, and the interference resistance of the system is improved, thereby saving resources of the Hadoop project.

[0085] Specifically, the preset time is not continuous, and the processing time is continuous. By setting the continuity of time, the abnormal increase of data volume caused by continuous information collection is avoided, and the stability of the Hadoop project is improved, thereby further saving the resources of the Hadoop project.

[0086] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for constructing a hadoop parameter optimization model, characterized in that: Include: Step S1, using a server to collect the amount of data generated by each data source within a certain period of time and analyze the maximum and minimum values ​​of data generated by a single grid data source within a preset time; Step S2, analyzing the characteristics of a single data source based on experience and entering the characteristics into the server, the server assigning a characteristic value to each data source in proportion to the amount of data generated by each data source, and estimating the size of the file to be processed based on the characteristic value; Step S3, the server collects the resource inventory of each standby node in each period under the normal operation state at a certain time period, and groups the standby nodes according to the size of the resource inventory and time; Step S4, the server estimates the number of nodes and processing time according to the size of the file to be processed; Step S5, adjusting the parameters of Hadoop according to the number of nodes and processing time estimated by the server; The amount of data generated by the data source within a preset period is D, and D changes regularly within a preset time T; For the amount of data Dij generated by the i-th data source when the preset time T is evenly divided into j time periods, there is a maximum value maxDij and a minimum value minDij, and i=1,2,3,…,N, j=1,2,3,…,m; The maximum amount of data generated by the data source within the preset time T is recorded as maxDT, and the minimum amount of data generated within the preset time T is recorded as minDT, and maxDT=maxDij×n×m, minDT=minDij×n×m is set; Take minDT as the standard file size, set the hadoop standard parameter A corresponding to minDT, and group the nodes based on A according to the running status of the standby nodes; For a single standby node, the resource inventory R of the standby node has a maximum value maxR within a preset time T, and a first preset resource inventory R1 and a second preset resource inventory R2 are set, wherein R1=0.3maxR, R2=0.7maxR, If R<R1, the server determines that the resource stock of the standby node is insufficient, and records the period of time when the node is in this situation as an unavailable period; If R1≤R<R2, the server determines that the resource stock of the standby node is low, and records the period of the node in this situation as an inefficient period; If R≥R2, the server determines that the resource stock of the standby node is high, and records the period of the node in this case as a high-efficiency period; For the kth preparation node, the state P in the jth time period kj Assignment, where k = 1, 2, 3, ..., n, If the period is recorded as unavailable, P kj Assign a value of 0, If the period is recorded as an inefficient period, P kj Assign a value of 1, If the period is recorded as an efficient period, P kj Assign a value of 2, The standard parameter A is used to group the preparation nodes according to the optimal number of nodes NA that meets A, so that the group of nodes is in P kj The project with the data volume minDT is completed after the optimal execution time tA under the state.

2. The method for constructing a hadoop parameter optimization model according to claim 1, characterized in that: For the kth preparation node, When j+t≤T, If P kj =P kj+1 =…=P kj+t ≥1, the server records the node as a stable node in the (j, j+t) period and includes it in the group; If P kj =P kj+1 =…=P kj+t =0, the server regards the node as an unavailable node for the period (j, j+t); When j+t>T, the server determines that the node is unavailable.

3. The method for constructing a hadoop parameter optimization model according to claim 2, characterized in that: The average of the maximum data volume maxDT and the minimum data volume minDT is The hadoop parameter is A', where the optimal number of nodes of A' is NA', the optimal execution time is tA', and the function of the optimal number of nodes and the optimal execution time is set to f(D), where f(D) = N×t, After obtaining the data volume D, the server calculates the adjustment method of the hadoop parameters of the job according to the function f(D).

4. The method for constructing a hadoop parameter optimization model according to claim 3, characterized in that: The data volume Dij corresponds to a characteristic attribute of the time period j. Its characteristic attributes affect the scale of data volume Dij. In the time period T, Dij is multiplied by the proportion of characteristic attributes of j in the time period T, and the approximate value of DT is obtained after summing up as the reference data for estimating the data volume for hadoop parameter optimization.

5. The method for constructing a hadoop parameter optimization model according to claim 4, characterized in that: If each of the nodes cannot be arranged when the maximum data volume maxDT is more than 70%, the node group operates normally.

6. The method for constructing a hadoop parameter optimization model according to claim 1, characterized in that: The preset time is not continuous, but the processing time is continuous.

Images