Distributed data deployment method, apparatus, processor and electronic device

Abstract

The application discloses a distributed data deployment method and device, a processor and electronic equipment, and relates to the field of big data. The method comprises the following steps: determining the comprehensive performance score of each data node in N data nodes of a distributed cluster by using a performance evaluation algorithm; determining the data allocation ratio of each data node according to the comprehensive performance score; dividing to-be-deployed data into N data blocks according to the data allocation ratio, and distributing the N data blocks to the N data nodes, wherein each data block comprises a plurality of data files; periodically acquiring the access records of each data file in the N data nodes, clustering each data file according to the access records, and obtaining a plurality of data file sets; and re-deploying the plurality of data file sets to the N data nodes according to the comprehensive performance score. Through the application, the technical problem that the data deployment scheme of the distributed cluster in the related art cannot fully exert the node capacity and the overall performance of the cluster is low is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, and in particular, to a distributed data deployment method and device, a processor and an electronic device. BACKGROUND

[0002] In a distributed data system, data generated by an upper application is usually stored in different storage devices according to a special method, and a data deployment strategy is mainly used to solve the problem of how to select a storage device, which needs to meet certain specific goals, and different goals require different data deployment strategies. Hadoop is a software framework capable of distributed processing of large amounts of data. With the characteristics of high reliability, high scalability, high efficiency, low cost, open source, etc., it has become the preferred solution for processing massive data and has been put into practical industrial applications. In a Hadoop cluster, data deployment has three characteristics: 1) data is distributed; 2) data is replicated between nodes; 3) data is processed in parallel on each node. Data deployment determines the data characteristics of the node and affects the distribution and execution efficiency of the task, and thus affects the processing capacity of the entire cluster.

[0003] In related technologies, it is assumed that the computer nodes in the cluster are homogeneous, and the data is similar, that is, all nodes have the same computing performance, I / O performance and communication ability, and all data will be accessed in the same way. Based on this assumption, a consistent hash algorithm is used for balanced deployment when deploying data. However, in most cases, the performance of each node in the cluster is very different, and the data access in the cluster is also skewed due to business scenarios, and the balanced data deployment method cannot fully utilize the computing power of each node, resulting in long task execution time and low overall performance of the cluster.

[0004] At present, there is no effective solution to the above technical problems. SUMMARY

[0005] The main purpose of the present application is to provide a distributed data deployment method and device, a processor and an electronic device to solve the technical problem that the data deployment scheme of the related art distributed cluster cannot fully utilize the node capacity and the overall performance of the cluster is low.

[0006] According to an aspect of the present application, a distributed data deployment method is provided, which comprises: determining a comprehensive performance score of each data node in N data nodes of a distributed cluster by a preset performance evaluation algorithm, wherein N is a positive integer not less than 2; determining a data allocation ratio of each data node according to the comprehensive performance score of each data node; dividing to-be-deployed data into N data blocks according to the data allocation ratio of the N data nodes, and distributing the N data blocks to the N data nodes, wherein each data block comprises a plurality of data files; periodically acquiring access records of each data file in the N data nodes, clustering each data file according to the access records to obtain a plurality of data file sets; and re-deploying the plurality of data file sets to the N data nodes according to the comprehensive performance score of each data node.

[0007] Optionally, the determining of the comprehensive performance score of each data node in the N data nodes of the distributed cluster by the preset performance evaluation algorithm comprises: for each data node, performing performance evaluation calculation on the data node by M performance evaluation algorithms respectively to obtain M performance scores, and grouping the M performance scores into a performance vector of the data node, wherein M is a positive integer not less than 2; determining a first correlation relationship between the performance vectors of the N data nodes; constructing a relationship matrix according to each performance vector and the first correlation relationship, and performing reduction calculation on the relationship matrix by a web page ranking algorithm to obtain a ranking of the M performance evaluation algorithms; determining a weight value of the performance score corresponding to each performance evaluation algorithm according to the ranking; and for each data node, performing weighted calculation on the M performance scores of the data node according to the weight value to obtain the comprehensive performance score of the data node.

[0008] Optionally, the determining of the data allocation ratio of each data node according to the comprehensive performance score of each data node comprises: summing the comprehensive performance scores of the N data nodes to obtain a total performance score; and for each data node, taking the ratio of the comprehensive performance score of the data node to the total performance score as the data allocation ratio of the data node.

[0009] Optionally, the dividing of the to-be-deployed data into N data blocks according to the data allocation ratio of the N data nodes, and the distributing of the N data blocks to the N data nodes comprises: dividing data files in the to-be-deployed data according to the data allocation ratio of each data node respectively to obtain N data blocks corresponding to the N data nodes, wherein the proportion of the data block corresponding to each data node in the to-be-deployed data is the data allocation ratio of the data node; and for each data block, deploying the data block to the data node corresponding to the data block.

[0010] Optionally, periodically obtaining access records of each data file in the N data nodes, clustering each data file according to the access records to obtain a plurality of data file sets, comprising: obtaining access records of each data file in each unit time slot in each target period; determining a second association relationship between each data file according to the access records; clustering each data file according to the second association relationship to obtain a plurality of data file sets, wherein each data file set includes at least one data file.

[0011] Optionally, re-deploying the plurality of data file sets to the N data nodes according to the comprehensive performance scores of each data node, comprising: cyclically executing the following processes until the plurality of data file sets are deployed: sorting each undeployed data file set according to the capacity size; sorting the N data nodes according to the comprehensive performance score size; deploying the target data file set with the largest capacity to the data node with the highest comprehensive score, and updating the comprehensive performance score of the target data node.

[0012] Optionally, updating the comprehensive performance score of the target data node, comprising: determining the proportion of the capacity of the target data file set in the total capacity of all data file sets; reducing the comprehensive performance score of the target data node according to the proportion.

[0013] According to another aspect of the present application, a distributed data deployment device is also provided, comprising: a scoring module configured to determine the comprehensive performance score of each data node in the N data nodes of the distributed cluster by a preset performance evaluation algorithm, wherein N is a positive integer not less than 2; a determination module configured to determine the data allocation proportion of each data node according to the comprehensive performance score of each data node; a first deployment module configured to divide the to-be-deployed data into N data blocks according to the data allocation proportion of the N data nodes, and distribute the N data blocks to the N data nodes, wherein each data block includes a plurality of data files; a clustering module configured to periodically obtain access records of each data file in the N data nodes, and cluster each data file according to the access records to obtain a plurality of data file sets; and a second deployment module configured to re-deploy the plurality of data file sets to the N data nodes according to the comprehensive performance score of each data node.

[0014] According to another aspect of the present application, a processor is also provided, which is used to run a program, wherein the program performs the above-mentioned distributed data deployment method when running.

[0015] According to another aspect of the present application, an electronic device is also provided, which includes one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the above-described distributed data deployment method.

[0016] In the present application, first, the comprehensive performance score of each data node in the N data nodes of the distributed cluster is determined through a preset performance evaluation algorithm, then the data allocation ratio of each data node is determined according to the comprehensive performance score of each data node, and then the to-be-deployed data is divided into N data blocks according to the data allocation ratio of the N data nodes, and the N data blocks are distributed and deployed to the N data nodes, wherein each data block includes a plurality of data files, and through this way, the computing capacity of each node can be fully utilized; thereafter, the access records of each data file in the N data nodes are periodically obtained, and each data file is clustered according to the access records to obtain a plurality of data file sets, and the plurality of data file sets are re-deployed to the N data nodes according to the comprehensive performance score of each data node, which can reduce the data migration impact and improve the cluster performance, thereby solving the technical problems in the related art that the data deployment scheme of the distributed cluster cannot fully utilize the node capacity and the overall performance of the cluster is low. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and are used to interpret the illustrative embodiments of the present application and their descriptions, and do not constitute improper limitations to the present application. In the drawings:

[0018] Figure 1 is a flowchart of a distributed data deployment method according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of calculating the comprehensive performance score of a data node according to an embodiment of the present application;

[0020] Figure 3 is a structural schematic diagram of a Hadoop cluster system according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a distributed cluster data deployment process according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of a distributed data deployment apparatus according to an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the person skilled in the art better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should fall within the protection scope of the present application. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0025] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0026] The relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties. For example, an interface is provided between the system and the relevant user or institution. Before obtaining the relevant data, the interface needs to send a request to the aforementioned user or institution, and after receiving the consent data feedback from the aforementioned user or institution, the relevant data is obtained.

[0027] The present application will be described below in combination with the preferred implementation steps, Figure 1 The flow chart of the distributed data deployment method provided by the embodiments of the present application is shown in Figure 1 The method comprises the following steps:

[0028] Step S101, determining the comprehensive performance score of each data node in the N data nodes of the distributed cluster by a preset performance evaluation algorithm, wherein N is a positive integer not less than 2;

[0029] Step S102, determining the data allocation ratio of each data node according to the comprehensive performance score of each data node;

[0030] In step S103, the to-be-deployed data is divided into N data blocks according to the data distribution ratio of the N data nodes, and the N data blocks are distributed and deployed to the N data nodes, wherein each data block includes a plurality of data files.

[0031] In step S104, the access records of each data file in the N data nodes are periodically obtained, each data file is clustered according to the access records, and a plurality of data file sets are obtained.

[0032] In step S105, the plurality of data file sets are re-deployed to the N data nodes according to the comprehensive performance score of each data node.

[0033] Among them, for a plurality of data nodes in a distributed cluster, the comprehensive performance score of each data node is first determined by a performance evaluation algorithm, then the data distribution ratio of each data node is determined according to the comprehensive performance score, and then the to-be-deployed data is divided into a plurality of data blocks according to the data distribution ratio and distributed and deployed to the plurality of data nodes. Each data block includes a plurality of data files. Through this deployment method, the computing power of each data node can be fully utilized. Then, the access records of each data file in the plurality of data nodes are periodically obtained, each data file is clustered according to the access records, a plurality of data file sets are obtained, and the plurality of data file sets are re-deployed to the plurality of data nodes according to the comprehensive performance score of each data node. This can reduce the influence of data bias and improve the performance of the cluster, thereby effectively solving the technical problems in the related art that the data deployment scheme of the distributed cluster cannot fully utilize the node capacity and the overall performance of the cluster is low.

[0034] As an optional implementation, when the comprehensive performance score of each data node in the N data nodes of the distributed cluster is determined by a preset performance evaluation algorithm, the steps shown in FIG. 8 can be performed, specifically including: Figure 2

[0035] In step S201, for each data node, the performance of the data node is calculated by M performance evaluation algorithms, M performance scores are obtained, and the M performance scores are combined into a performance vector of the data node, wherein M is a positive integer not less than 2.

[0036] ​The performance evaluation algorithm can adopt the following algorithms: 1) a performance benchmark evaluation algorithm Linpack, which is a popular performance test benchmark in the world at present, and can evaluate the floating point performance of a high-performance computer system by testing the ability of the high-performance computer to solve a dense linear algebraic equation group; 2) an HPCG (High Performance Conjugate Gradient) benchmark test, which can evaluate the floating point performance by testing the ability of the high-performance computer to solve a sparse matrix equation group; 3) an NPB (NAS Parallel Benchmark) test, which is a set of application programs developed by the United States Aerospace Administration to represent fluid dynamics calculation, and is composed of 5 core programs and 3 simulation programs, and is used to evaluate the performance of large-scale parallel machines and supercomputers; and 4) an IDC (Internet Data Center) balance evaluation index, which is used for comprehensive detection and evaluation of a data center in terms of hardware equipment, software environment, security system, service level and the like.

[0037] In actual testing, the value of M and the selected performance evaluation algorithm can be adjusted as required, and are not specifically limited here.

[0038] In step S202, a first correlation relationship between the performance vectors of the N data nodes is determined.

[0039] The first correlation relationship can be reflected by the Euclidean distance. If the Euclidean distance between two performance vectors is less than a preset threshold, it is considered that there is no correlation between the two performance vectors. Otherwise, there is a correlation between the two performance vectors.

[0040] In step S203, a relationship matrix is constructed according to each performance vector and the first correlation relationship, and a ranking of the M performance evaluation algorithms is obtained by performing reduction calculation on the relationship matrix through a web page ranking algorithm.

[0041] The web page ranking algorithm PageRank is a function defined on a set of web pages, which gives a positive real number to each web page to represent the importance of the web page, and constitutes a vector as a whole. The higher the PageRank value, the more important the web page, and the web page can be ranked in the front in the sorting of Internet search. The basic idea of the algorithm is to define a random walk model on a directed graph, that is, a first-order Markov chain, to describe the behavior of a random walker visiting each node along the directed graph at random. Under certain conditions, the probability of visiting each node in the limit case converges to a stationary distribution, and the stationary probability value of each node is its PageRank value, which represents the importance of the node. PageRank is recursively defined, and its calculation can be performed through an iterative algorithm.

[0042] In actual calculation, step S203 can be performed in a loop until the results of the last two times are approximate or the same, at which time the ranking of each performance evaluation method can be obtained.

[0043] In step S204, the weight value of the performance score corresponding to each performance evaluation algorithm is determined according to the ranking.

[0044] In step S205, for each data node, the M performance scores of the data node are weighted and calculated according to the weight value to obtain a comprehensive performance score of the data node.

[0045] Specifically, a corresponding weight value can be set for different rankings in advance. For example, when M is 4, the weight value of the first ranking is 0.4, the weight value of the second ranking is 0.3, the weight value of the third ranking is 0.2, and the weight value of the fourth ranking is 0.1. For a certain data node, the performance score given by algorithm one is a, the performance score given by algorithm two is b, the performance score given by algorithm three is c, and the performance score given by algorithm four is d. Then the comprehensive performance score of the data node is S = 0.4a + 0.3b + 0.2c + 0.1d.

[0046] Optionally, in the embodiment of the present application, when determining the data allocation ratio of each data node according to the comprehensive performance score of each data node, the following steps can be performed: summing the comprehensive performance scores of the N data nodes to obtain a total performance score; for each data node, taking the ratio of the comprehensive performance score of the data node to the total performance score as the data allocation ratio of the data node.

[0047] For example, there are 5 data nodes, and their comprehensive performance scores are 90, 85, 80, 75, and 70, respectively. The total performance score is 400. Then the data allocation ratios of the 5 data nodes are 90 / 400 = 22.5%, 85 / 400 = 21.25%, 80 / 400 = 20%, 75 / 400 = 18.75%, and 70 / 400 = 17.5%, respectively.

[0048] After that, as an optional implementation, the data files in the data to be deployed can be divided according to the data allocation ratios of the respective data nodes to obtain N data blocks corresponding to the N data nodes, wherein the proportion of the data block corresponding to each data node in the data to be deployed is the data allocation ratio of the data node. For each data block, the data block is deployed to the data node corresponding to the data block.

[0049] Still taking the above 5 data nodes as an example, data files with data amounts of 22.5%, 21.25%, 20%, 18.75%, and 17.5% can be divided from the data to be deployed as 5 data blocks, respectively, and the 5 data blocks are deployed to the corresponding data nodes, respectively.

[0050] Through the above deployment mode, the computing capability of each data node can be fully utilized.

[0051] Considering that data access in the cluster may be skewed due to business scenarios, the deployed data files can also be adjusted according to the access records of the data files.

[0052] As an optional implementation, the access records of each data file in the N data nodes can be periodically obtained in the following manner, and each data file is clustered according to the access records, so as to obtain a plurality of data file sets: in each target period, the access records of each data file in each unit time slot are obtained; the second association relationship between each data file is determined according to the access records; each data file is clustered according to the second association relationship, and a plurality of data file sets are obtained, wherein each data file set includes at least one data file.

[0053] The target period is the period of adjusting the deployed data files of each data node in the cluster, which can be in units of weeks or months, and is adjusted according to requirements, which is not limited here; the unit time slot can be in units of seconds or minutes, and is also adjusted according to requirements, which is not limited here; the data file access record can include the specific access times.

[0054] Through the data file access records in the unit time slot, the association relationship between the data files can be indirectly reflected, for example, the user successively accesses data file 1 and data file 2 within 5s, and it is preliminarily judged that there is an association relationship between data file 1 and data file 2. Then, the determined association relationship is processed through a clustering algorithm, so that all data files can be divided into a plurality of data file sets. For each data file, the relationship between it and other data files in the same set is close, and the relationship between it and data files in other sets is distant.

[0055] By grouping and redeploying the data files according to the association relationship, the influence of data skew can be effectively reduced.

[0056] As an optional implementation, the plurality of data file sets can be redeployed to the N data nodes according to the comprehensive performance score of each data node in the following manner: the following process is executed in a loop until the plurality of data file sets are deployed: each undeployed data file set is sorted according to the capacity size; the N data nodes are sorted according to the comprehensive performance score; the target data file set with the largest capacity is deployed to the data node with the highest comprehensive performance score, and the comprehensive performance score of the target data node is updated.

[0057] Optionally, when updating the comprehensive performance score of the target data node, the proportion of the capacity of the target data file set in the total capacity of all data file sets can be determined first, and then the comprehensive performance score of the target data node is reduced according to the proportion.

[0058] It can be understood that deploying data file sets with larger capacity to data nodes with higher comprehensive performance scores can make full use of the computing capacity of the data nodes. However, the comprehensive performance score of a data node needs to be adjusted in real time after a data file set is deployed in the data node. Assuming that the total amount of data file sets is 100, the initial comprehensive performance score of a data node is 90, and a data file set with a capacity of 20 is deployed in the data node, the comprehensive performance score of the data node will be adjusted to 90*(1-20 / 100) = 72. Then, the deployment of the remaining data file sets is determined according to the new comprehensive performance score, until all data file sets are deployed.

[0059] In this way, the computing capacity of the data nodes can be fully utilized, and data skew can be avoided.

[0060] As an optional implementation, the distributed cluster in the embodiments of the present application can be a Hadoop cluster, Figure 3 An optional structure diagram of a Hadoop cluster system is shown, which includes a client 31, a master node (NameNode) 32, a backup node (Secondary NameNode) 33, data nodes (DataNode) 341, 342, 343, and local disks 351, 352. The client 31 includes a management module (DistributedFileSystem) 311, a node performance evaluation module (BenchmarkSystem) 312, a performance file distribution module (BenchmarkDistributedFileSystem) 313, a deployment module (FSDataOutputStream) 314, a file adjustment module (ReFileSystem) 315, and a redeployment module (ReFSDataOutputStream) 316. The master node 32 includes a record file (FsImage) 321, an operation log (EditLog) 322, and a performance database (RatioTable) 323.

[0061] When the Hadoop cluster is initialized, the performance database 323 is created in the master node 32, which is used to record the performance score, the comprehensive performance score, the performance evaluation method ranking, and other related data of the data nodes.

[0062] The node performance evaluation module 312 is used to intercept the file deployment request, execute after the file deployment request, calculate the comprehensive performance score of the data node, and record the result in the performance database 323;

[0063] The performance file distribution module 313 is executed after the node performance evaluation device, used to determine the deployment mode of the data file, and write it in the record file 321, and then deploy the data through the deployment module 314;

[0064] The file adjustment module 315 is periodically started, used to read the operation log 322 and the comprehensive performance score of the data node, re-determine the deployment of the data file, and write it in the record file 321, and then re-deploy the data through the re-deployment module 316.

[0065] Figure 4 An optional flowchart of distributed cluster data deployment is shown, and the specific steps are as follows:

[0066] S1, calculate the performance score of each data node;

[0067] S2, normalize the performance score;

[0068] S3, determine whether the result is similar to the last result, if not, re-execute step S2; if similar, execute step S4;

[0069] S4, determine the performance evaluation method ranking and determine the weight;

[0070] S5, calculate the comprehensive performance score of each data node;

[0071] S6, perform distributed deployment of data according to the comprehensive performance score;

[0072] S7, read the data file access record;

[0073] S8, cluster and group the data files according to the access record;

[0074] S9, sort the undeployed data file set by size;

[0075] S10, sort the data nodes by comprehensive performance score;

[0076] S11, deploy the largest data file set to the data node with the highest comprehensive performance score;

[0077] S12, reduce the comprehensive performance score of the data node according to the size of the data file set;

[0078] S13, determine whether there are still undeployed data file sets, if there are, re-execute step S9; if not, the process ends.

[0079] In the embodiment of the present application, for a plurality of data nodes in a distributed cluster, the comprehensive performance scores of the data nodes are determined by a performance evaluation algorithm, and then the data allocation ratios of the data nodes are determined according to the comprehensive performance scores, and then the to-be-deployed data is divided into a plurality of data blocks according to the data allocation ratios, and the plurality of data blocks are distributed to the plurality of data nodes, and each data block includes a plurality of data files. Through this deployment mode, the computing capacity of each data node can be fully utilized. Then, the access records of the data files in the plurality of data nodes are periodically obtained, and the data files are clustered according to the access records to obtain a plurality of data file sets, and the plurality of data file sets are re-deployed to the plurality of data nodes according to the comprehensive performance scores of the data nodes, which can reduce the influence of data skew and improve the performance of the cluster, thereby effectively solving the technical problems that the data deployment scheme of the distributed cluster in the related art cannot fully utilize the node capacity and the overall performance of the cluster is low.

[0080] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0081] The embodiment of the present application also provides a distributed data deployment device. It should be noted that the distributed data deployment device of the embodiment of the present application can be used to execute the distributed data deployment method provided by the embodiment of the present application. The distributed data deployment device provided by the embodiment of the present application is introduced as follows.

[0082] Figure 5 is a schematic diagram of the distributed data deployment device according to the embodiment of the present application. As shown in Figure 5 , the device includes a scoring module 51, a determination module 52, a first deployment module 53, a clustering module 54, and a second deployment module 55, wherein:

[0083] The scoring module 51 is configured to determine the comprehensive performance scores of the data nodes in the N data nodes of the distributed cluster by a preset performance evaluation algorithm, wherein N is a positive integer not less than 2;

[0084] The determination module 52 is configured to determine the data allocation ratios of the data nodes according to the comprehensive performance scores of the data nodes;

[0085] The first deployment module 53 is configured to divide the to-be-deployed data into N data blocks according to the data allocation ratios of the N data nodes, and distribute the N data blocks to the N data nodes, wherein each data block includes a plurality of data files;

[0086] The clustering module 54 is configured to periodically acquire access records of each data file in the N data nodes, cluster each data file according to the access records, and obtain a plurality of data file sets.

[0087] The second deployment module 55 is configured to re-deploy the plurality of data file sets to the N data nodes according to the comprehensive performance scores of the data nodes.

[0088] Optionally, in the distributed data deployment apparatus provided in the embodiments of the present application, the scoring module 51 comprises a first calculation unit, a first determination unit, a ranking unit, a second determination unit and a second calculation unit, and the functions of each unit are described below.

[0089] The first calculation unit is configured to calculate the performance of each data node by using M performance evaluation algorithms respectively, to obtain M performance scores, and to form a performance vector of the data node by using the M performance scores, wherein M is a positive integer greater than or equal to 2.

[0090] The performance evaluation algorithms can include the following algorithms: Linpack, HPCG benchmark test, NPB parallel benchmark test and IDC balance evaluation index. The value of M and the selected performance evaluation algorithms can be adjusted according to requirements in actual tests, and are not limited herein.

[0091] The first determination unit is configured to determine a first correlation relationship between the performance vectors of the N data nodes.

[0092] The first correlation relationship can be reflected by the Euclidean distance. If the Euclidean distance between two performance vectors is less than a preset threshold, it is considered that there is no correlation between the two performance vectors. Otherwise, there is a correlation between the two performance vectors.

[0093] The ranking unit is configured to construct a relationship matrix according to each performance vector and the first correlation relationship, and to calculate the relationship matrix by using a web page ranking algorithm to obtain the ranking of the M performance evaluation algorithms.

[0094] The web page ranking algorithm PageRank is recursively defined, and its calculation can be performed by using an iterative algorithm. Therefore, in actual calculation, step S203 can be executed in a loop until the results of the last two times are approximately the same or the same, at which time the ranking of each performance evaluation method can be obtained.

[0095] The second determination unit is configured to determine the weight value of the performance score corresponding to each performance evaluation algorithm according to the ranking.

[0096] The second calculation unit is configured to calculate a comprehensive performance score of each data node by weighting the M performance scores of the data node according to the weight values.

[0097] Optionally, when determining the data allocation ratio of each data node according to the comprehensive performance score of each data node, the determining module 52 can first sum the comprehensive performance scores of the N data nodes to obtain a total performance score; and for each data node, the ratio of the comprehensive performance score of the data node to the total performance score is taken as the data allocation ratio of the data node.

[0098] Optionally, the first deployment module 53 can divide the data files in the data to be deployed according to the data allocation ratio of each data node to obtain N data blocks corresponding to the N data nodes, wherein the proportion of the data block corresponding to each data node in the data to be deployed is the data allocation ratio of the data node; and for each data block, the data block is deployed to the data node corresponding to the data block.

[0099] Optionally, the clustering module 54 can periodically obtain the access records of each data file in the N data nodes and cluster each data file according to the access records, so as to obtain a plurality of data file sets, by the following method: in each target period, the access records of each data file in each unit time slot are obtained; a second association relationship between each data file is determined according to the access records; and each data file is clustered according to the second association relationship to obtain a plurality of data file sets, wherein each data file set includes at least one data file.

[0100] The target period is the period of adjusting the deployed data files of each data node in the cluster, which can be in units of weeks or months, and is adjusted according to the needs, which is not limited here; the unit time slot can be in units of seconds or minutes, and is also adjusted according to the needs, which is not limited here; the data file access record can include the specific access times.

[0101] The access records of the data files in the unit time slot can indirectly reflect the association relationship between the data files, for example, the user successively accesses data file 1 and data file 2 within 5s, and it is preliminarily judged that there is an association relationship between data file 1 and data file 2. Then, the determined association relationship is processed by a clustering algorithm, so that all the data files can be divided into a plurality of data file sets. For each data file, the relationship between the data file and other data files in the same set is close, and the relationship between the data file and data files in other sets is distant.

[0102] Optionally, the second deployment module 55 can re-deploy the plurality of data file sets to the N data nodes according to the comprehensive performance scores of the data nodes by: performing the following process in a loop until all the plurality of data file sets are deployed: sorting the data file sets that have not been deployed according to the capacity sizes; sorting the N data nodes according to the comprehensive performance scores; deploying a target data file set with the largest capacity to a data node with the highest comprehensive performance score, and updating the comprehensive performance score of the target data node.

[0103] In the updating of the comprehensive performance score of the target data node, the proportion of the capacity of the target data file set in the total capacity of all the data file sets can be determined first, and then the comprehensive performance score of the target data node is reduced according to the proportion.

[0104] It can be understood that deploying data file sets with larger capacities to data nodes with higher comprehensive performance scores can make full use of the computing power of the data nodes. After a data file set is deployed in a data node, the comprehensive performance score of the data node needs to be updated in real time, and then the deployment of the remaining data file sets is determined according to the new comprehensive performance scores until all the data file sets are deployed.

[0105] In the distributed data deployment apparatus provided in the embodiments of the present application, for the plurality of data nodes in the distributed cluster, the scoring module first determines the comprehensive performance scores of the data nodes by the performance evaluation algorithm, then the determining module determines the data allocation proportions of the data nodes according to the comprehensive performance scores, and then the first deployment module divides the data to be deployed into a plurality of data blocks according to the data allocation proportions and distributes the plurality of data blocks to the plurality of data nodes, each data block including a plurality of data files. Through this deployment method, the computing power of each data node can be fully utilized. Then, the clustering module periodically acquires the access records of the data files in the plurality of data nodes, and clusters the data files according to the access records to obtain a plurality of data file sets. The second deployment module re-deploys the plurality of data file sets to the plurality of data nodes according to the comprehensive performance scores of the data nodes, which can reduce the data skew effect and improve the cluster performance, thereby effectively solving the technical problems in the related art that the data deployment scheme of the distributed cluster cannot fully utilize the node capacity and the overall performance of the cluster is low.

[0106] Optionally, the distributed data deployment apparatus can include a processor and a memory, and the scoring module 51, the determining module 52, the first deployment module 53, the clustering module 54 and the second deployment module 55 are stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0107] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the distributed data deployment method described above can be implemented by adjusting kernel parameters.

[0108] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0109] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the aforementioned distributed data deployment method.

[0110] This invention provides a processor for running a program, wherein the program executes the above-described distributed data deployment method during runtime.

[0111] like Figure 6 As shown, this embodiment of the invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: determining the comprehensive performance score of each data node in a distributed cluster of N data nodes using a preset performance evaluation algorithm, where N is a positive integer not less than 2; determining the data allocation ratio of each data node based on the comprehensive performance score of each data node; dividing the data to be deployed into N data blocks according to the data allocation ratio of the N data nodes, and distributing the N data blocks to the N data nodes, where each data block includes multiple data files; periodically acquiring the access records of each data file in the N data nodes, clustering the data files according to the access records to obtain multiple data file sets; and redeploying the multiple data file sets to the N data nodes according to the comprehensive performance score of each data node.

[0112] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0113] The application further provides a computer program product, which is suitable for executing the program of the following method steps when executed on a data processing device: determining a comprehensive performance score of each data node in N data nodes of a distributed cluster by a preset performance evaluation algorithm, wherein N is a positive integer not less than 2; determining a data allocation ratio of each data node according to the comprehensive performance score of each data node; dividing to-be-deployed data into N data blocks according to the data allocation ratio of the N data nodes, and distributing the N data blocks to the N data nodes in a distributed manner, wherein each data block comprises a plurality of data files; periodically acquiring access records of each data file in the N data nodes, clustering each data file according to the access records, and obtaining a plurality of data file sets; and re-deploying the plurality of data file sets to the N data nodes according to the comprehensive performance score of each data node.

[0114] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0115] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce an apparatus for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0116] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product comprising instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0117] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0118] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0119] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), flash memory, or a combination of non-volatile memories in different types. The memory can also include a compact disk read only memory (CD-ROM), digital versatile disk (DVD), Blu-ray, or another non-transitory computer readable medium, which is non-volatile and non-transitory in nature, but volatile in that it can lose its content if the power to the computer is turned off or if the computer crashes. The memory is an example of a computer readable medium.

[0120] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carriers.

[0121] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0122] ​​The above merely provides an example of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall fall into the scope of claims of the present application.

Claims

1. A distributed data deployment method, characterized in that, include: The comprehensive performance score of each data node in the N data nodes of the distributed cluster is determined by a preset performance evaluation algorithm, where N is a positive integer not less than 2. The data allocation ratio for each data node is determined based on the comprehensive performance score of each data node. The data to be deployed is divided into N data blocks according to the data allocation ratio of the N data nodes, and the N data blocks are distributed and deployed to the N data nodes. Each data block includes multiple data files. The system periodically acquires access records for each data file in the N data nodes, and clusters the data files based on the access records to obtain multiple data file sets. Specifically, in each target period, it acquires access records for each data file within preset time slots; determines a second association relationship between the data files based on the access records; and clusters the data files based on the second association relationship to obtain multiple data file sets, wherein each data file set includes at least one data file. Based on the comprehensive performance score of each data node, the multiple data file sets are redeployed to the N data nodes.

2. The method according to claim 1, characterized in that, The overall performance score of each data node in the N data nodes of the distributed cluster is determined by a preset performance evaluation algorithm, including: For each data node, the performance of the data node is evaluated and calculated using M performance evaluation algorithms to obtain M performance scores. The M performance scores are then combined to form the performance vector of the data node, where M is a positive integer not less than 2. Determine the first correlation relationship among the performance vectors of the N data nodes; A relationship matrix is ​​constructed based on each of the performance vectors and the first association relationship, and the relationship matrix is ​​reduced and calculated using a webpage ranking algorithm to obtain the rankings of the M performance evaluation algorithms; The weight values ​​of the performance scores corresponding to each performance evaluation algorithm are determined based on the ranking. For each data node, the M performance scores of the data node are weighted according to the weight value to obtain the comprehensive performance score of the data node.

3. The method according to claim 1, characterized in that, The data allocation ratio for each data node is determined based on the comprehensive performance score of each data node, including: The overall performance score is obtained by summing the comprehensive performance scores of the N data nodes; For each data node, the proportion of the data node's overall performance score to the total performance score is used as the data allocation ratio for that data node.

4. The method according to claim 3, characterized in that, Based on the data allocation ratio of the N data nodes, the data to be deployed is divided into N data blocks, and the N data blocks are distributed and deployed to the N data nodes, including: The data files in the data to be deployed are divided according to the data allocation ratio of each data node to obtain N data blocks corresponding to the N data nodes, wherein the proportion of the data block corresponding to each data node in the data to be deployed is the data allocation ratio of the data node. For each data block, the data block is deployed to the data node corresponding to the data block.

5. The method according to claim 1, characterized in that, Based on the comprehensive performance score of each data node, the multiple data file sets are redeployed to the N data nodes, including: Repeat the following process until all the multiple data file sets have been deployed: Sort the collections of undeployed data files according to their size; The N data nodes are sorted according to their overall performance scores. Deploy the largest set of target data files to the data node with the highest overall score, and update the overall performance score of the target data node.

6. The method according to claim 5, characterized in that, Update the overall performance score of the target data node, including: Determine the proportion of the target data file set's capacity to the total capacity of all data file sets; The overall performance score of the target data node is reduced according to the stated ratio.

7. A distributed data deployment device, characterized in that, include: The scoring module is used to determine the comprehensive performance score of each data node among the N data nodes of the distributed cluster through a preset performance evaluation algorithm, where N is a positive integer not less than 2. The determination module is used to determine the data allocation ratio of each data node based on the comprehensive performance score of each data node; The first deployment module is used to divide the data to be deployed into N data blocks according to the data allocation ratio of the N data nodes, and to distribute the N data blocks to the N data nodes in a distributed manner, wherein each data block includes multiple data files; A clustering module is used to periodically obtain access records of each data file in the N data nodes, and cluster each data file according to the access records to obtain multiple data file sets. In each target period, access records of each data file in each preset unit time slot are obtained; a second association relationship between each data file is determined according to the access records; and the data files are clustered according to the second association relationship to obtain multiple data file sets, wherein each data file set includes at least one data file. The second deployment module is used to redeploy the multiple data file sets to the N data nodes based on the comprehensive performance score of each data node.

8. A processor, characterized in that, The processor is used to run a program, wherein the program executes the distributed data deployment method according to any one of claims 1 to 6 when it runs.

9. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the distributed data deployment method according to any one of claims 1 to 6.

Images