Method and system for dynamically determining copy position of HDFS (Hadoop Distributed File System) based on node historical state

By building node load evaluation indicators and using LSTM neural network for load prediction, dynamically determine the placement of file copies in HDFS cluster, solving the problem of load imbalance in heterogeneous cluster environments, achieving more efficient load balancing and system performance improvement.

CN119513061APending Publication Date: 2025-02-25CHINA TELECOM CLOUD TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311698027.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-12-12
Publication Date
2025-02-25

AI Technical Summary

Technical Problem

The existing HDFS replica strategy lacks load distinction in heterogeneous cluster environments, ignoring the performance of nodes, resulting in load imbalance, affecting system performance and reliability.

Method used

By constructing node load evaluation indicators, using LSTM neural network for load prediction, dynamically determine the placement of file replicas, and preferring to select nodes with lower load in the future for file replica writing.

Benefits of technology

It realizes load balancing in a heterogeneous cluster environment, improves the overall performance and stability of the system, and enhances the flexibility and applicability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119513061A_ABST
    Figure CN119513061A_ABST
Patent Text Reader

Abstract

The invention provides a method and system for dynamically determining a copy position of an HDFS (Hadoop Distributed File System) based on a node historical state, and the method comprises the steps: constructing a node load evaluation index, recording the current load of a node at regular time, carrying out the training through an LSTM neural network, and constructing a node load prediction model for predicting the future load of a cluster node, determining a copy creation position of the file in the cluster according to the predicted value; when a file copy is written, the load of each node at the next moment is predicted, and nodes with relatively low loads in the future are written preferentially. According to the method, the write-in positions of the file copies in the HDFS cluster are dynamically set, so that load balancing in a heterogeneous environment can be realized. Compared with an existing fixed copy strategy, the method is more flexible, and the availability of the system is enhanced. Compared with an existing replica placement strategy based on the current node state, burst state interference is eliminated, the balance is higher, meanwhile, the prediction accuracy is higher through the neural network, and the overall performance of the system can be better improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the intersection of deep learning and big data, and in particular to a method and system for dynamically determining a replica location of an HDFS based on a node's historical status. Background Art

[0002] The default replication strategy of HDFS is based on rack awareness. When uploading files to the HDFS cluster, the replica placement is determined according to the relative relationship between nodes and racks. The placement process is as follows: Figure 1 Although this method does not evenly distribute file copies in the cluster, it reduces the amount of data transmission between multiple racks and reduces network bandwidth usage while ensuring data reliability and read performance.

[0003] However, the replica placement strategy assumes that all nodes in the cluster are homogeneous by default, with the same processor frequency, number of cores, memory, and hard disk capacity. However, in actual production environments, multiple batches and multiple specifications of equipment often appear in the cluster, and there is obvious heterogeneity between the nodes.

[0004] In addition, the running load of each node in the cluster is also different. For example, the resources in some nodes often need to perform computing tasks, which will occupy a large amount of CPU resources. The resources in some nodes are often requested concurrently, requiring a large number of I / O operations, and the disk and network bandwidth are high. It is inaccurate to evaluate the node load based on the current status of a single indicator. It is necessary to comprehensively consider multiple influencing factors and historical data for evaluation.

[0005] Therefore, dynamically determining the replica location based on the node specifications and load within the cluster is particularly important in distributed storage technology, and is also the key to improving the overall system response speed and enhancing data reliability. Summary of the invention

[0006] In view of this, the replica strategy is not able to differentiate the load of heterogeneous clusters enough, and in order to solve the problem of ignoring node performance when placing file replicas in the HDFS cluster, the present invention proposes a method and system for dynamically determining the replica location of HDFS based on the node historical status, by establishing a node load evaluation index and predicting the future load of the node, dynamically setting the placement location of the file replica, and realizing load balancing in a cluster heterogeneous environment, so as to improve the overall performance of the system.

[0007] Based on the above purpose, in a first aspect, the present invention provides a method for dynamically determining a replica location based on a node's historical status in HDFS, comprising the following steps:

[0008] Construct node load evaluation indicators to measure the current load of the node;

[0009] Regularly record the current load of the node to provide data support for load prediction;

[0010] Use LSTM neural network for training and build a node load prediction model to predict the future load of cluster nodes, and determine the location where the copy of the file in the cluster is created based on the predicted value;

[0011] When writing file copies, the future load of the node is dynamically calculated, the load of each node at the next moment is predicted, and the nodes with lower future load are written first.

[0012] As a further solution of the present invention, when constructing the node load evaluation index, assuming that the CPU main frequency of the node is f, the number of cores is n, the performance conversion coefficient in the multi-core environment is t, and the CPU performance C of the node k is k It is expressed as:

[0013]

[0014] The CPU load of node k at time t is

[0015] As a further solution of the present invention, when constructing the node load evaluation index, let the memory capacity of the node be r, the memory rate coefficient be u, and the memory performance M of the node k be k It is expressed as:

[0016] M k =u k r k

[0017] The memory load of the node at time t is

[0018] As a further solution of the present invention, when constructing the node load evaluation index, let the HDFS disk capacity of the node be e, the hard disk rate coefficient be b, and the disk performance of node k be H. k It is expressed as:

[0019] H k =b k e k

[0020] The disk load of the node at time t is

[0021] As a further solution of the present invention, based on the CPU performance C of node k k , the memory performance M of node k k and the disk performance H of node k k , the comprehensive load of the node at time t is:

[0022]

[0023] α+β+γ=1

[0024] Among them, α is the CPU impact coefficient, β is the memory impact coefficient, and γ is the disk impact coefficient.

[0025] As a further solution of the present invention, when the current load of the node is recorded regularly, a scheduled task is set to record the load status of each node in the cluster every hour.

[0026] As a further solution of the present invention, the node load prediction model uses the node load data at equal time intervals to predict the node load in the future time period based on the LSTM network, including data preprocessing, construction and training of the LSTM network, and the steps are:

[0027] Data preprocessing: normalize the data using the maximum and minimum normalization functions;

[0028] LSTM model parameters: The model input layer uses one hundred memory units, the hidden layer uses one hundred memory units, and the output layer uses a single unit;

[0029] Optimizer and loss function: Adaptive moment estimation (Adam) is used as the optimizer and mean absolute error (MAE) is selected as the loss function;

[0030] Evaluation index: The root mean square error (RMSE) is selected as the evaluation index of the prediction result to measure the deviation between the prediction result and the true value.

[0031] As a further solution of the present invention, the LSTM model parameter setting includes an input layer of one hundred memory units, a hidden layer of one hundred memory units, and an output layer of a single unit.

[0032] In a second aspect, the present invention also provides a system for dynamically determining a replica location based on a node's historical status in HDFS, comprising the following modules:

[0033] The node load evaluation module is used to construct node load evaluation indicators and regularly record the current load of the node;

[0034] The load prediction module includes an LSTM neural network, which is used to train the node load prediction model and determine the location of the replica creation of the file in the cluster based on the predicted value;

[0035] The file writing module dynamically calculates the future load of the node, predicts the load of each node at the next moment, and gives priority to nodes with lower future load for writing file copies.

[0036] As a further solution of the present invention, when the node load evaluation module records the current load of the node at a regular interval, the node load evaluation module records the load status of each node in the cluster every hour by setting a regular task.

[0037] As a further solution of the present invention, the LSTM neural network of the load prediction module includes the following steps:

[0038] The data preprocessing step is used to normalize the node load data using the maximum and minimum normalization function;

[0039] Model parameter settings, including an input layer of one hundred memory units, a hidden layer of one hundred memory units, and an output layer of a single unit;

[0040] Optimizer and loss function settings, using adaptive moment estimation (Adam) as the optimizer and selecting mean absolute error (MAE) as the loss function;

[0041] Evaluation index setting, root mean square error (RMSE) is selected as the evaluation index of the prediction results.

[0042] As a further solution of the present invention, when a file copy needs to be written, the file writing module predicts the future load and preferentially selects a node with a lower future load to write the file copy, so as to achieve load balancing.

[0043] In another aspect of the present invention, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, any one of the above methods for dynamically determining the replica location based on the HDFS historical status of the node according to the present invention is executed.

[0044] In another aspect of the present invention, a computer-readable storage medium is provided, storing computer program instructions, which, when executed, implement any of the above methods for dynamically determining replica locations based on HDFS node historical status according to the present invention.

[0045] Compared with the prior art, the method and system for dynamically determining the replica location of HDFS based on the node historical status provided by the present invention have the following beneficial effects:

[0046] 1. Load balancing optimization: By building node load evaluation indicators and using LSTM neural networks for load prediction, this system can accurately predict the future load of cluster nodes. This enables the system to dynamically calculate the future load of nodes when creating file copies, and give priority to nodes with lower future loads for writing file copies, thereby achieving load balancing in HDFS.

[0047] 2. Efficient load prediction model: LSTM neural network is used as the load prediction model, and it is trained through node load data at equal time intervals. This model has high accuracy and can more accurately predict the future load of nodes compared to traditional models based on Markov and autoregressive sliding average, thus improving the stability and performance of the system.

[0048] 3. Flexibility and applicability: The function of timing recording node load in this system increases the flexibility of the system by setting a timing task to record the load of each node in the cluster every hour. At the same time, the deep learning network is adopted. This method is suitable for various complex distributed storage scenarios, can better adapt to heterogeneous cluster environments, and improves the applicability of the system.

[0049] 4. Enhanced system stability: By dynamically calculating the future load of nodes, the system can better cope with load fluctuations between nodes, reduce the load of hot nodes, optimize the storage structure, and thus improve the stability of the entire system.

[0050] 5. High adaptability: This system uses the adaptive moment estimation (Adam) as the optimizer, which is highly adaptable. This enables the system to flexibly adjust the model parameters according to the actual situation to achieve the best training effect, increasing the robustness and maintainability of the system.

[0051] In summary, the method and system for dynamically determining the replica location of HDFS based on the node historical status of the present invention can achieve load balancing in a heterogeneous environment by dynamically setting the write location of the file replica in the HDFS cluster. Compared with the existing fixed replica strategy, it is more flexible and enhances system availability. Compared with the existing replica placement strategy based on the current node status, it eliminates the interference of sudden states, making the balance stronger, and at the same time, the prediction accuracy of the neural network is higher, which can better improve the overall performance of the system.

[0052] These and other aspects of the present application will be more concise and understandable in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0054] In the figure:

[0055] Figure 1 The placement flowchart for the replica placement strategy based on rack awareness.

[0056] Figure 2 A flowchart of a method for dynamically determining a replica location based on a node's historical status in HDFS according to an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the LSTM memory unit structure in the method for dynamically determining the replica location of HDFS based on the historical status of the node in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are intended to distinguish two non-identical entities or non-identical parameters with the same name. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, other steps or units inherent to a process, method, device, product or equipment that includes a series of steps or units.

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0063] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0064] Since the replica strategy is not able to differentiate the load of heterogeneous clusters, in order to solve the problem of ignoring node performance when placing file replicas in an HDFS cluster, the present invention proposes a method and system for dynamically determining the replica location of HDFS based on the historical status of the node. By establishing a node load evaluation index and predicting the future load of the node, the placement location of the file replica is dynamically set to achieve load balancing in a cluster heterogeneous environment, thereby improving the overall performance of the system.

[0065] The present invention first defines the node load evaluation index in the cluster, and then regularly records the node load. When a file copy needs to be created, the historical load of each node is used to predict the future node status, and then the node with lower comprehensive load is selected for writing, so as to realize the dynamic determination of the file copy location. Figure 2 shown.

[0066] See also Figure 1 As shown, an embodiment of the present invention provides a method for dynamically determining a replica location of HDFS based on a node historical state, the method comprising the following steps:

[0067] Construct node load evaluation indicators to measure the current load of the node;

[0068] Regularly record the current load of the node to provide data support for load prediction;

[0069] Use LSTM neural network for training and build a node load prediction model to predict the future load of cluster nodes, and determine the location where the copy of the file in the cluster is created based on the predicted value;

[0070] When writing file copies, the future load of the node is dynamically calculated, the load of each node at the next moment is predicted, and the nodes with lower future load are written first.

[0071] In the embodiment of the present invention, the node load evaluation index includes:

[0072] Assume that the CPU main frequency of the node is f, the number of cores is n, and the performance conversion coefficient in the multi-core environment is t, then the CPU performance C of node k is k It can be expressed as:

[0073]

[0074] The CPU load of node k at time t is Assume that the memory capacity of the node is r and the memory rate coefficient is u, then the memory performance of node k is M k It can be expressed as:

[0075] M k =u k r k

[0076] The memory load of the node at time t is

[0077] Assume that the HDFS disk capacity of the node is e and the hard disk rate coefficient is b, then the disk performance of node k is H k It can be expressed as:

[0078] H k =b k e k

[0079] The disk load of the node at time t is From the above three equations, the comprehensive load of the node at time t is:

[0080]

[0081] Among them, α is the CPU impact coefficient, β is the memory impact coefficient, and γ is the disk impact coefficient. The above three coefficients are determined by the design goals and service objects of the system. If the calculation in the system is more frequent, α can be appropriately increased, and if there is a lot of cold data in the system, γ can be appropriately increased.

[0082] In this embodiment, the current load of the node is recorded regularly. When the cluster is running, a scheduled task is set to record the load status of each node in the cluster every hour.

[0083] In this embodiment, when creating a replica, the future load of the node is dynamically calculated. When creating a replica, the system uses LSTM to build a node load prediction model to predict the future load of the cluster node, and determines the location of the replica creation of the file in the cluster based on the predicted value.

[0084] The node load prediction model uses the node load data at equal time intervals and predicts the node load in the future time period based on the LSTM network, including the following steps:

[0085] i) Data preprocessing

[0086] In order to speed up the convergence of the model and improve the accuracy of the model, the data needs to be normalized. The normalization function uses maximum and minimum normalization.

[0087] ii) Model training

[0088] The Long Short-Term Memory (LSTM) network solves the problem of gradient vanishing and gradient exploding in recurrent neural networks when the training sequence is long to a large extent by adding memory unit gates to neurons.

[0089] In this embodiment, the memory cell gates are divided into three categories: input gate it, output gate ot and forget gate ft. Among them, the input gate is responsible for controlling the value of the current network input xt saved to the current memory cell; the output gate is responsible for controlling the value of the current memory cell output to the LSTM network output value ht; the forget gate is used to determine the value of the previous memory cell data retained in the current memory cell. LSTM memory cell structure see Figure 3 shown.

[0090] In this embodiment, the model input layer uses one hundred memory units, the hidden layer uses one hundred memory units, and the output layer uses a single unit. At the same time, in order to speed up the convergence of the model, adaptive moment estimation (Adam) is used as the optimizer. The loss function selects the mean absolute error (MAE). Compared with the mean square error (MSE), although the calculation speed is lower than that of MSE, MAE has better robustness for the processing of outliers. The root mean square error (RMSE) is selected as the evaluation index of the prediction result, which can indicate the deviation between the prediction result and the true value, and is an evaluation index in the prediction model.

[0091] When writing file replicas, priority is given to nodes with lower future load.

[0092] The present invention provides a method for dynamically determining the replica location of HDFS based on the node historical status. By dynamically setting the write location of the file replica in the HDFS cluster, load balancing in a heterogeneous environment can be achieved. Compared with the existing fixed replica strategy, it is more flexible and enhances system availability. Compared with the existing replica placement strategy based on the current node status, it eliminates the interference of sudden states, making the balance stronger. At the same time, the prediction accuracy of the neural network is higher, which can better improve the overall performance of the system.

[0093] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0094] It should be understood that, although described in a certain order, these steps are not necessarily performed in sequence in the above order. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of the steps or stages in other steps or other steps.

[0095] In a second aspect of the embodiments of the present invention, the present invention further provides a system for dynamically determining a replica location based on a HDFS node's historical status, comprising the following modules:

[0096] The node load evaluation module is used to construct node load evaluation indicators and regularly record the current load of the node;

[0097] The load prediction module includes an LSTM neural network, which is used to train the node load prediction model and determine the location of the replica creation of the file in the cluster based on the predicted value;

[0098] The file writing module dynamically calculates the future load of the node, predicts the load of each node at the next moment, and gives priority to nodes with lower future load for writing file copies.

[0099] In this embodiment, when the node load evaluation module records the current load of the node at a regular interval, it records the load status of each node in the cluster every hour by setting a scheduled task.

[0100] In this embodiment, the LSTM neural network of the load prediction module includes the following steps:

[0101] The data preprocessing step is used to normalize the node load data using the maximum and minimum normalization function;

[0102] Model parameter settings, including an input layer of one hundred memory units, a hidden layer of one hundred memory units, and an output layer of a single unit;

[0103] Optimizer and loss function settings, using adaptive moment estimation (Adam) as the optimizer and selecting mean absolute error (MAE) as the loss function;

[0104] Evaluation index setting, root mean square error (RMSE) is selected as the evaluation index of the prediction results.

[0105] In this embodiment, when a file copy needs to be written, the file writing module predicts the future load and preferentially selects a node with a lower future load to write the file copy, so as to achieve load balancing.

[0106] The method and system for dynamically determining the replica location of HDFS based on the node historical status proposed in the present invention have the following advantages over the prior art:

[0107] 1. Load balancing optimization: By building node load evaluation indicators and using LSTM neural networks for load prediction, this system can accurately predict the future load of cluster nodes. This enables the system to dynamically calculate the future load of nodes when creating file copies, and give priority to nodes with lower future loads for writing file copies, thereby achieving load balancing in HDFS.

[0108] 2. Efficient load prediction model: LSTM neural network is used as the load prediction model, and it is trained through node load data at equal time intervals. This model has high accuracy and can more accurately predict the future load of nodes compared to traditional models based on Markov and autoregressive sliding average, thus improving the stability and performance of the system.

[0109] 3. Flexibility and applicability: The function of timing recording node load in this system increases the flexibility of the system by setting a timing task to record the load of each node in the cluster every hour. At the same time, the deep learning network is adopted. This method is suitable for various complex distributed storage scenarios, can better adapt to heterogeneous cluster environments, and improves the applicability of the system.

[0110] 4. Enhanced system stability: By dynamically calculating the future load of nodes, the system can better cope with load fluctuations between nodes, reduce the load of hot nodes, optimize the storage structure, and thus improve the stability of the entire system.

[0111] 5. High adaptability: This system uses the adaptive moment estimation (Adam) as the optimizer, which is highly adaptable. This enables the system to flexibly adjust the model parameters according to the actual situation to achieve the best training effect, increasing the robustness and maintainability of the system.

[0112] In summary, the method and system for dynamically determining the replica location of HDFS based on the node historical status of the present invention can achieve load balancing in a heterogeneous environment by dynamically setting the write location of the file replica in the HDFS cluster. Compared with the existing fixed replica strategy, it is more flexible and enhances system availability. Compared with the existing replica placement strategy based on the current node status, it eliminates the interference of sudden states, making the balance stronger, and at the same time, the prediction accuracy of the neural network is higher, which can better improve the overall performance of the system.

[0113] According to a third aspect of an embodiment of the present invention, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method of any one of the above embodiments is implemented.

[0114] The computer device includes a processor and a memory, and may also include: an input device and an output device. The processor, memory, input device and output device may be connected via a bus or other means, and the input device may receive input digital or character information, and generate signal input related to the migration of the replica location dynamically determined by HDFS based on the historical state of the node. The output device may include a display device such as a display screen.

[0115] As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the method for dynamically determining the replica location based on the HDFS historical status of the node in the embodiment of the present application. The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating device, an application required for at least one function; the storage data area may store data created by the use of the method for dynamically determining the replica location based on the HDFS historical status of the node, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the local module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0116] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data. The processors of multiple computer devices of the computer device of this embodiment execute various functional applications and data processing of the server by running non-volatile software programs, instructions and modules stored in the memory, that is, implementing the steps of the method for dynamically determining the replica location of HDFS based on the historical state of the node in the above method embodiment.

[0117] It should be understood that, without conflicting with each other, all the embodiments, features and advantages described above for the method of dynamically determining the replica location based on the historical status of the node according to the present invention are also applicable to the method and storage medium for dynamically determining the replica location based on the historical status of the node according to the present invention.

[0118] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given of the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire device. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the scope disclosed in the embodiments of the present invention.

[0119] Finally, it should be noted that the computer-readable storage medium (e.g., memory) herein may be a volatile memory or a nonvolatile memory, or may include both a volatile memory and a nonvolatile memory. As an example and not by way of limitation, a nonvolatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. A volatile memory may include a random access memory (RAM), which may act as an external cache memory. As an example and not by way of limitation, RAM may be obtained in a variety of forms, such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to include, but are not limited to, these and other suitable types of memory.

[0120] The various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure herein may be implemented or executed using the following components designed to perform the functions herein: a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and / or any other such configuration.

[0121] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope disclosed in the embodiments of the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0122] It should be understood that, as used herein, the singular form "a" or "an" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the items listed in association. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments.

[0123] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the embodiments of the present invention as above, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A method for dynamically determining replica locations in HDFS based on node history status, characterized in that: The following steps are involved: Construct node load evaluation indicators to measure the current load of the node; Regularly record the current load of the node to provide data support for load prediction; Use LSTM neural network for training and build a node load prediction model to predict the future load of cluster nodes, and determine the location where the copy of the file in the cluster is created based on the predicted value; When writing file copies, the future load of the node is dynamically calculated, the load of each node at the next moment is predicted, and the nodes with lower future load are written first.

2. The method for dynamically determining the replica location of HDFS based on the node historical status according to claim 1 is characterized in that: When constructing the node load evaluation index, assume that the CPU main frequency of the node is f, the number of cores is n, the performance conversion coefficient in the multi-core environment is t, and the CPU performance of node k is C k It is expressed as: The CPU load of node k at time t is 3. The method for dynamically determining the replica location of HDFS based on the node historical status according to claim 2 is characterized in that: When constructing the node load evaluation index, let the node's memory capacity be r, the memory rate coefficient be u, and the memory performance of node k be M k It is expressed as: M k =u k r k The memory load of the node at time t is 4. The method for dynamically determining the replica location of HDFS based on the node historical status according to claim 3 is characterized in that: When constructing the node load evaluation index, let the HDFS disk capacity of the node be e, the hard disk rate coefficient be b, and the disk performance of node k be H k It is expressed as: H k =b k e k The disk load of the node at time t is 5. The method for dynamically determining the replica location of HDFS based on the node historical status according to claim 4 is characterized in that: CPU performance C based on node k k , the memory performance M of node k k and the disk performance H of node k k , the comprehensive load of the node at time t is: α+β+γ=1 Among them, α is the CPU impact coefficient, β is the memory impact coefficient, and γ is the disk impact coefficient.

6. The method for dynamically determining the replica location of HDFS based on the node historical status according to claim 1, characterized in that: When periodically recording the current load of a node, set a scheduled task to record the load of each node in the cluster every hour.

7. The method for dynamically determining the replica location of HDFS based on the node historical status according to claim 6, characterized in that: The node load prediction model uses node load data at equal time intervals and predicts the node load in the future time period based on the LSTM network, including data preprocessing, LSTM network construction and training. The steps are: Data preprocessing: normalize the data using the maximum and minimum normalization functions; LSTM model parameters: The model input layer uses one hundred memory units, the hidden layer uses one hundred memory units, and the output layer uses a single unit; Optimizer and loss function: Use adaptive moment estimation as the optimizer and choose mean absolute error as the loss function; Evaluation index: The root mean square error is selected as the evaluation index of the prediction result to measure the deviation between the prediction result and the true value.

8. The method for dynamically determining the replica location of HDFS based on the node historical status according to claim 7, characterized in that: The LSTM model parameter setting includes an input layer of one hundred memory units, a hidden layer of one hundred memory units, and an output layer of a single unit.

9. A system for dynamically determining replica locations based on HDFS node history status, characterized in that: The system for dynamically determining the replica location of HDFS based on the node historical status is used to execute the method for dynamically determining the replica location of HDFS based on the node historical status according to any one of claims 1 to 7, and the system includes the following modules: The load prediction module includes an LSTM neural network, which is used to train the node load prediction model and determine the location of the replica creation of the file in the cluster based on the predicted value; The node load evaluation module is used to construct node load evaluation indicators and regularly record the current load of the node; The file writing module dynamically calculates the future load of the node, predicts the load of each node at the next moment, and gives priority to nodes with lower future load for writing file copies.

10. The system for dynamically determining the replica location of HDFS based on the node historical status according to claim 9, characterized in that: The LSTM neural network of the load prediction module includes the following steps: The data preprocessing step is used to normalize the node load data using the maximum and minimum normalization function; Model parameter settings, including an input layer of one hundred memory units, a hidden layer of one hundred memory units, and an output layer of a single unit; Optimizer and loss function settings, use adaptive moment estimation as the optimizer, and choose mean absolute error as the loss function; Evaluation index setting, select the root mean square error as the evaluation index of the prediction result.