Data redundancy system of cpu heterogeneous cluster

Abstract

The invention provides a data redundancy system of a cpu heterogeneous cluster. The data redundancy system comprises data objects, virtual nodes, a copy placement mapping table, data nodes and a data distribution strategy system. The data object is mapped to the virtual nodes through the hash function, the hash function performs modular operation with the total number of the virtual nodes through the object identifier, and the virtual node to which the data object belongs is determined; the virtual nodes are mapped to the data nodes through the generated copy placement mapping table, and the virtual nodes can be copied on the multiple data nodes for multiple times; the data distribution strategy system comprises a distribution decision-making agent and a migration decision-making agent; the distribution decision-making agent and the migration decision-making agent adopt a Transform-XL model based on a DQN algorithm; the distribution decision agent is responsible for selecting data nodes and adding a mapping relation into the copy placement mapping table; and the migration decision agent is used for adding nodes or manually triggering data migration.

Description

Technical Field

[0001] The invention belongs to the technical field of storage optimization, aims at the problem of path tracking and relates to a data redundancy system of a CPU heterogeneous cluster. Background Art

[0002] High-performance computing (HPC) is a technology that uses powerful processor clusters to process massive multidimensional data sets (also known as big data) in parallel and solve complex problems at extremely high speeds. HPC solves some of today's most complex computing problems in real time. Driven by demand, the domestic chip industry is developing rapidly, and ARM64 architecture CPUs represented by Kunpeng and Feiteng have become a high-quality choice for domestic high-performance computing clusters. However, most of the existing computing clusters are X86 CPU clusters. How to expand ARM architecture clusters on this basis requires data distribution between and within heterogeneous clusters, especially considering fairness, scalability, reliability and high performance, and taking into account the low power consumption of RAM architecture and the high performance of X86 architecture. These issues have brought new challenges to existing storage technologies. Traditional data distribution strategies have shown great limitations in the face of these requirements, especially when the heterogeneity of the architecture is not considered, resulting in problems such as uneven data distribution or poor performance.

[0003] Traditional strategies are usually based on feedback systems and require the support of specific hardware; methods based on expert systems and strategy selection require pre-building knowledge bases or strategy rules for specific systems. All three strategies are customized for specific systems, lack flexibility and adaptability, and the tuning effect is unstable and requires manual intervention. Although methods based on search algorithms and machine learning can achieve automated parameter tuning, there is still room for improvement in tuning efficiency, performance, and stability. Summary of the invention

[0004] The present invention proposes an efficient data distribution system based on reinforcement learning, particularly involving a data redundancy mechanism and performance optimization of a domestically produced CPU heterogeneous cluster, overcoming the limitations of traditional data distribution strategies.

[0005] The present invention discloses a data redundancy system for a CPU heterogeneous cluster, wherein the data redundancy system for the CPU heterogeneous cluster comprises a data object, a virtual node, a replica placement mapping table, a data node and a data distribution strategy system;

[0006] The data object is mapped to a virtual node through a hash function, and the hash function performs a modulo operation on the object identifier and the total number of virtual nodes to determine the virtual node to which the data object belongs;

[0007] Virtual nodes are mapped to data nodes through the generated replica placement mapping table. Virtual nodes can be replicated multiple times on multiple data nodes.

[0008] The data distribution strategy system includes a distribution decision agent and a migration decision agent; the distribution decision agent and the migration decision agent adopt a Transformer-XL model based on the DQN algorithm;

[0009] The distribution decision agent is responsible for selecting data nodes and adding mapping relationships to the replica placement mapping table;

[0010] The migration decision agent is used to add nodes or manually trigger data migration.

[0011] Furthermore, the data distribution strategy system includes a memory pool, which is used to store experience samples, neural network parameters and intermediate results;

[0012] The experience sample is used to store state-action-reward samples. The experience sample is a tuple (s t ,a,r t ,s t+1 ), including four types of information: the status of the current data node s t , data distribution / migration action a, reward value r t , and the next state s after executing action a t+1 ;

[0013] The neural network parameters and intermediate results are used to store the neural network parameters in the DQN and the intermediate results updated during the training process.

[0014] Furthermore, the input parameters of the distributed decision agent include the number of replicas r num , neural network Q(s,a), current state s t , exploration parameter ε and experience replay buffer D;

[0015] The distributed decision agent initializes an empty action list a list , define the next state s t+1 and reward r t , initialize counter i=1;

[0016] Choose action a = ChooseAct(ε,i,Q), if a is not in a list , then add a to a list and get new status and rewards t+1 ,r t =UpdateEnv(a,s t ), and the counter is incremented by 1 until a list The length is equal to r num ;

[0017] And the updated sample (st , a, r t , s t+1 ) Added to D, train and update the parameters of the distributed decision-making agent.

[0018] Further, at each step of the interaction between the distributed decision-making agent and the environment, according to the current state of the data node, it determines the action of distributing the replica location for each virtual node, calculates the reward and feeds it back to the distributed decision-making agent;

[0019] When calculating the replica location of the b-th virtual node VN b , the state-action-reward configuration can be described as follows:

[0020] The state at the t-th time step is described as s t , s t used to represent the load state of the data node when considering the replica placement of VN t , s t defined by the list of relative weights of all data nodes:

[0021]

[0022] where n is the number of data nodes in the system, W i t is equal to the total number of virtual nodes in the i-th data node DN i at time step t divided by the capacity of DN i , W i t used to represent the relative load state of the node;

[0023] The action space is represented by A as:

[0024] A = {DN0, DN1…, DN n-1}

[0025] A represents the set of numbers of all data nodes where replicas can be placed, DN0,…DN n-1 represent the 0 to n-1 data nodes; at time step t, the action is represented by a k-tuple:

[0026]

[0027] A t represents the set of data nodes storing k replicas of VN b , and the selected data nodes will not be repeated; represents the data node storing VN b at time step t.

[0028] If n < k, that is, the number of available data nodes is less than the number of replicas, there will be duplicate replicas on the same node;

[0029] If the selected data node reaches the capacity limit, the data node will not be selected and the action needs to be reselected.

[0030] Furthermore, the reward value reflects the current node equilibrium situation:

[0031] r t =-STDt

[0032] r t is the reward value, STDt represents s t The standard deviation of the set.

[0033] Furthermore, when adding a data node, the migration decision agent is used to select a virtual node to migrate to the new node; except for the action configuration, the state-action-reward configuration of the migration decision agent is the same as that of the distribution decision agent:

[0034] For each virtual node, the action space is a set of constants from 0 to k, that is, A = {0, 1, ..., k}, where different values ​​of action a represent migration commands for different copies of each virtual node; when action a is equal to 0, the virtual node will not migrate any copies to the new node; when action a is equal to any value i from 1-k, it means that the i-th copy of the virtual node is migrated to the new node.

[0035] Furthermore, the data distribution strategy system further includes a universal interface;

[0036] The general interface is used for interaction between the data distribution strategy system and the environment, and the general interface includes an indicator collector and an action controller; the main task of the indicator collector is to collect relevant indicators reflecting the status of data nodes, system performance or delay from the environment, and these indicators are sent to the reinforcement learning agent as status information and reward information respectively; the action controller executes the replica distribution or data migration operation recommended by the reinforcement learning agent by updating the RPM table.

[0037] Furthermore, the model training and model testing of the distributed decision agent and the migration decision agent include the following states:

[0038] 1) The initialization state is used to initialize all training parameters and parameters of the agent model, including the initialization of network weights and biases, experience replay pool, optimizer, ε value of greedy strategy and hyperparameters;

[0039] 2) The training state sets the lower limit Emin and upper limit Emax of the cycle. In each training cycle, the tuples in the experience replay buffer D are periodically replayed to generate mini-batches. For each mini-batch of tuples (s t ,a,r t,s t+1 ), the target value is calculated as follows:

[0040] y=r t +γmax a′ Q(s t+1 ,a′;θ)

[0041] Where γ is the discount factor, θ is the weight vector of the Q network, y represents the target value, and r t is the reward value, max a′ Q(s t+1 ,a′;θ) is the next state s t+1 Select the maximum Q value of action a', which represents the maximum expected reward that can be obtained by taking the optimal action a' in the next state.

[0042] Use the mini-batch stochastic gradient descent method to update the parameters θ of the Q network and minimize the following training objective, which is to minimize the expected value of the difference between the target value and the output of the Q network:

[0043]

[0044] E represents expected value;

[0045] 3) When the training cycle epoch reaches E min When it turns to the inspection state, the standard deviation R of the data node state after training is calculated. Only when R is less than or equal to 1 can the training result be judged to be qualified;

[0046] 4) If the training result is qualified, it will enter the test state, otherwise it will return to the training state; the test state uses the parameter stop as the stop counter. In each test cycle, if the test result is qualified, it will increase stop, otherwise it will return to the inspection state;

[0047] 5) When qualified results are achieved N times or more in a row under the test state, the training process is completed and the end state is reached;

[0048] 6) To prevent training from not converging, if the training cycle exceeds E max , it enters the timeout state, and chooses to restart training or end directly by judging the parameter Re specified by the user.

[0049] Furthermore, a segmented training method is used in the model training process of the distribution decision agent and the transfer decision agent:

[0050] First, a larger number of samples are used as the training data set and randomly divided into many small samples; each sample maintains a training state machine, and a small sample is trained starting from the initialization state of the training state machine. The trained model is called the basic model; the state changes from s0 to s1, and then based on state s1, the basic model is tested directly in the test state of the training state machine in the second small sample; if the test result is qualified, it enters the next sample stage, otherwise the second small sample will be used to retrain the basic model in the training state of the training state machine, and so on, until the test of the last sample is completed.

[0051] Furthermore, a model fine-tuning method is adopted in the model training process of the distributed decision-making agent and the migration decision-making agent to update the network parameters based on the old model to reduce the training time of the new model when the number of nodes increases.

[0052] The beneficial effects achieved by the present invention are:

[0053] The data redundancy mechanism and performance optimization of the domestically produced CPU heterogeneous cluster proposed in this paper is a fair, adaptive, redundant, high-performance, and time-space efficient data distribution strategy for high-performance computing clusters. Compared with existing strategies, the strategy proposed in this paper can significantly reduce read latency, and has obvious effects in heterogeneous environments. It fully utilizes the advantages of X86 processors and ARM processors to allocate data partitions to suitable architectures. For example, data with high performance requirements is allocated to x86 nodes, while data with high energy efficiency requirements is allocated to ARM nodes. For energy-sensitive tasks, ARM nodes are used first, while performance-intensive tasks can be allocated to x86 nodes. Data is stored on its main processing nodes to reduce latency and bandwidth consumption caused by data transmission. Effectively improve read throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 For: Data distribution system architecture and interactive environment.

[0055] Figure 2 For: Model training and optimization process

[0056] Figure 3 For: Distributed model based on Transformer-XL in heterogeneous environment. DETAILED DESCRIPTION

[0057] The present invention will be further described below in conjunction with specific embodiments, and the advantages and features of the present invention will become clearer as the description proceeds. However, these embodiments are exemplary only and do not constitute any limitation to the scope of the present invention. It should be understood by those skilled in the art that the details and forms of the technical solution of the present invention may be modified or replaced without departing from the spirit and scope of the present invention, but these modifications and replacements all fall within the scope of protection of the present invention.

[0058] This paper mainly introduces the storage data distribution strategy based on reinforcement learning. First, the system data distribution process is set as an interactive environment, the mapping of objects to data nodes is defined, and an interface is provided for the upper-level operations of objects (including object creation, deletion and migration). The distribution strategy mainly consists of a general interface, a reinforcement learning agent and a memory pool. The agent is responsible for the multi-copy distribution algorithm and data migration strategy based on reinforcement learning, as well as model training optimization and data distribution optimization under heterogeneous mitigation. The segmented training and model fine-tuning methods are used to speed up the training process of reinforcement learning models with large-scale state and action spaces.

[0059] In one embodiment, a heterogeneous high-performance computing (HPC) cluster based on x86 architecture and ARM architecture can take advantage of the advantages of two different architectures to improve computing performance, energy efficiency and adaptability. x86 processors (such as Intel and AMD) generally have high single-thread performance and are suitable for running complex computing-intensive tasks. ARM processors are known for their energy efficiency and are suitable for running large-scale parallel computing tasks. The operating system selects an operating system that supports heterogeneous architectures, such as Linux. It is necessary to ensure that the OS has good support on both x86 and ARM, and use container technology (such as Docker, Singularity) to provide a consistent operating environment. Use a tool chain that supports cross-architecture compilation (such as GCC, LLVM) to compile and optimize code on different architectures. Use hardware acceleration libraries (such as BLAS, OpenBLAS) and vectorization tools (such as AVX, NEON) for performance optimization. Use middleware that supports heterogeneous computing (such as MPI, OpenMP) to implement parallel computing. Use heterogeneous computing frameworks (such as OpenCL, CUDAforARM) to give full play to the performance advantages of different architectures. Use schedulers that support heterogeneous clusters (such as Slurm, PBSPro, Torque), which can identify and manage the resources of nodes with different architectures.

[0060] In one embodiment, data objects in the environment setting are mapped to virtual nodes through a hash function. The hash function determines the virtual node to which the data object belongs by performing a modulo operation on the object identifier (such as the object ID or name) and the total number of virtual nodes. The virtual node is the basic unit of object management, data migration, and fault recovery, which can simplify the process of managing a large number of objects and reduce the burden of data distribution. The number of virtual nodes is set at system startup and is not often changed, otherwise it will cause a large amount of data migration.

[0061] By default, it is assumed that the number of data nodes is N. d , the number of replicas is R, and the recommended number of virtual nodes is N V The calculation process is as follows:

[0062] V = 100 × N d / R

[0063] N V Select the integer power of 2 that is closest to V. For example, if the number of replicas is 4 and the number of data nodes is 120, 240, and 360, V is 3000, 6000, and 9000 respectively, then the number of virtual nodes is 2048, 4096, and 8192 respectively.

[0064] The mapping of virtual nodes to data nodes is defined by the generated Replica Placement Mapping (RPM). A virtual node can be replicated multiple times on multiple data nodes, and the specific number of replicas (i.e., replication factor) is determined by the upper-layer application. Figure 1 , the replication factor is 2, and the virtual node VN1 selects DN0 and DN1 as the data nodes for storing replicas, where DN0 is the primary replica node.

[0065] When creating each virtual node, the distributed decision agent is responsible for selecting the data node and adding the mapping relationship to the RPM table. The read operation of each object is first located to the corresponding virtual node through the hash function, and then the corresponding data node is found through the RPM table. In addition, depending on whether the configuration of the data nodes is consistent, the environment can be divided into homogeneous environment and heterogeneous environment. In a homogeneous environment, the CPU is of the same architecture (x86 or ARM), and the data nodes are configured the same except for capacity, while the heterogeneous environment allows the two architectures to coexist and use multiple different configurations at the same time.

[0066] The data distribution strategy system includes a distribution decision agent, a migration decision agent, a memory pool and a general interface.

[0067] The data distribution strategy system interacts with the environment through a common interface, which includes a metric collector and an action controller. The main task of the metric collector is to collect relevant metrics from the environment that reflect the status of data nodes, system performance, or latency. Specific metrics vary between homogeneous and heterogeneous environments, and these metrics are sent to the reinforcement learning agent as status information and reward information, respectively. The action controller executes the replica distribution or data migration operations recommended by the reinforcement learning agent by updating the RPM table.

[0068] In one embodiment, the reinforcement learning agent can be regarded as the core or "brain" of the data distribution strategy system, which receives reward and state information in the environment and updates the strategy to guide how to distribute replicas (action information) to obtain higher rewards. The agent consists of a distribution decision agent and a migration decision agent, which are responsible for data distribution and migration decisions, respectively. Both use the standard DQN algorithm, but the defined elements are different. In order to speed up model training, the agent is able to generate experience in parallel and store it in an experience replay buffer. When the buffer reaches the training batch size, experience replay is performed for model training.

[0069] The agent will consider various factors that affect performance, take the sequence into account as the model input, and use the Transformer model, which combines the advantages of sequence modeling and the Transformer self-attention mechanism.

[0070] In one embodiment, the memory pool is a key component of DQN (DeepQ-Network) and is used to store the experience of the agent interacting with the environment. It helps to break the temporal correlation between data, improve sample efficiency and training stability. In the implementation of this article, the memory pool is used to store two main contents:

[0071] Experience samples: used to store state-action-reward samples. Each experience sample is a tuple (s t ,a,r t ,s t+1 ), including four types of information: the status of the current data node s t , data distribution / migration action a, reward value r t , and the next state s after executing action a t+1 .

[0072] Neural network parameters and intermediate results: used to store the neural network parameters in DQN and the intermediate results updated during the training process.

[0073] DeepQ-Network (DQN) was proposed by Google DeepMind team in 2015. It is a reinforcement learning algorithm based on Q-learning. It solves the reinforcement learning problem in large-scale state space by using deep neural network to approximate Q function. DQN introduces mechanisms such as experience replay and target network, which significantly improves the stability and performance of the algorithm. Its birth marks a major breakthrough in deep reinforcement learning.

[0074] Transformer-XL is an improvement on the classic Transformer model. It mainly solves the context length limitation and computational efficiency problems of the classic Transformer when processing long sequences through a fragment-based recurrence mechanism and relative position encoding. Through these improvements, Transformer-XL can effectively capture long-range dependencies and process longer sequences while maintaining high computational efficiency. It is widely used in natural language processing tasks, especially in scenarios that need to process long texts.

[0075] In one embodiment, the distributed decision agent determines the action of distributing the replica location for each virtual node based on the current state of the data node at each step of its interaction with the environment. Subsequently, the reward is calculated and fed back to the distributed decision agent. Let n be the number of data nodes in the system and k be the replication factor. When calculating the bth virtual node (VN b ), the state-action-reward configuration can be described as follows:

[0076] The state at the tth time step is described as s t ,s t Consider VN b The load status of the data node when the replica is placed. t The relative weight list of all data nodes is defined as:

[0077]

[0078] Where W i t Equal to the number of data nodes DN at time step t. i The total number of virtual nodes in the DN i Capacity, W i t Indicates the relative load status of the node.

[0079] The action space is represented by A

[0080] A={DN0,DN1…,DN n-1}

[0081] A is the set of numbers of all data nodes where data can be placed. At time step t, the action is represented by a k-tuple:

[0082]

[0083] A t represents the set of data nodes storing k copies of VN b . By default, the selected data nodes are non-repetitive to improve reliability. However, if n < k, that is, the number of available data nodes is less than the number of copies, there will be duplicate copies on the same node. In addition, if the selected data nodes reach the capacity limit, the data nodes will not be selected and the action needs to be reselected.

[0084] The reward value is a scalar, denoted as r t . Assume that STDt represents the standard deviation of the s t set, which directly reflects the balance of the current node. r t is equal to the negative of the standard deviation of s t , that is:

[0085] r t = -STDt

[0086] s t The larger the standard deviation of s t , the more unbalanced the data nodes are, and the smaller r t . On the contrary, the larger r

[0087] , the more it can encourage the agent to distribute virtual nodes in the direction of a smaller standard deviation.

[0088] The input of the distribution decision agent is the state information of the data nodes, and the output is the set of data nodes for k storage copies, which will be added to the RPM table. The value range of each cell is 0, 1, 2, and the size is |D||V|, where D is the number of data nodes and V is the number of virtual nodes. One column represents a data node, and one row represents a virtual node. A virtual node can be replicated k times. If the value of the RPMdv cell is 1 or 2, one copy of the virtual node corresponding to that row is replicated to the data node corresponding to that column, otherwise it is 0. Among them, 1 represents the primary copy node of the virtual node, and 2 represents other copies.

[0089] The multi-copy distribution algorithm describes the process of the distribution decision agent selecting actions and updating the model. A list a list is used to store all actions (the data node numbers for replica distribution). When a listWhen the length of reaches the replication factor, the algorithm stops. By default, the number of data nodes is greater than the replication factor, so a list The same elements cannot exist in .

[0090] The algorithm will give priority to actions that maximize the output value Q-value (Q value). The Q value directly reflects the quality of the action. The action with a larger Q value will receive a greater reward. list , the action with the second largest Q-value is chosen instead, and so on. The ChooseAct() function implements action selection, which applies the ∈-greedy method, that is, randomly selecting an action with probability ∈, otherwise selecting an action from the largest to the smallest value in Q-value. The parameter ∈ increases from 0 to 1 during training and remains at 1 during the test phase and the final model.

[0091] Input parameters include the number of replicas r num , neural network Q(s,a), current state s t , exploration parameter ε and experience replay buffer D.

[0092]

[0093]

[0094] FunctionChooseAct(ε,i,Q):

[0095] 1. Generate a random number p∈[0,1].

[0096] 2.ifp<εthen

[0097] 3. Select the action a corresponding to the i-th largest Q value.

[0098] 4.else

[0099] 5. Randomly select action a from the action space.

[0100] 6.return a.

[0101] FunctionUpdateEnv(a,s t ):

[0102] 1. Store the replica in the data node numbered by a.

[0103] 2. Update node status t+1 And return the new state s t+1 ;

[0104] 3. Calculate and update the RPM status.

[0105] 4. The reward r is the negative of the deviation of the node state.

[0106] 5. returns t+1 ,r t .

[0107] If the number of data nodes is less than the replication factor, each action selection will not consider whether there is overlap, and will be judged separately. After selecting each action a, call the function UpdateEnv() to use a list Execute replica distribution and update RPM table. The state of data node is in the current state s t Based on the update, and as the new state s t Returns the standard deviation calculated based on the new state as the reward r t Returns. A tuple (s t ,a,r t ,s t+1 ) will be stored in the experience replay buffer D, which can be used to train the agent and update the neural network parameters during the training process.

[0108] In one embodiment, data migration refers to adjusting the replica location to achieve fair redistribution when the storage scale changes (increase or decrease of data nodes). In the storage system, the change of data nodes can be divided into two cases: adding nodes and deleting nodes. Deleting nodes still uses distributed decision-making agents to make adjustments. Specifically, assuming that the node DN m Deleted, DN m The virtual node replicas in DN are remapped to other nodes through distributed decision agents, but there are two limitations: First, DN m It cannot be selected as an action; secondly, when n>k, replica selection conflicts are not allowed, that is, data nodes with the same virtual node replica cannot be selected.

[0109] In one embodiment, the migration decision agent is used to add nodes or manually trigger data migration. When adding a data node, it is necessary to consider which virtual nodes' replicas should be migrated to the new node. In the state-action-reward configuration of the migration decision agent, except for the action configuration, the rest is the same as the distribution decision agent:

[0110] In one embodiment, for each virtual node, the action space is a set of constants from 0 to k (replication factor), i.e., A = {0, 1, ..., k}, where different values ​​of action a represent migration commands for different replicas of each virtual node. If action a is equal to 0, the virtual node will not migrate any replicas to the new node. If action a is equal to any value i from 1 to k, it means that the i-th replica of the virtual node is migrated to the new node, and the replica location can be found from the RPM. Taking a three-replica system as an example, the action space is A = {0, 1, 2, 3}, and the virtual node VN i A copy of the k ,DN j ,DN l ), when action a is equal to 0, VN i No replica migration is performed; when action a is 1, k The replicas in will be migrated to the new node; when action a is 2 or 3, in DN j Or the DN1 replica will be migrated. Through the above action definition, it can be ensured that some replicas of the virtual node are migrated to the new node after joining the system. The standard deviation is still used as a reward to ensure the balance of data nodes after migration and meet the adaptability requirements. The algorithm steps of the migration decision agent are similar to those of the distribution decision agent and will not be introduced in detail. In addition, the distribution decision agent needs to be retrained after the data node changes so that the subsequent data can be distributed.

[0111] like Figure 2 As shown in the figure, the model training and optimization process can be described by a finite state machine, which mainly includes the model training and model testing stages. The finite state machine mainly includes six states:

[0112] 1) The initialization state is used to initialize all training parameters and parameters of the agent model, including the initialization of the network weights and biases, the experience replay pool, the optimizer, the ε value of the ε-greedy strategy, and hyperparameters such as the discount factor and batch size.

[0113] 2) The training state is after initialization. Unlike the traditional fixed training cycle, the finite state machine sets the lower limit Emin and upper limit Emax of the cycle. In each training cycle, the training process is the classic DQN training algorithm. First, the tuples in the experience replay buffer D are periodically replayed to generate mini-batches. For each mini-batch of tuples (s t ,a,r t ,s t+1 ), the target value is calculated as follows:

[0114] y=r t +γmax a′ Q(s t+1 ,a′;θ)

[0115] Where γ is the discount factor and θ is the weight vector of the Q network. a′ Q(s t+1 ,a′;θ) is the next state s t+1 Select the maximum Q value of action a', which represents the maximum expected reward that can be obtained by taking the optimal action a' in the next state. Unlike the target value of the traditional DQN, it lacks the final state because there is no final state in the problem of this article. Next, use the mini-batch stochastic gradient descent method to update the parameters θ of the Q network and minimize the following training objective, that is, minimize the expected value of the difference between the target value and the output of the Q network:

[0116]

[0117] E stands for expected value.

[0118] 3) When the training cycle epoch reaches E min When the training result is checked, the state is changed to the inspection state. The standard deviation R of the data node state after training is calculated. Only when R is less than or equal to 1 can the training result be judged to be qualified.

[0119] 4) If the training result is qualified, it will enter the test state, otherwise it will return to the training state. The test state uses the parameter stop as the stop counter. In each test cycle, if the test result is qualified, it will increase stop, otherwise it will return to the check state.

[0120] 5) The training process is completed and the end state is reached only when qualified results are obtained N times (a constant, the default is 20) or more times in a row in the test state.

[0121] 6) To prevent training from not converging, if the training cycle exceeds E max , it enters the timeout state, and the system will report an error. By judging the parameter Re specified by the user, choose to restart the training (return to the initialization state and initialize new parameters) or end it directly. In the actual training process, as the number of objects and data nodes increases, in order to meet the above high requirements (R<1 for N consecutive times), the training time increases accordingly. To speed up training, two optimization strategies are adopted:

[0122] In modern storage systems, the number of objects is very large. Although they are grouped by hashing, the number of virtual nodes is still large. Since the number of steps in a training epoch is equal to the number of virtual nodes, if a small sample training set is selected in actual training, the training results are not ideal; if a large sample is selected, the training time will be very long (several hours).

[0123] In one embodiment, a segmented training method is adopted. First, a relatively large number of samples n are used as the training dataset and randomly divided into many small samples m. Assume that n = k × m + b (b < m), and the large sample can be divided into k + 1 small samples in total. By default, k is set to 10, and m and b can be calculated based on the large sample number n. Each sample maintains a training state machine. Starting from the initialization state of the training state machine, a small sample of size m is trained, and the trained model is called the basic model. The state changes from s0 to s1, and then based on state s1, the basic model is tested in the test state of the training state machine in the second small sample. If the test result is qualified, the next sample stage is entered; otherwise, the basic model will be retrained in the training state of the training state machine using the second small sample. And so on until the testing of the last sample is completed.

[0124] In one embodiment, the increase in data nodes makes training more difficult in two aspects. More data nodes result in longer training time; as the number of data nodes increases, the dimensions of the state, action space, and neural network increase, making the original model unavailable. In the embodiment, a model fine-tuning method is adopted to update the neural network structure based on the old model to reduce the training time of the new model when the number of nodes increases. Specifically, when the dimension of the neural network increases, the parameters of the old model will be used as the initial parameters of the low dimension of the new model, and the new dimensions will be set to zero or randomly initialized. Taking the neural network as an example, only the dimensions of W1, W n and B n are related to the number n of data nodes. Therefore, after adding nodes, all parameters except them are initialized using the parameters of the old model. Assume that the number of data nodes increases to n'. This causes changes in the dimensions of these three parameters: W1 (from [n, h1] to [n', h1]), E n (from [b k , n] to [h k , n']) and B n (from [n] to [n']). In this case, the new variables W1', W n ', B n ' are initialized as follows

[0125] W1' = [W10]

[0126]

[0127]

[0128] where R() represents random initialization. The first n dimensions of the new variables are initialized using the parameters W1, W n and B n of the old model. W' n and B'n The initialization of the new dimensions of will be random, which ensures that the symmetry between the new dimensions is not broken. Initializing the new dimensions of W1′ to zero ensures that the newly added variables do not affect the output of the fully connected layer. If only some of the weights are reset to zero, subsequent training will not be affected.

[0129] In one embodiment, data nodes in a heterogeneous storage environment differ in capacity, performance, network, etc., so the agent model needs to take these factors into account to ensure the fairness of distribution and the high performance of the system. Four indicators of data nodes are used to measure node status, namely capacity, CPU utilization, disk I / O, and network resources.

[0130] 1) Capacity: still expressed by relative weight, reflecting the spatial status of the node;

[0131] 2) CPU utilization: It indicates the ratio of the time the CPU spends executing a process to the total CPU running time. The lower the CPU utilization, the smaller the CPU load and the stronger the CPU processing capability.

[0132] 3) Disk I / O reflects the disk read and write capabilities and is an important indicator for data distribution considerations;

[0133] 4) Network utilization is the percentage of bandwidth used by data nodes to run the system as a percentage of the total bandwidth. All metrics are obtained from the storage system by the metric collector. For example, to measure network utilization, it collects rxkb / s and txkb / s, which are the amount of network data received and transmitted from the data node (in kilobytes), and divides these values ​​by the overall network bandwidth to obtain network utilization. Using the tuple τ i =(Net,IO,CPU,Weight) i To indicate the status of data node DNi.

[0134] In a heterogeneous environment, states and rewards are defined as follows:

[0135] State: At time step t, s t It can be defined as a sequence of tuples of all data nodes: in

[0136] Reward: The reward is defined as the inverse of the average read (GET request) response time after distribution.

[0137] Distributed model: In the intelligent agents in heterogeneous environments, the Transfoemer-XL model based on relative position encoding and memory mechanism is used. Because the four indicators of node status under different loads have different impacts on performance, the attention mechanism can pay more attention to the indicators that have a greater impact on performance and give the corresponding performance a greater weight. Figure 3 shown.

[0138] The distribution model consists of an encoder-decoder structure based on stacked Transformer-XL units. The input to the model is a previously defined sequence of states, which are stored as adjustable embedding vectors. The main reason for using a sequence model is that the number of data nodes is uncertain and may change with expansion or contraction. The Transformer-XL model can change the length of the input sequence without modifying its internal structure. Transformer-XL can handle any number of data nodes, and the hidden layer state of each data node can be obtained during the calculation process. The decoder is a Transformer-XL model with the same number of decoding steps as the input sequence. At each step, the decoder outputs the Q value of each action, which is the same as the MLP in the isomorphism. The attention mechanism enhances the important parts of the input data and fades out the rest. It calculates the contrast score between the previous decoder hidden state and each encoder hidden state and stores the score as a contrast vector. The encoder hidden state and their respective contrast scores are multiplied to form a context vector. In practical applications, there may be more factors that affect system performance, including object access patterns and load types.

[0139] The above are only specific steps of the present invention and do not constitute any limitation to the protection scope of the present invention; any technical solutions formed by equivalent transformation or equivalent replacement shall fall within the protection scope of the present invention; some parts not elaborated in detail in the present invention belong to the known technologies of those skilled in the art.

Claims

1. A data redundancy system for a CPU heterogeneous cluster, characterized in that: The data redundancy system of the CPU heterogeneous cluster includes data objects, virtual nodes, replica placement mapping tables, data nodes, and a data distribution strategy system; The data objects are mapped to virtual nodes through a hash function. The hash function performs a modulo operation with the object identifier and the total number of virtual nodes to determine the virtual node to which the data object belongs; The virtual nodes are mapped to data nodes through the generated replica placement mapping table. The virtual nodes can be replicated multiple times on multiple data nodes; The data distribution strategy system includes a distribution decision agent and a migration decision agent; the distribution decision agent and the migration decision agent adopt a Transformer-XL model based on the DQN algorithm; The distribution decision agent is responsible for selecting data nodes and adding the mapping relationship to the replica placement mapping table; The migration decision agent is used to add nodes or manually trigger data migration.

2. The data redundancy system of the CPU heterogeneous cluster according to claim 1, characterized in that: The data distribution strategy system includes a memory pool, which is used to store experience samples, neural network parameters, and intermediate results; The experience sample is used to store state-action-reward samples. The experience sample is a tuple (s t ,a,r t ,s t+1 ), including four types of information: the status of the current data node s t , data distribution / migration action a, reward value r t , and the next state s after executing action a t+1 ; The neural network parameters and intermediate results are used to store the neural network parameters in DQN and the intermediate results updated during the training process.

3. The data redundancy system of the CPU heterogeneous cluster according to claim 2, characterized in that: The input parameters of the distributed decision agent include the number of replicas r num , neural network Q(s,a), current state s t , exploration parameter ε and experience replay buffer D; The distributed decision agent initializes an empty action list a list , define the next state s t+1 and reward r t , initialize counter i=1; Choose action a = ChooseAct(ε,i,Q), if a is not in a list , then add a to a list and get new status and rewards t+1 ,r t =UpdateEnv(a,s t ), and the counter is incremented by 1 until a list The length is equal to r num ; And the updated sample (s t ,a,r t ,s t+1 ) is added to D to train and update the parameters of the distributed decision-making agent.

4. The data redundancy system of the CPU heterogeneous cluster according to claim 3 is characterized in that: At each step of the interaction between the distribution decision agent and the environment, it will determine the action of distributing the replica location for each virtual node according to the current state of the data node, calculate the reward, and feedback it to the distribution decision agent; When calculating the bth virtual node VN b When the replica location is , the state-action-reward configuration can be described as follows: The state at the tth time step is described as s t ,s t Used to indicate that VN is considered b The load status of the data node when the replica is placed, s t Defined by a list of relative weights of all data nodes: Where n is the number of data nodes in the system, Equal to the number of data nodes DN at time step t. i The total number of virtual nodes in the DN i The capacity, Used to indicate the relative load status of a node; The action space is represented by A as: A={DN0,DN1…,DN n-1 } A represents the set of numbers of all nodes where data can be placed, DN0,…DN n-1 represents 0 to n-1 data nodes; at time step t, the action is represented by a k-tuple: A t Indicates storage VN b The set of k replicas of data nodes, the selected data nodes will not be repeated; Represents the storage of VN at time step t b The data nodes of the replica. If n < k, that is, the number of available data nodes is less than the number of replicas, there will be duplicate replicas on the same node; If the selected data node reaches the capacity limit, the data node will not be selected, and an action needs to be reselected.

5. The data redundancy system of the CPU heterogeneous cluster according to claim 4, characterized in that: The reward value reflects the balance of the current node: r t =-STDt r t is the reward value, STDt represents s t The standard deviation of the set.

6. The data redundancy system of the CPU heterogeneous cluster according to claim 2, characterized in that: When adding a data node, the migration decision agent is used to select the virtual node to migrate to the new node; in the state-action-reward configuration of the migration decision agent, except for the action configuration, the rest is the same as that of the distribution decision agent: For each virtual node, the action space is a constant set from 0 to k, that is, A = {0, 1,..., k}, where different values of the action a represent migration commands for different replicas of each virtual node; when the action a is equal to 0, this virtual node will not migrate any replicas to the new node; when the action a is equal to any value i from 1 - k, it means that the i-th replica of this virtual node migrates to the new node.

7. The data redundancy system of the CPU heterogeneous cluster according to claim 2, characterized in that: The data distribution strategy system also includes a general interface; The general interface is used for interaction between the data distribution strategy system and the environment. The general interface includes a metric collector and an action controller; the main task of the metric collector is to collect relevant metrics reflecting the status of data nodes, system performance, or latency from the environment, and these metrics are sent to the reinforcement learning agent as state information and reward information respectively; the action controller executes the replica distribution or data migration operation recommended by the reinforcement learning agent by updating the RPM table.

8. The data redundancy system of the CPU heterogeneous cluster according to claim 1, characterized in that: The model training and model testing of the distribution decision agent and the migration decision agent include the following states: 1) The initialization state is used to initialize all training parameters and parameters of the agent model, including the initialization of network weights and biases, experience replay pool, optimizer, ε value of greedy strategy and hyperparameters; 2) The training state sets the lower limit Emin and upper limit Emax of the cycle. In each training cycle, the tuples in the experience replay buffer D are periodically replayed to generate mini-batches. For each mini-batch of tuples (s t ,a,r t ,s t+1 ), the target value is calculated as follows: y=r t +γmax a′ Q(s t+1 ,a′;θ) Where γ is the discount factor, θ is the weight vector of the Q network, y represents the target value, and r t is the reward value, max a′ Q(s t+1 ,a′;θ) is the next state s t+1 Select the maximum Q value of action a', which represents the maximum expected reward that can be obtained by taking the optimal action a' in the next state. Use the mini-batch stochastic gradient descent method to update the parameters θ of the Q network and minimize the following training objective, which is to minimize the expected value of the difference between the target value and the output of the Q network: Indicates expected value; 3) When the training cycle epoch reaches E min When it turns to the inspection state, the standard deviation R of the data node state after training is calculated. Only when R is less than or equal to 1 can the training result be judged to be qualified; 4) If the training result is qualified, it will enter the test state, otherwise it will return to the training state; The test state uses the parameter stop as the stop counter. In each test cycle, if the test result is qualified, stop is increased, otherwise it returns to the check state; 5) When qualified results are achieved N times or more in a row under the test state, the training process is completed and the end state is reached; 6) To prevent training from not converging, if the training cycle exceeds E max , it enters the timeout state, and chooses to restart training or end directly by judging the parameter Re specified by the user.

9. The data redundancy system of the CPU heterogeneous cluster according to claim 8, characterized in that: The distributed decision agent and the transfer decision agent model training process adopts a segmented training method: First, a larger number of samples are used as the training data set and randomly divided into many small samples; each sample maintains a training state machine, and a small sample is trained starting from the initialization state of the training state machine. The trained model is called the basic model; the state changes from s0 to s1, and then based on state s1, the basic model is tested directly in the test state of the training state machine in the second small sample; if the test result is qualified, it enters the next sample stage, otherwise the second small sample will be used to retrain the basic model in the training state of the training state machine, and so on, until the test of the last sample is completed.

10. The data redundancy system of a CPU heterogeneous cluster according to claim 8, characterized in that: The model fine-tuning method is adopted in the model training process of the distributed decision-making agent and the migration decision-making agent to update the network parameters based on the old model to reduce the training time of the new model when the number of nodes increases.

Images