Distributed storage system parameter adjustment method, device, electronic device and medium
By using neural network models to automatically adjust parameters in the storage system, the problem that traditional manual parameter adjustment cannot adapt to large-scale storage systems is solved, and automatic tuning and cost reduction of storage clusters are achieved.
Patent Information
- Application Number
- CN202010956173.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-11
- Publication Date
- 2025-02-18
- Estimated Expiration
- 2040-09-11
AI Technical Summary
Parameter adjustment in existing storage systems relies on manual operations, cannot adapt to the complexity of large-scale storage systems, and there are problems such as high labor costs and inability to monitor around the clock.
By pre-creating a neural network model for evaluating client delay, obtaining the actual internal state data of the current storage cluster, searching for the target parameters that make the output value of the neural network model the largest, and adjusting the parameters of the storage cluster according to this parameter to achieve automatic Tune.
It realizes automatic tuning of the storage cluster, automatically obtains the optimal parameters without manual parameter adjustment, reduces labor costs and improves the system's energy consumption and performance efficiency.
Smart Images

Figure CN112162966B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of distributed storage technology, and more specifically, to a distributed storage system parameter adjustment method and device, an electronic device, and a computer-readable storage medium. Background Art
[0002] Current storage systems have a large number of adjustable parameters, and the adjustable range of each parameter is very large. Modifying these parameters often has different effects on system performance. Usually the default parameter configuration is provided by the vendor, and the combination of these parameter values is generally not the optimal parameter set. Even adjusting a small part of the parameter values can increase the system's energy consumption and performance efficiency by several times.
[0003] Traditional parameter adjustment is done by system administrators based on their own professional knowledge and experience. As the complexity of storage systems continues to increase, manual parameter adjustment will not be able to adapt to large-scale storage systems. In addition, manual parameter adjustment has the disadvantages of being unable to monitor all day and having high labor costs. Currently, there is an urgent need to study automatic tuning methods for storage systems. Therefore, how to solve the above problems is what technicians in this field need to focus on. Summary of the invention
[0004] The purpose of the present application is to provide a distributed storage system parameter adjustment method, device, electronic device and computer-readable storage medium, which can realize automatic tuning of a storage cluster.
[0005] To achieve the above objectives, the present application provides a distributed storage system parameter adjustment method, comprising:
[0006] Obtain a neural network model that is created in advance based on data obtained by the interaction between the vdbench client and the storage cluster, where the neural network model is used to evaluate the latency;
[0007] Determine the actual internal state data of the current storage cluster; wherein the actual internal state data includes the interface delay data of Ganesha;
[0008] The neural network model is used to search for a target parameter that maximizes the output value of the neural network model according to the actual internal state data, and the parameters of the current storage cluster are adjusted according to the target parameter, wherein the output value is negatively correlated with the client delay.
[0009] Optionally, the interface delay data includes: interface name, number of requests received, number of requests processed, average processing delay, maximum processing delay, minimum processing delay, average request waiting delay, maximum request waiting delay and minimum request waiting delay, any one or a combination of any several of the items.
[0010] Optionally, the process of creating the neural network model includes:
[0011] Constructing a fully connected neural network as an initial neural network model, and initializing neural network parameters in the initial neural network model;
[0012] Obtain the first internal state of the storage cluster at the current moment, select a set of parameters in the parameter space and perform dynamic configuration, perform IO read and write operations on the storage system after the configuration is completed, and obtain the average latency data of the client; the parameter space is the parameter space of the ganesha configuration data generated by setting the minimum value, maximum value and sampling interval corresponding to the ganesha configuration data;
[0013] Query the second internal state of the storage cluster after the IO read and write operation is completed, and generate a set of training data by combining the average latency data, the ganesha configuration data and the first internal state;
[0014] The initial neural network model is trained using multiple sets of the training data to generate a final neural network model.
[0015] Optionally, the Ganesha configuration data includes: any one or a combination of any two of the following: the number of worker threads, the total number of read worker threads, the number of read worker thread groups, the maximum number of cached files, and the cache time.
[0016] Optionally, using the neural network model to search for a target parameter that maximizes the output value of the neural network model according to the actual internal state data includes:
[0017] Traversing the parameter space of Ganesha configuration data, inputting the actual internal state data and each set of parameters into the neural network model, and obtaining corresponding output values;
[0018] A maximum value is selected from all the output values, and a parameter corresponding to the maximum value is obtained as the target parameter.
[0019] Optionally, the using the neural network model to search for a target parameter that maximizes the output value of the neural network model according to the actual internal state data, and adjusting the parameters of the current storage cluster according to the target parameter, further includes:
[0020] The actual internal state data and the target parameters are recorded as the latest training parameters so as to update the neural network model using the latest training parameters.
[0021] To achieve the above objectives, the present application provides a distributed storage system parameter adjustment device, comprising:
[0022] A model acquisition module is used to acquire a neural network model that is pre-created based on data obtained by the interaction between the vdbench client and the storage cluster, and the neural network model is used to evaluate the delay size;
[0023] A state determination module, used to determine the actual internal state data of the current storage cluster; wherein the actual internal state data includes the interface delay data of Ganesha;
[0024] A parameter adjustment module is used to use the neural network model to search for a target parameter that maximizes the output value of the neural network model according to the actual internal state data, and adjust the parameters of the current storage cluster according to the target parameter, wherein the output value is negatively correlated with the client delay.
[0025] Optionally, also include:
[0026] The model updating module is used to record the actual internal state data and the target parameters as the latest training parameters so as to update the neural network model using the latest training parameters.
[0027] To achieve the above objectives, the present application provides an electronic device, including:
[0028] Memory for storing computer programs;
[0029] A processor is used to implement the steps of any of the above-disclosed distributed storage system parameter adjustment methods when executing the computer program.
[0030] To achieve the above objectives, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-disclosed distributed storage system parameter adjustment methods are implemented.
[0031] Through the above scheme, it can be seen that a distributed storage system parameter adjustment method provided by the present application includes: obtaining a neural network model created in advance according to the data obtained by the interaction between the vdbench client and the storage cluster, the neural network model is used to evaluate the delay size; determining the actual internal state data of the current storage cluster; wherein the actual internal state data includes the interface delay data of Ganesha; using the neural network model, searching for the target parameter that maximizes the output value of the neural network model according to the actual internal state data, and adjusting the parameters of the current storage cluster according to the target parameter, wherein the output value is negatively correlated with the client delay. From the above, it can be seen that the present application can pre-create a neural network model for evaluating client delay, and when parameter adjustment is actually required, the actual internal state data of the current storage cluster can be obtained, and the target parameter that can maximize the output value of the neural network model can be searched based on the created neural network model, and the larger the output value, the smaller the client delay, thereby obtaining the target parameter that makes the client delay smaller to perform storage cluster tuning, and the optimal parameter can be automatically obtained without manual parameter adjustment, thereby reducing labor costs and realizing automatic tuning of the storage cluster.
[0032] The present application also discloses a distributed storage system parameter adjustment device, an electronic device and a computer-readable storage medium, which can also achieve the above-mentioned technical effects.
[0033] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0035] Figure 1 A flow chart of a distributed storage system parameter adjustment method disclosed in an embodiment of the present application;
[0036] Figure 2 A flowchart of a neural network model creation process disclosed in an embodiment of the present application;
[0037] Figure 3 A structural diagram of a distributed storage system parameter adjustment device disclosed in an embodiment of the present application;
[0038] Figure 4 A structural diagram of an electronic device disclosed in an embodiment of the present application;
[0039] Figure 5 This is a structural diagram of another electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0041] Traditional parameter adjustment is done by system administrators based on their own professional knowledge and experience. As the complexity of storage systems continues to increase, manual parameter adjustment will not be able to adapt to large-scale storage systems. In addition, manual parameter adjustment has the disadvantages of being unable to monitor all day and having high labor costs. Currently, there is an urgent need to study automatic tuning methods for storage systems. Therefore, how to solve the above problems is what technicians in this field need to focus on.
[0042] Therefore, an embodiment of the present application discloses a distributed storage system parameter adjustment method, which can automatically obtain optimal parameters and realize automatic tuning of the storage cluster.
[0043] See also Figure 1 As shown, a distributed storage system parameter adjustment method disclosed in an embodiment of the present application includes:
[0044] S101: Obtain a neural network model that is pre-created based on data obtained by the interaction between the vdbench client and the storage cluster, where the neural network model is used to evaluate latency;
[0045] In an embodiment of the present application, the vdbench client can be used to interact with the storage cluster in advance, collect the internal status of the storage cluster, the average latency of the client, and the ganesha configuration data, and create a neural network model through offline learning. The neural network model is used to evaluate the latency of the client.
[0046] It should be pointed out that vdbench is an I / O workload generator used to verify data integrity and measure the performance of directly attached and network-connected storage. Due to its advantages of easy installation and good compatibility, it has become a common tool used in testing and benchmarking. Ganesha specifically refers to NFS-ganesha, which is an NFS (Network File System) file server running in user mode.
[0047] Specifically, the above-mentioned Ganesha configuration data may include but is not limited to: the number of working threads, the total number of read working threads, the number of reading working thread groups, the maximum number of cached files, and the cache time.
[0048] S102: Determine actual internal state data of the current storage cluster; wherein the actual internal state data includes interface delay data of Ganesha;
[0049] In this step, the actual internal state data of the storage cluster that currently needs parameter tuning can be obtained. Specifically, the actual internal state data may include the interface delay data of Ganesha, which is a set of statistical indicators, including but not limited to: interface name, number of requests received, number of requests processed, average processing delay, maximum processing delay, minimum processing delay, average request waiting delay, maximum request waiting delay, and minimum request waiting delay.
[0050] S103: Using the neural network model, searching for a target parameter that maximizes the output value of the neural network model according to the actual internal state data, adjusting the parameters of the current storage cluster according to the target parameter, and the output value is negatively correlated with the client latency.
[0051] It should be noted that after obtaining the actual internal state data of the current storage cluster, the actual internal state data and the Ganesha configuration data can be used as input data of the neural network model. Specifically, the parameter space of the Ganesha configuration data can be traversed, and the actual internal state data and each set of parameters can be input into the neural network model to obtain the corresponding output value, and the maximum value is selected from all the output values, and the parameter corresponding to the maximum value is obtained as the target parameter, so that the target parameter can be used to adjust the parameters of the current storage cluster.
[0052] As a preferred implementation, after searching for the target parameters and adjusting the parameters of the current storage cluster according to the target parameters, the embodiment of the present application can further record the actual internal state data and the target parameters as the latest training parameters, so as to subsequently update the neural network model using the latest training parameters to improve the accuracy of the neural network model. For example, after obtaining the target parameters, that is, the optimal configuration parameters, online configuration can be performed according to the target parameters, and after running for a period of time, the feedback of the system and the internal state of the cluster at the current moment are obtained according to the reward function, and the data is saved and recorded for further updating of the neural network model.
[0053] Through the above scheme, it can be seen that a distributed storage system parameter adjustment method provided by the present application includes: obtaining a neural network model created in advance according to the data obtained by the interaction between the vdbench client and the storage cluster, the neural network model is used to evaluate the delay size; determining the actual internal state data of the current storage cluster; wherein the actual internal state data includes the interface delay data of Ganesha; using the neural network model, searching for the target parameter that maximizes the output value of the neural network model according to the actual internal state data, and adjusting the parameters of the current storage cluster according to the target parameter, wherein the output value is negatively correlated with the client delay. From the above, it can be seen that the present application can pre-create a neural network model for evaluating client delay, and when parameter adjustment is actually required, the actual internal state data of the current storage cluster can be obtained, and the target parameter that can maximize the output value of the neural network model can be searched based on the created neural network model, and the larger the output value, the smaller the client delay, thereby obtaining the target parameter that makes the client delay smaller to perform storage cluster tuning, and the optimal parameter can be automatically obtained without manual parameter adjustment, thereby reducing labor costs and realizing automatic tuning of the storage cluster.
[0054] The following is a further introduction to the creation process of the neural network model. Figure 2 As shown, specifically:
[0055] S201: constructing a fully connected neural network as an initial neural network model, and initializing neural network parameters in the initial neural network model;
[0056] The embodiment of the present application can specifically construct a four-layer fully connected neural network as a neural network model, and initialize the neural network parameters. Among them, the input layer is the internal state of the cluster and the ganesha core parameters, the first hidden layer contains M neurons, the activation function is Relu, the second hidden layer contains N neurons, the activation function is Relu, and the output layer contains one neuron. The values of M and N above can be set according to actual conditions in the specific implementation, and are not limited here. For example, M can be set to 100 and N can be set to 10.
[0057] S202: Obtain the first internal state of the storage cluster at the current moment, select a set of parameters in the parameter space and perform dynamic configuration, perform IO read and write operations on the storage system after the configuration is completed, and obtain average latency data of the client;
[0058] In this step, the first internal state s of the storage cluster at the current moment can be obtained. tSpecifically, the interface delay of Ganesha can be used as the internal state of the cluster, including but not limited to the interface name, the number of requests received, the number of requests processed, the average processing delay, the maximum processing delay, the minimum processing delay, the average request waiting delay, the maximum request waiting delay, the minimum request waiting delay, etc., which can be regarded as a 24x8 two-dimensional table. Expand the two-dimensional table row by row to obtain the internal state vector s of the cluster with a length of 192 dimensions. t .
[0059] It should be noted that the vdbench client is in cluster status s t , parameter a t When the average latency of IO read and write operations is as low as possible, the reward function is as follows: t =-latency t There are five core configuration data of Ganesha. By setting the minimum value, maximum value and sampling interval corresponding to the Ganesha configuration data, the parameter space corresponding to the Ganesha configuration data can be generated. For example, a specific parameter space can be shown in Table 1 below:
[0060] Table 1
[0061] Parameter name Parameter meaning [minimum value, maximum value, sampling interval] Nb_Worker Number of worker threads [1,128,10] nb_worker_req Total number of read worker threads [1,128,10] nb_worker_queue Read worker threads are divided into 4 groups [1,100,10] Entries_HWMark Maximum number of files to cache [100,200000,1000] Attr_Expiration_Time Cache time (in seconds) [1,100,10]
[0062] For example, a set of parameter vectors is: t =[10,10,5,100,8].
[0063] It should be noted that during offline learning, a set of parameters needs to be obtained from the parameter space A. The specific method is as follows: Use the ε-greedy method to obtain the configuration parameter a t , where ε is a probability value close to 1, such as 0.99. A set of parameters is randomly selected from the parameter space with a smaller probability of 1-ε, and the internal state of the current cluster and each set of parameters in the parameter space are input into the neural network with a larger probability of ε, and the set of parameters with the maximum output of the neural network is taken.
[0064] Specifically, a set of parameters a can be obtained according to the internal state of the current cluster and the above parameter acquisition method. t And dynamically configure parameters, randomly sample and execute vdbench scripts from multiple application scenarios, perform IO read and write operations on the storage system, and set the running time, which can be selected as 2 minutes. After the IO is completed, the system feedback can be obtained according to the reward function.
[0065] S203: querying the second internal state of the storage cluster after the IO read and write operation is completed, and generating a set of training data by combining the average latency data, the ganesha configuration data and the first internal state;
[0066] In this step, you can query the second internal state s of the storage cluster after the IO read and write operation is completed. t+1 Then, a set of training data is generated by combining the average delay data, the Ganesha configuration data, the first internal state and the second internal state. For example, (s t , a t , r t ,s t+1 ) is saved to the cache list.
[0067] S204: Using multiple groups of the training data to train the initial neural network model to generate a final neural network model.
[0068] It is understandable that multiple sets of training data can be randomly selected from the cache list to train the neural network model to generate the required neural network model. i ,s i The target value y i By the current instant feedback t and target neural network model The evaluation is that According to the input s i ,s i and the target value y i , use the gradient descent algorithm to update the neural network model Q. Every time the Q neural network model is updated C times, it is updated once The above C can be set according to the actual implementation scenario, and this embodiment does not specifically limit it. For example, C can be set to 10, that is, every time the Q neural network model is updated ten times, it is updated once. Right now
[0069] A distributed storage system parameter adjustment device provided in an embodiment of the present application is introduced below. The distributed storage system parameter adjustment device described below and the distributed storage system parameter adjustment method described above can refer to each other.
[0070] See also Figure 3 As shown, a distributed storage system parameter adjustment device provided in an embodiment of the present application includes:
[0071] A model acquisition module 301 is used to acquire a neural network model that is pre-created based on data obtained by the interaction between the vdbench client and the storage cluster, where the neural network model is used to evaluate the latency;
[0072] The state determination module 302 is used to determine the actual internal state data of the current storage cluster; wherein the actual internal state data includes the interface delay data of Ganesha;
[0073] The parameter adjustment module 303 is used to use the neural network model to search for the target parameter that maximizes the output value of the neural network model according to the actual internal state data, and adjust the parameters of the current storage cluster according to the target parameter, and the output value is negatively correlated with the client delay.
[0074] For the specific implementation process of the above modules 301 to 303, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described again here.
[0075] Based on the above embodiments, as a preferred implementation, the distributed storage system parameter adjustment device provided in the embodiment of the present application may further include:
[0076] The model updating module is used to record the actual internal state data and the target parameters as the latest training parameters so as to update the neural network model using the latest training parameters.
[0077] The present application also provides an electronic device, see Figure 4 As shown, an electronic device provided in an embodiment of the present application includes:
[0078] A memory 100, used for storing computer programs;
[0079] The processor 200 can implement the steps provided in the above embodiment when being used to execute the computer program.
[0080] Specifically, the memory 100 includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer-readable instructions, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. In some embodiments, the processor 200 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, which provides computing and control capabilities for electronic devices. When executing the computer program stored in the memory 100, the distributed storage system parameter adjustment method disclosed in any of the aforementioned embodiments can be implemented.
[0081] Based on the above embodiments, as a preferred embodiment, see Figure 5 As shown, the electronic device also includes:
[0082] The input interface 300 is connected to the processor 200 and is used to obtain externally imported computer programs, parameters and instructions, and save them in the memory 100 under the control of the processor 200. The input interface 300 can be connected to an input device to receive parameters or instructions manually input by a user. The input device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the terminal housing, or a keyboard, touchpad or mouse.
[0083] The display unit 400 is connected to the processor 200 and is used to display data processed by the processor 200 and to display a visual user interface. The display unit 400 can be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc.
[0084] The network port 500 is connected to the processor 200 and is used to communicate with external terminal devices. The communication technology used for the communication connection can be a wired communication technology or a wireless communication technology, such as mobile high-definition link technology (MHL), universal serial bus (USB), high-definition multimedia interface (HDMI), wireless fidelity technology (WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, communication technology based on IEEE802.11s, etc.
[0085] Figure 5 Only an electronic device having components 100-500 is shown, and those skilled in the art will appreciate that Figure 5 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0086] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes. The storage medium stores a computer program, and when the computer program is executed by a processor, the distributed storage system parameter adjustment method disclosed in any of the aforementioned embodiments is implemented.
[0087] The present application can pre-create a neural network model for evaluating client latency. When parameter adjustment is actually needed, the actual internal state data of the current storage cluster can be obtained. Based on the created neural network model, the target parameter that can maximize the output value of the neural network model is searched. The larger the output value, the smaller the client latency. In this way, the target parameter that reduces the client latency can be obtained to tune the storage cluster. The optimal parameters can be automatically obtained without manual parameter adjustment, which reduces labor costs and realizes automatic tuning of the storage cluster.
[0088] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.
[0089] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
Claims
1. A distributed storage system parameter adjustment method, characterized in that: include: Obtain a neural network model that is created in advance based on data obtained by the interaction between the vdbench client and the storage cluster, where the neural network model is used to evaluate the latency; The creation process of the neural network model includes: constructing a fully connected neural network as an initial neural network model, and initializing the neural network parameters in the initial neural network model; obtaining the first internal state of the storage cluster at the current moment, and selecting a set of parameters in the parameter space and dynamically configuring them, and performing IO read and write operations on the storage system after the configuration is completed to obtain the average latency data of the client; the parameter space is the parameter space of the ganesha configuration data generated by setting the minimum value, maximum value and sampling interval corresponding to the ganesha configuration data; querying the second internal state of the storage cluster after the IO read and write operation is completed, and generating a set of training data in combination with the average latency data, the ganesha configuration data and the first internal state; using multiple sets of the training data to train the initial neural network model to generate the final neural network model; Determine the actual internal state data of the current storage cluster; wherein the actual internal state data includes the interface delay data of Ganesha; The neural network model is used to search for a target parameter that maximizes the output value of the neural network model according to the actual internal state data, and the parameters of the current storage cluster are adjusted according to the target parameter, wherein the output value is negatively correlated with the client delay.
2. The distributed storage system parameter adjustment method according to claim 1, characterized in that: The interface delay data includes: any one or a combination of any several of the following: interface name, number of requests received, number of requests processed, average processing delay, maximum processing delay, minimum processing delay, average request waiting delay, maximum request waiting delay, and minimum request waiting delay.
3. The distributed storage system parameter adjustment method according to claim 1, characterized in that: The Ganesha configuration data includes any one or a combination of any two of the following: the number of working threads, the total number of reading working threads, the number of reading working thread groups, the maximum number of cached files, and the cache time.
4. The distributed storage system parameter adjustment method according to any one of claims 1 to 3, characterized in that: The step of using the neural network model to search for a target parameter that maximizes the output value of the neural network model according to the actual internal state data includes: Traversing the parameter space of Ganesha configuration data, inputting the actual internal state data and each set of parameters into the neural network model, and obtaining corresponding output values; A maximum value is selected from all the output values, and a parameter corresponding to the maximum value is obtained as the target parameter.
5. The distributed storage system parameter adjustment method according to claim 4, characterized in that: After using the neural network model to search for a target parameter that maximizes the output value of the neural network model according to the actual internal state data, and adjusting the parameters of the current storage cluster according to the target parameter, the method further includes: The actual internal state data and the target parameters are recorded as the latest training parameters so as to update the neural network model using the latest training parameters.
6. A distributed storage system parameter adjustment device, characterized in that: include: A model acquisition module is used to acquire a neural network model that is pre-created based on data obtained by the interaction between the vdbench client and the storage cluster, and the neural network model is used to evaluate the delay size; The creation process of the neural network model includes: constructing a fully connected neural network as an initial neural network model, and initializing the neural network parameters in the initial neural network model; obtaining the first internal state of the storage cluster at the current moment, and selecting a set of parameters in the parameter space and dynamically configuring them, and performing IO read and write operations on the storage system after the configuration is completed to obtain the average latency data of the client; the parameter space is the parameter space of the ganesha configuration data generated by setting the minimum value, maximum value and sampling interval corresponding to the ganesha configuration data; querying the second internal state of the storage cluster after the IO read and write operation is completed, and generating a set of training data in combination with the average latency data, the ganesha configuration data and the first internal state; using multiple sets of the training data to train the initial neural network model to generate the final neural network model; A state determination module, used to determine the actual internal state data of the current storage cluster; wherein the actual internal state data includes the interface delay data of Ganesha; A parameter adjustment module is used to use the neural network model to search for a target parameter that maximizes the output value of the neural network model according to the actual internal state data, and adjust the parameters of the current storage cluster according to the target parameter, wherein the output value is negatively correlated with the client delay.
7. The distributed storage system parameter adjustment device according to claim 6, characterized in that: Also includes: The model updating module is used to record the actual internal state data and the target parameters as the latest training parameters so as to update the neural network model using the latest training parameters.
8. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the distributed storage system parameter adjustment method as claimed in any one of claims 1 to 5 when executing the computer program.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the distributed storage system parameter adjustment method according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Method, device and system for automatically adjusting and optimizing parameters of key value pair storage engine
CN110134697A
Resource allocation method and device, monitor and machine readable storage medium
CN110515724A