Parameter configuration method and device of storage system, equipment and storage medium

Abstract

The embodiment of the invention provides a parameter configuration method for a storage system, and the method comprises the steps: obtaining a message request, carrying out the analysis of the message request, obtaining a request type and parameter information corresponding to the message request, carrying out the prediction analysis of the message request through a pre-trained input / output model, and obtaining a prediction result, and formulating an optimization strategy for the distributed file storage system based on a prediction result, and performing parameter configuration on the distributed file storage system according to the optimization strategy. According to the embodiment of the invention, different optimization strategies are formulated and applied for the distributed file storage system in a targeted manner; and the distributed file storage system is ensured to provide services with the optimal performance in various scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a parameter configuration method, device, equipment and storage medium of a storage system. Background Art

[0002] In a distributed file storage system, performance evaluation mainly relies on three key indicators: IOPS (Input / Output Per Second), OPS (Operations Per Second), and bandwidth (data transfer rate). IOPS and OPS are usually related to I / O (Input / Output) operations of small blocks of data. Such operations put less pressure on the network but have higher requirements on the back-end processing speed. In contrast, bandwidth is related to I / O operations of large blocks of data. The bottleneck of such operations often lies in the throughput capacity of the network, which requires the network to have the ability to quickly receive a large number of data packets. The diversity of storage scenarios includes small file small IO reading and writing, large file large IO reading and writing, large file small IO reading and writing, and mixed scenarios.

[0003] Therefore, different tuning methods need to be applied to different scenarios, but current technology makes it difficult to develop a universal tuning strategy to meet the needs of all storage scenarios. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a storage system parameter configuration method, apparatus, device and storage medium, and to formulate and apply different tuning strategies for distributed file storage systems in different scenarios, so as to enable the distributed file storage system to provide services with optimal performance. The specific technical solutions are as follows:

[0005] In a first aspect of the present invention, a method for configuring parameters of a storage system is provided, the method comprising:

[0006] Get message request;

[0007] Parsing the message request to obtain the request type and parameter information corresponding to the message request;

[0008] Performing a prediction analysis on the message request using a pre-trained input-output model to obtain a prediction result, wherein the input-output model is used to predict a reading mode of the message request;

[0009] Formulate an optimization strategy for the distributed file storage system based on the prediction result;

[0010] Parameters of the distributed file storage system are configured according to the optimization strategy.

[0011] Optionally, parsing the message request includes:

[0012] Parsing the message request according to a preset message protocol to obtain header information corresponding to the message request, wherein the message protocol includes a message format and fields;

[0013] The header information is identified to obtain the request type and parameter information of the message request, where the request type includes a read / write request and a metadata request.

[0014] Optionally, the method further includes:

[0015] In the case where the request type of the message request is the read-write request, extracting a first key parameter from the read-write request, the first key parameter including a file identifier, a read-write offset position, and a read-write length;

[0016] In a case where the request type of the message request is the metadata request, a second key parameter is extracted from the metadata request, where the second key parameter includes a file identifier and file attribute information.

[0017] Optionally, the input-output model is generated in the following manner:

[0018] Acquire an initial prediction model and historical read data of the distributed file storage system, and divide the historical read data into a training set and a verification set, wherein the training set includes a number of original training data, and the verification set includes a number of original verification data;

[0019] Parsing the original training data, and extracting first parameter information corresponding to each original training data from the original training data;

[0020] Statistically analyzing the first parameter information to identify a first reading mode corresponding to each of the original training data;

[0021] Training the initial prediction model according to the original training data and the first reading mode to obtain a model to be verified;

[0022] Parsing the original verification data, and extracting second parameter information corresponding to each original verification data from the original verification data;

[0023] Statistically analyzing the second parameter information to identify a second reading mode corresponding to each of the original verification data;

[0024] Inputting the original verification data into the model to be verified for prediction, and obtaining a third reading mode output by the model to be verified;

[0025] The third reading pattern is compared with the second reading pattern, and when the third reading pattern is completely identical to the second reading pattern, it is determined that the model to be verified is an input-output model.

[0026] Optionally, when the request type of the message request is the metadata request, the method further includes:

[0027] Obtaining an input and output model in the distributed file storage system;

[0028] The attribute information of the input-output model is updated according to the second key parameter corresponding to the metadata request.

[0029] Optionally, when the request type of the message request is the read-write request, the predictive analysis of the message request using a pre-trained input-output model includes:

[0030] The read / write request is input into the input / output model for prediction to obtain a read mode corresponding to the read / write request.

[0031] Optionally, determining an optimization strategy for the distributed file storage system based on the prediction result includes:

[0032] According to the reading mode predicted by the input-output model, an optimization strategy for the distributed file storage system is formulated, and the optimization strategy at least includes adjusting the cache distribution and network threads of the distributed file storage system.

[0033] In a second aspect of the present invention, a storage system parameter configuration device is provided, which is applied to a distributed file storage system and includes:

[0034] A first acquisition module, used to acquire a message request;

[0035] A first parsing module, used to parse the message request to obtain a request type and parameter information corresponding to the message request;

[0036] A first prediction module, used to perform prediction analysis on the message request using a pre-trained input-output model to obtain a prediction result;

[0037] A strategy formulation module, used to formulate an optimization strategy for the distributed file storage system based on the prediction result;

[0038] A parameter configuration module is used to configure parameters of the distributed file storage system according to the optimization strategy.

[0039] In a third aspect of the implementation of the present invention, there is also provided a communication device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor;

[0040] The processor is used to read the program in the memory to implement any of the above-mentioned parameter configuration methods of the storage system.

[0041] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-mentioned storage system parameter configuration methods.

[0042] The storage system parameter configuration method provided in the embodiment of the present invention obtains a message request, parses the message request, obtains the request type and parameter information corresponding to the message request, uses a pre-trained input-output model to predict and analyze the message request, obtains a prediction result, formulates an optimization strategy for the distributed file storage system based on the prediction result, and configures the parameters of the distributed file storage system according to the optimization strategy. The embodiment of the present invention ensures that the distributed file storage system provides services with optimal performance in various scenarios by formulating and applying different tuning strategies for the distributed file storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0044] Figure 1 A flow chart of the steps of a parameter configuration method of a storage system provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the structure of a distributed file storage system in an embodiment of the present invention;

[0046] Figure 3 is a structural diagram of a parameter configuration device for a storage system provided by an embodiment of the present invention;

[0047] Figure 4 It is a structural diagram of a communication device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0049] Reference Figure 1 , shows a flowchart of the steps of a parameter configuration method of a storage system provided by an embodiment of the present invention, the method is applied to a distributed file storage system, and specifically may include the following steps:

[0050] Step 101, obtain a message request.

[0051] The message request in the embodiment of the present invention is also called an IO request, which refers to a request from a client to read or write files stored on multiple servers in a distributed environment. The client's IO request first needs to be correctly routed to the server responsible for storing the relevant data.

[0052] The distributed file storage system in the embodiment of the present invention can be an NFS system (Network File System), which is one of the current mainstream heterogeneous platform shared file systems. It is mainly used in the UNIX (multi-user, multi-tasking operating system) environment, allowing a system to share directories and files with others on the network. The working principle of NFS is to use a client / server architecture, which consists of a client program and a server program. The NFS server provides access to the file system to other computers, and the process is called output. When the NFS client program accesses the shared file system, it "transmits" them from the NFS server, where shared files are usually transmitted in blocks.

[0053] IO operation refers to the process of reading and writing files in a computer system, which specifically involves the operating system kernel's read and write operations on the file system, including opening files, reading file contents, writing file contents, closing files, etc. When performing IO operations in a distributed file storage system, it is necessary to use the corresponding file system API (Application Programming Interface) or commands to implement read and write operations on files. Common IO operations include using file pointers for positioning reading and writing, reading and writing file contents, checking file status, etc. When performing IO operations, it is necessary not only to pay attention to effectively managing file handles to avoid resource leaks and too many files being opened, but also to consider exception handling of file operations to ensure the stability and reliability of file operations.

[0054] In the performance test of IO operation, IOPS is examined. IOPS refers to the maximum number of I / Os that the system can handle per unit time. It is generally measured in the number of I / O requests processed per second. I / O requests are usually read or write data operation requests. Another important indicator is data throughput, which refers to the amount of data that can be successfully transmitted per unit time. For applications with a large number of sequential reads and writes, such as VOD (Video On Demand), more attention is paid to the throughput indicator.

[0055] Step 102: parse the message request to obtain the request type and parameter information corresponding to the message request.

[0056] When dealing with network communications or data transmission, the message protocol is the key to ensuring the correct exchange of data. The message protocol usually defines the message format and fields. The specific execution steps include:

[0057] Parse the message request according to the preset message protocol to obtain the header information corresponding to the message request;

[0058] The header information is identified to obtain the request type and parameter information of the message request. The request type includes read / write request and metadata request.

[0059] The received message first needs to be parsed according to the predefined message format to obtain the message header information. The message header information usually contains some fixed-length fields, such as version number, message length, request type code, etc. These fields are extracted using the parsing algorithm or tool specified in the protocol. The request type code is usually a predefined value or string corresponding to a specific operation. According to the request type code in the header information, it can be determined whether the message is a read-write request or a metadata request.

[0060] In the embodiment of the present invention, for different request types of message requests, the parameter information obtained by parsing is also different, and the specific steps are as follows:

[0061] When the request type of the message request is a read-write request, extracting a first key parameter from the read-write request, the first key parameter including a file identifier (inode number), a read-write offset position, and a read-write length;

[0062] When the request type of the message request is a metadata request, a second key parameter is extracted from the metadata request, where the second key parameter includes a file identifier (inode number) and file attribute information.

[0063] In addition, the parsed information needs to be verified to ensure that it complies with the protocol and has no errors or anomalies.

[0064] In an example, suppose there is a simple message protocol with the following header format:

[0065] Version number (2 bytes)

[0066] Message length (2 bytes)

[0067] Request type code (1 byte): 0x01 for read request, 0x02 for write request, 0x03 for metadata request

[0068] Parameter length (2 bytes)

[0069] Parameter data (variable length)

[0070] The corresponding parsing process can be as follows:

[0071] Read the first 2 bytes to determine the version number;

[0072] Read the next 2 bytes to determine the total length of the message;

[0073] Read the fifth byte and determine the request type based on the request type code;

[0074] Read the 6th and 7th bytes to determine the length of the parameter;

[0075] Read parameter data according to the parameter length.

[0076] Step 103, using the pre-trained input-output model to perform prediction analysis on the message request to obtain a prediction result, where the input-output model is used to predict the reading mode of the message request.

[0077] After determining the request type of the message request, the embodiment of the present invention needs to use the pre-trained input-output model to predict the message request. The specific steps of training the input-output model include:

[0078] The initial prediction model and historical read data of the distributed file storage system are obtained, and the historical read data are divided into a training set and a validation set. The training set contains a number of original training data, and the validation set contains a number of original validation data.

[0079] Obtaining the historical read data of the storage system is an important step in training the input-output model. The historical read data records the detailed information of the read and write operations occurring in the storage system, including the time, type, size, source and target of the operation. This step requires obtaining the historical IO log of the distributed file storage system. The present invention does not specifically limit how to obtain the IO log of the storage system. For those skilled in the art, the acquisition of the IO log can be achieved through the log call function, log call interface, log file, etc. corresponding to the distributed file storage system. Many storage systems provide specific APIs or functions to record and obtain IO logs. For example, the log information can be sent to the corresponding process by calling the syslog() function. The storage system may provide a special log interface to allow users or administrators to query and obtain log information through the network or local interface. For example, some storage systems provide a Web (Web Interface) interface or a command line tool to access log data. Some storage systems save log files in the local file system. These files may be located in a specific directory, such as / var / log or / var / log / storage. Users can obtain IO logs by reading these files. In some storage systems, IO logs can be stored in a database. Users can obtain log information through a database query language. In addition, since different types of storage systems may have different architectures and designs, the specific methods for obtaining IO logs may be different, such as different corresponding log calling functions, etc., which should all be within the protection scope of the present invention.

[0080] The original training data in the training set is parsed, and the first parameter information corresponding to each original training data is extracted from the original training data, such as file path, reading time, reading size, etc.

[0081] The first parameter information is statistically analyzed to identify a first reading pattern corresponding to each original training data, which may include a reading frequency, a reading sequence, a reading hotspot, etc.

[0082] The initial prediction model is trained based on the original training data and the first reading pattern. This process may involve steps such as feature engineering, model selection, and parameter tuning. After the training is completed, a model to be verified is obtained.

[0083] The original verification data in the verification set are parsed, and the second parameter information corresponding to each original verification data is extracted from the original verification data. The second parameter information is statistically analyzed to identify the second reading mode corresponding to each original verification data.

[0084] The original verification data is input into the model to be verified for prediction, and a third reading mode output by the model to be verified is obtained.

[0085] The third reading pattern is compared with the second reading pattern. When the third reading pattern is completely identical to the second reading pattern, it is determined that the model to be verified is an input-output model.

[0086] Specifically, the third read pattern is compared with the second read pattern to check whether the two are exactly the same. If the third read pattern is exactly the same as the second read pattern, it is determined that the model to be verified is an input-output model, that is, the model can accurately predict IO behavior. If the patterns do not match, it may be necessary to return to the training stage, adjust model parameters or redesign features, and then train and verify again.

[0087] The above model training process ensures that the prediction model can accurately reflect and predict the IO behavior of the distributed file storage system, thereby helping to optimize system performance and resource allocation.

[0088] In the embodiment of the present invention, for different request types of message requests, the specific steps are as follows:

[0089] When the request type of the message request is a metadata request, the input-output model in the distributed file storage system is obtained, and the attribute information of the input-output model is updated according to the second key parameter corresponding to the metadata request.

[0090] Specifically, in a distributed file storage system, metadata requests usually involve querying or modifying descriptive information about files or objects, such as the creation time, permissions, size, etc. When the request type is a metadata request, the system may need to update the attribute information of the input and output model based on the specific content of the request in order to more accurately predict and optimize future IO behavior.

[0091] When a metadata request is received, the request is first parsed to determine its specific content, including the metadata type to be queried or modified and the related second key parameter. According to the parsed metadata request, the corresponding input-output model is obtained from the distributed file storage system. This model is previously trained through machine learning and is used to predict and optimize IO behavior. The second key parameter in the metadata request is used to update the attribute information of the input-output model, which may include the access frequency of the file, the last access time, user permissions, etc. The update process may involve modifying the internal state of the model, or updating the records in the database or cache used by the model. After updating the model attribute information, it is necessary to verify whether the update is successful and ensure that the updated model still maintains consistency and correctness. In addition, the operation of updating the model attribute information should be recorded in the system log for technical personnel to conduct audits and troubleshooting.

[0092] In this way, the distributed file storage system can dynamically adjust its input and output model based on the latest metadata requests, thereby managing and optimizing storage resources more efficiently.

[0093] When the request type of the message request is a read-write request, the read-write request is input into the input-output model for prediction to obtain a read mode corresponding to the read-write request.

[0094] In a distributed file storage system, when the request type of a message request is a read-write request, the system can use a previously trained input-output model to predict the read patterns of these requests. The read patterns may include the access frequency, sequence, and hotspot areas of the requests.

[0095] When a read or write request is received, the request is first parsed to determine its specific content, including the requested operation type (read or write), the target file or object, the requested size, the offset, etc. The parsed read or write request is input into the input-output model. The input-output model processes the input read or write request and outputs a predicted read mode. For example, the display of the prediction result can be a probability distribution, a sequence, a heat map, or other forms of representation. In an embodiment of the present invention, the data layout in the cache can be adjusted according to the predicted access frequency and order to improve the cache hit rate. Storage resources can also be reallocated according to the predicted hot spot areas to avoid overloading certain nodes. Data that may be needed can also be loaded into the cache in advance according to the predicted reading order to reduce delays.

[0096] After obtaining the prediction results, you can record the prediction results and the actual IO behavior of the message request for future analysis and model adjustment. If the prediction results deviate significantly from the actual behavior, you may need to retrain the model or adjust the parameters. At the same time, the system continuously monitors the performance of the input and output models and makes necessary adjustments based on the actual IO behavior and system load.

[0097] In this way, the distributed file storage system can use the input-output model to predict and optimize read and write requests, thereby improving the overall performance and responsiveness of the system.

[0098] Step 104: formulate an optimization strategy for the distributed file storage system based on the prediction result.

[0099] In the embodiment of the present invention, an optimization strategy for the distributed file storage system is formulated according to the reading mode predicted by the input-output model. The optimization strategy at least includes adjusting the cache distribution and network threads of the distributed file storage system.

[0100] For example, adjust the receive mode of the network thread according to the characteristics of the IO request (such as the file size and the size of the IO operation):

[0101] For small files and small IO operations, since the amount of data in each request is small, processing each request separately may lead to inefficient use of network and processing resources. Therefore, the network thread can be set to adopt batch receiving mode, that is, receiving multiple requests at a time, and then processing these requests in batches. Batch processing can reduce the overhead of network communication and improve processing efficiency, especially in high-concurrency small IO request scenarios.

[0102] For large files and large IO operations, batch processing may cause resource locking and increased latency due to the large amount of data in each request. Therefore, the network thread can be set to adopt a single request receiving mode, that is, each network thread only processes one request at a time. The single request receiving mode ensures that each request can be processed quickly, reducing waiting time, and is suitable for scenarios with high real-time requirements.

[0103] In one embodiment of the present invention, in terms of adjusting cache distribution: according to the predicted read pattern, frequently accessed data (hot data) can be cached to a location closer to the request source, such as a memory cache or a local SSD (Solid State Drive) to reduce access latency. It is also possible to predict data that is about to be accessed and load it into the cache in advance to reduce the response time of read requests. It is also possible to adjust the cache elimination algorithm, such as LRU (Least Recently Used) or LFU (Least Frequently Used), based on the prediction of the read pattern to better adapt to the predicted access pattern.

[0104] The basic idea of ​​the LRU strategy is that the least recently used data items are the most likely not to be used in the future, so when space needs to be made for new data, these data items should be removed first. When implementing the LRU strategy, it is usually necessary to maintain a data structure to record the access order of each data item. This data structure can be a linked list or a combination of a hash table and a linked list (such as LinkedHashMap). The basic implementation steps of the LRU strategy are as follows:

[0105] Whenever a data item is accessed (read or written), it should be moved to the front (or back, depending on the implementation) of the data structure. When cache space is insufficient, remove the oldest data item (the least recently used data item) in the data structure. Combine hash tables and linked lists to achieve fast search, insertion, and deletion at the same time.

[0106] The basic idea of ​​the LFU strategy is that the data items with the lowest access frequency are the most likely not to be used in the future, so when space needs to be made for new data, these data items should be removed first. When implementing the LFU strategy, it is usually necessary to maintain a data structure to record the access frequency of each data item. This data structure can be a combination of a hash table and a priority queue (such as a minimum heap). The basic implementation steps of the LFU strategy are as follows:

[0107] Whenever a data item is accessed (either read or written), its access count should be increased. When cache space is insufficient, remove the data item with the lowest access frequency. If there are multiple data items with the same lowest access frequency, you can further decide which one to eliminate based on other strategies (such as LRU). Combine hash tables and priority queues to achieve fast search and elimination at the same time.

[0108] In terms of optimizing network threads: the size of the network thread pool can be dynamically adjusted based on the predicted reading pattern and system load to ensure that there are enough threads to handle upcoming requests while avoiding resource waste. It is also possible to set priorities for different types of read and write requests, and prioritize high-frequency or critical requests based on the predicted reading pattern. It is also possible to adjust the allocation of network threads based on the predicted reading pattern to ensure load balancing across servers and network paths and avoid single-point bottlenecks.

[0109] In addition, it also includes adjusting the physical layout of data on storage devices according to the predicted reading pattern to reduce disk seek time and improve data access efficiency; dynamically adjusting the distribution of data copies according to the predicted hot spots to ensure that there are enough copies of hot data to support high concurrent access; selecting or adjusting the I / O scheduling algorithm to better adapt to the predicted reading pattern, such as using a completely fair queue algorithm.

[0110] When implementing optimization strategies, it is necessary to consider the actual operating status of the system, resource constraints, and user needs. In addition, the optimization strategy should be dynamic and able to adjust based on real-time feedback from the system and continuous learning of the model. Through these optimization measures, the distributed file storage system can respond to read and write requests more efficiently, improving overall performance and user experience.

[0111] Step 105: configure parameters of the distributed file storage system according to the optimization strategy.

[0112] The storage system parameter configuration method provided in the embodiment of the present invention obtains a message request, parses the message request, obtains the request type and parameter information corresponding to the message request, uses a pre-trained input-output model to predict and analyze the message request, obtains a prediction result, formulates an optimization strategy for the distributed file storage system based on the prediction result, and configures the parameters of the distributed file storage system according to the optimization strategy. The embodiment of the present invention ensures that the distributed file storage system provides services with optimal performance in various scenarios by formulating and applying different tuning strategies for the distributed file storage system.

[0113] Reference Figure 2 , shows a schematic diagram of the structure of a distributed file storage system according to an embodiment of the present invention, in which each service node (i.e., storage server) in the distributed file storage system needs to run an NFS server. NFS is a network file system protocol that allows clients to access files and directories stored on a server through a network, just like accessing local storage.

[0114] After the NFS server of each service node is started, it needs to establish a connection with the data center through TCP (Transmission Control Protocol Port). TCP is a reliable network communication protocol that ensures the accuracy and order of data transmission.

[0115] The NFS server listens on a specific TCP port, waiting to receive requests from the client. The TCP port is a logical interface in network communication, used to distinguish different services.

[0116] When the NFS server receives a request, it parses the NFS message of the request. The header information of the NFS message contains key information, such as the request type (read, write, get file attributes, etc.) and related parameters. After the NFS server parses the key parameters, it uploads them to the data center.

[0117] For read and write requests, the key parameters uploaded may include the file's unique identifier (inode), the length of the data to be read or written (len), and the starting position of the read or write (offset). For metadata requests (such as obtaining file attributes), the key parameters uploaded may include the inode and the file's attributes (such as file size, permissions, etc.).

[0118] After receiving the uploaded key parameters, the data center will use the inode (unique identifier of the file) as an index and store this information. The data center will also process this information, maintain the size of each file, the size and distribution of IO operations, and count the file distribution of the entire distributed storage system. At the same time, the data center will analyze the collected key parameters to predict the actual IO model (i.e., file access pattern) and determine which optimization strategy should be executed based on these predictions. Optimization strategies may include adjusting network modes, caching strategies, load balancing, etc. to improve system performance.

[0119] The data center will calculate the optimal network mode configuration and send it to the NFS server that actually accesses the service. These configurations will guide the NFS server on how to optimize network reception, such as choosing to receive batch read and write network requests to speed up network reception. The NFS server will optimize based on the sent configuration to ensure more efficient network reception.

[0120] Through the above process, the distributed file storage system can collect each IO information of each node around the clock and dynamically adjust its configuration to adapt to the ever-changing IO requests and file access patterns, thereby improving the overall performance and response speed of the system.

[0121] In an embodiment of the present invention, when formulating an optimization strategy, a user configuration interface can be designed to allow users to set optimization preferences according to their own needs. The system dynamically adjusts the optimization strategy according to these preferences and provides personalized performance optimization solutions.

[0122] In addition, a fault-tolerant mechanism is designed, which can automatically switch to the backup model when the input and output models fail, or use model checkpoint and recovery technology to quickly restore the normal operation of the model and minimize the impact on system performance.

[0123] Reference Figure 3 , which shows a schematic diagram of the structure of a parameter configuration device for a storage system provided by an embodiment of the present invention, such as Figure 3 As shown, the device may include:

[0124] The first acquisition module 301 is used to acquire a message request.

[0125] The first parsing module 302 is used to parse the message request to obtain the request type and parameter information corresponding to the message request.

[0126] The first prediction module 303 is used to perform prediction analysis on the message request using the pre-trained input-output model to obtain a prediction result.

[0127] The strategy formulation module 304 is used to formulate an optimization strategy for the distributed file storage system based on the prediction results.

[0128] The parameter configuration module 305 is used to configure the parameters of the distributed file storage system according to the optimization strategy.

[0129] Optionally, the first parsing module 302 further includes:

[0130] The header information acquisition module is used to parse the message request according to the preset message protocol to obtain the header information corresponding to the message request. The message protocol includes the message format and fields.

[0131] The header information identification module is used to identify the header information to obtain the request type and parameter information of the message request. The request type includes a read-write request and a metadata request.

[0132] Optionally, the first parsing module 302 further includes:

[0133] The first key parameter extraction module is used to extract the first key parameter from the read / write request when the request type of the message request is a read / write request. The first key parameter includes a file identifier, a read / write offset position, and a read / write length.

[0134] The second key parameter extraction module is used to extract the second key parameter from the metadata request when the request type of the message request is a metadata request, where the second key parameter includes a file identifier and file attribute information.

[0135] Optionally, the parameter configuration device of the storage system further includes:

[0136] A training data acquisition module is used to acquire the initial prediction model and the historical read data of the distributed file storage system, and divide the historical read data into a training set and a verification set. The training set contains a number of original training data, and the verification set contains a number of original verification data.

[0137] A training data parsing module, used to parse the original training data and extract first parameter information corresponding to each original training data from the original training data;

[0138] A training data identification module, used for statistically analyzing the first parameter information to identify a first reading mode corresponding to each original training data;

[0139] A model training module, used to train the initial prediction model according to the original training data and the first reading mode to obtain a model to be verified;

[0140] A verification data parsing module, used to parse the original verification data and extract second parameter information corresponding to each original verification data from the original verification data;

[0141] A verification data identification module, used for statistically analyzing the second parameter information to identify the second reading mode corresponding to each original verification data;

[0142] A model verification module, used for inputting original verification data into the model to be verified for prediction, and obtaining a third reading mode output by the model to be verified;

[0143] The model generation module is used to compare the third reading pattern with the second reading pattern, and when the third reading pattern is completely identical to the second reading pattern, determine that the model to be verified is an input-output model.

[0144] Optionally, when the request type of the message request is a metadata request, the first parsing module 302 further includes:

[0145] The model acquisition submodule is used to obtain the input and output models in the distributed file storage system.

[0146] The model updating submodule is used to update the attribute information of the input and output models according to the second key parameter corresponding to the metadata request.

[0147] Optionally, when the request type of the message request is a read / write request, the first prediction module 303 further includes:

[0148] The prediction submodule is used to input the read and write requests into the input and output model for prediction to obtain the read mode corresponding to the read and write requests.

[0149] Optionally, the policy formulation module 304 further includes:

[0150] The strategy formulation module submodule is used to formulate an optimization strategy for the distributed file storage system according to the reading mode predicted by the input-output model. The optimization strategy at least includes adjusting the cache distribution and network threads of the distributed file storage system.

[0151] The storage system parameter configuration method provided in the embodiment of the present invention obtains a message request, parses the message request, obtains the request type and parameter information corresponding to the message request, uses a pre-trained input-output model to predict and analyze the message request, obtains a prediction result, formulates an optimization strategy for the distributed file storage system based on the prediction result, and configures the parameters of the distributed file storage system according to the optimization strategy. The embodiment of the present invention ensures that the distributed file storage system provides services with optimal performance in various scenarios by formulating and applying different tuning strategies for the distributed file storage system.

[0152] The embodiment of the present invention also provides a communication device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0153] Memory 403, used for storing computer programs;

[0154] The processor 401 is used to execute the program stored in the memory 403 to implement the following steps:

[0155] Get message request;

[0156] Parse the message request to obtain the request type and parameter information corresponding to the message request;

[0157] Use the pre-trained input-output model to perform predictive analysis on the message request to obtain a prediction result. The input-output model is used to predict the reading mode of the message request.

[0158] Formulate optimization strategies for distributed file storage systems based on prediction results;

[0159] Configure parameters of the distributed file storage system based on the optimization strategy.

[0160] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0161] The communication interface is used for communication between the above terminal and other devices.

[0162] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0163] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0164] The present invention also provides a readable storage medium, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can execute the parameter configuration method of the storage system of the foregoing embodiment.

[0165] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0166] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. According to the above description, it is obvious that the structure required for constructing this type of device is. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific language is to disclose the best mode of the present invention.

[0167] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0168] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all of the features of the individual embodiments disclosed above. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0169] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0170] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor may be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention may also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0171] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0173] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0174] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0175] It should be noted that the various data-related processes in the embodiments of the present application are all carried out in compliance with the corresponding data protection laws and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

Claims

1. A parameter configuration method for a storage system, characterized in that: Applied to a distributed file storage system, the method includes: Get message request; Parsing the message request to obtain the request type and parameter information corresponding to the message request; Performing a prediction analysis on the message request using a pre-trained input-output model to obtain a prediction result, wherein the input-output model is used to predict a reading mode of the message request; Formulate an optimization strategy for the distributed file storage system based on the prediction result; Parameters of the distributed file storage system are configured according to the optimization strategy.

2. The method according to claim 1, characterized in that: The parsing of the message request includes: Parsing the message request according to a preset message protocol to obtain header information corresponding to the message request, wherein the message protocol includes a message format and fields; The header information is identified to obtain the request type and parameter information of the message request, where the request type includes a read / write request and a metadata request.

3. The method according to claim 2, characterized in that The method further comprises: In the case where the request type of the message request is the read-write request, extracting a first key parameter from the read-write request, the first key parameter including a file identifier, a read-write offset position, and a read-write length; In a case where the request type of the message request is the metadata request, a second key parameter is extracted from the metadata request, where the second key parameter includes a file identifier and file attribute information.

4. The method according to claim 3, characterized in that: The input-output model is generated in the following way: Acquire an initial prediction model and historical read data of the distributed file storage system, and divide the historical read data into a training set and a verification set, wherein the training set includes a number of original training data, and the verification set includes a number of original verification data; Parsing the original training data, and extracting first parameter information corresponding to each original training data from the original training data; Statistically analyzing the first parameter information to identify a first reading mode corresponding to each of the original training data; Training the initial prediction model according to the original training data and the first reading mode to obtain a model to be verified; Parsing the original verification data, and extracting second parameter information corresponding to each original verification data from the original verification data; Statistically analyzing the second parameter information to identify a second reading mode corresponding to each of the original verification data; Inputting the original verification data into the model to be verified for prediction, and obtaining a third reading mode output by the model to be verified; The third reading pattern is compared with the second reading pattern, and when the third reading pattern is completely identical to the second reading pattern, it is determined that the model to be verified is an input-output model.

5. The method according to claim 3, characterized in that: In a case where the request type of the message request is the metadata request, the method further includes: Obtaining an input and output model in the distributed file storage system; The attribute information of the input-output model is updated according to the second key parameter corresponding to the metadata request.

6. The method according to claim 3, characterized in that When the request type of the message request is the read-write request, the predictive analysis of the message request using the pre-trained input-output model includes: The read / write request is input into the input / output model for prediction to obtain a read mode corresponding to the read / write request.

7. The method according to claim 1, characterized in that The determining of an optimization strategy for the distributed file storage system based on the prediction result includes: According to the reading mode predicted by the input-output model, an optimization strategy for the distributed file storage system is formulated, and the optimization strategy at least includes adjusting the cache distribution and network threads of the distributed file storage system.

8. A parameter configuration device for a storage system, characterized in that: Applied to a distributed file storage system, the device comprises: A first acquisition module, used to acquire a message request; A first parsing module, used to parse the message request to obtain a request type and parameter information corresponding to the message request; A first prediction module, used to perform prediction analysis on the message request using a pre-trained input-output model to obtain a prediction result; A strategy formulation module, used to formulate an optimization strategy for the distributed file storage system based on the prediction result; A parameter configuration module is used to configure parameters of the distributed file storage system according to the optimization strategy.

9. A communication device, characterized in that: include: A transceiver, a memory, a processor, and a program stored on the memory and executable on the processor; The processor is used to read the program in the memory to implement the steps in the parameter configuration method of the storage system as described in any one of claims 1-7.

10. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps in the parameter configuration method of the storage system as described in any one of claims 1 to 7 are implemented.

Images